Elasticsearch Will Soon Get Machine Learning with Elastic’s Prelert Acquisition
Elastic, the company behind search and analytics engine Elasticsearch, liked the behavioral analytics integration that Prelert unveiled at Elastic{con} in February that it has purchased the company.
The move will jump-start Elastic’s foray into machine learning, which increasingly is becoming a business imperative for big data and infrastructure-monitoring vendors.
“They just brought so much functionality out of the box to the stack,” said Gaurav Gupta, vice president of products at Elastic.
“We had customers asking for this, and this is really hard stuff to build. … A lot of systems deliver false positives or they don’t work well on a lot of data sets. We thought Prelert’s technology was very generic, but at the same time very accurate from talking to customers. Their seven years of work was not something we’d be able to whip out, so it made a lot of sense to take them on.”
The new technology will allow users to look at anomalies within IT infrastructure, within transactional data for security or to detect fraud, according to Jeff Yoshimura, Elastic’s worldwide head of marketing.
“This technology allows us to offer our users more sophistication to go beyond just search — to make data consumable and take action on it,” he said.
It should also be able to build out its IT monitoring capabilities where others already have a head start. Datadog, which made its own acquisition last year, recently released an application performance monitoring (APM) extension that includes machine-learning-based anomaly detection. And new APM vendor Instana uses an Elasticsearch cluster in explaining how its Dynamic Graph can drill down into problems.
The Prelert integration covers Elasticsearch and the visualization tool Kibana. In the short run, its API Engine and Integration for Elastic Stack will be labeled beta while Elastic more formally integrates machine learning into its X-Pack commercial product line, which includes the Graph plugin announced in March.
Some Guidance Needed
Unsupervised machine learning, which Prelert uses, has no “teacher” guiding it or correcting it as the learning occurs. Algorithms are left on their own to discover and present the interesting structure in the data.
However, that doesn’t mean there’s a magic button you can push and — Voila! — it spits out compelling insights. And it will not replace the human touch, said Mike Paquette, Prelert head of products. He describes it more as having an algorithmic assistant.
“The pragmatic reality is that the subject-matter expertise of your own folks is needed to give guidance to the machine learning. They need to say, ‘There’s this class of data and I think there might be something interesting there. Go model this class of data and tell me if there’s anything unusual there.’
“You can have people who don’t have degrees in data science, but they do understand how data can affect their business, so that level of expertise is what can really put this technology to work,” he said.
He described two previous Prelert use cases:
- It was called in after a large online retailer’s back-end system that generated time-stamped invoices failed to generate them for its largest customer for 21 days. No one had noticed, but the problem was so severe it affected the company’s annual earnings. Using machine learning, Prelert modeled the normal behavior over time for each of the client’s thousands of customers and found the anomaly — as well as two other interruptions that the client hadn’t known about.
- A global telecom had a problem with customer-facing applications both online and in store. After going through the process for sales of plans and equipment, users would find the button to actually submit the order had disappeared. That problem also was affecting the client’s bottom line. Using a proof of concept for this still-active problem, looked at all the systems supporting that application — database servers, network connections, middleware servers, application servers — to help find and fix this bug.
Business Imperative
Artificial intelligence, of which machine learning is just one facet, increasingly is becoming the “secret sauce” in business technology as well as IT tools. Consider, just this month:
- Salesforce introduced Einstein, which puts a platform for machine learning, natural language processing and predictive analytics in the background of its CRM offerings.
- At OpenWorld in San Francisco, Larry Ellison outlined a strategy for “Adaptive Intelligent Applications” that combine third-party data, corporate data and behavioral analytics.
- IBM introduced its OpenPOWER servers designed to boost power for AI workloads.
- And security vendor Webroot acquired CyberFlow Analytics, which applies machine learning to automatically detect cybersecurity threats.
AI over the years has proved pretty disappointing, Gupta points out, adding that he believes now the time is right to see real benefits, including return on investment.
A new report on AI from Stanford University points to various factors behind the increased interest in the technology now, including maturing of machine learning, fueled by improved computing capabilities and web-based data-gathering; a leap in performance of information-processing algorithms; and new platforms and markets for data-driven products, and the economic incentives to find those products and markets.
The big challenge now, it says, is scaling existing algorithms to work with extremely large data sets — the kind Elastic’s customers have.
AI or cognitive computing offers three primary benefits, Manoj Saxena, founding general partner of The Entrepreneur’s Fund and former general manager for IBM Watson, says in a Goldman Sachs report. They are enhanced ability to provide highly personalized insights and experiences to customers; better and more timely decision-making; and creating more profitable relationships with customers and suppliers.
Companies that are slow to adopt it will lose competitive advantage to more agile competitors, he says.
