Ever hear of the Gordie Howe hat trick? Probably not, unless you’re Canadian (like me). After reading the obscure reference, you likely reflexively pulled out your smartphone to consult Google because let’s face it, you’re not going to the library to search for the answer. Your friends probably can’t help either. But Google can and — something that should be considered amazing — it predicted what you’d write before you even finished typing “Howe” in the search bar.
Not too long ago, problem-solving like this wasn’t predictive; rather, it was largely backwards-looking. For example, let’s say you wanted to know how many units of red socks you sold last quarter by geography. An analyst team would gather a data set and, after some manipulation, figure it out and then create a weekly historical report.
The future for businesses looks a lot more like Google auto-predict.
Google is our go-to for any query these days because it taps into “pervasive intelligence”: the concept that organizations use their data as a competitive advantage by making it available and consumable across the organization, in real time. Pervasive intelligence means that understanding a problem is no longer backwards-looking; rather, it’s predictive and prescriptive. With Google, you are the analyst, and you can interact with big data as you need to.
Pervasive Intelligence + Batch and Streaming Data
This pervasive intelligence concept is being used everywhere. The algorithm that powers Airbnb’s engine, for example, applies pervasive intelligence to batch data in order to match a renter and an owner using standardized text. The owner that submits a listing is steered toward using a certain few words and phrases to describe their unit — “two bedrooms, two bathrooms.” Similarly, the renter sees Google-like predictive text when they type in their unit requirements. Voila! The renter and owner are matched.
Similarly, you can see pervasive intelligence applied to batch data when you shop on Amazon. A massive index across Amazon’s huge pool of data yields items that you might also like — things that have similar attributes to what you’re looking for. Again, the idea is to take the guesswork out of users’ searches.
In short, pervasive intelligence understands individuals — their browsing patterns, product preferences and so on — and makes up their minds for them.
Applying the pervasive intelligence concept at scale to big data yields significant benefits for businesses too. Take Netflix, for example. Using Google-type pervasive intelligence with streaming data yields real-time insight into show preferences, which allows Netflix to promote particular shows to specific users that might be interested. Likewise, it provides insight into its users’ consumption patterns so that Netflix can figure out peak usage times and optimize its AWS bands to suit. And, while it’s a different analytics manipulation, pervasive intelligence can even be applied to Netflix’s offline experience, as data points specific to programs watched offline factor into the data stream and the resulting analytics as well. Clearly, applying the concept of pervasive intelligence to both batch and streaming data yields incredible value for today’s businesses.
Build vs. Buy
Achieving pervasive intelligence requires some change in organizations’ teams, practices and systems so that they have visibility into data movement and can make new data available for analysis, faster. For some organizations, this may be a matter of tapping their engineering staff or data scientists to write custom code through APIs. Many Silicon Valley tech startups are lucky (or well-funded) enough to have attracted the scarce, specialized data engineering resources to tackle the time-consuming and non-scalable chore of hand-coding everything. However, most organizations don’t have the resources to devote to writing code and must find other avenues to gain insight into their streaming and batch data. Essentially, it comes down to a build-or-buy choice: Do you spend the countless hours on developing and maintaining your own algorithms, or do you look for purpose-built technology to meet your pervasive intelligence needs?
Regardless of which camp your organization identifies with or your choice to build or buy, your path to pervasive intelligence must consider the following four points.
Make achieving pervasive intelligence as simple a process as possible. Your coders must be proficient in the different APIs for all the different engines out there. Then when data movement hits critical scale, they can navigate through it more easily.
Ensure your organization has a constant view of its data in every state. Paying attention to data movement is important, as is what’s happening within that data set. You must know that your data is moving freely in order to ensure pervasive intelligence.
Put in place a self-healing mechanism. When was the last time Google search was down? You probably can’t think of one instance. Data semantics and structure are so dynamic these days, data integrity is going to be compromised no matter what. Address this proactively to ensure you can continue to gain pervasive intelligence in the face of constant data change.
Protect sensitive information, regardless of where it is. Regulations like HIPAA and GDPR require that mechanisms be put in place to protect sensitive information. This must be done for both data at rest and data in motion (as it’s moving around your organization).
Data is the lifeblood of any organization, and the ability to gain insight into it and use it effectively has great benefit. In fact, organizations that achieve pervasive intelligence can expect to maintain a competitive edge in any market. By the way, don’t forget to Google “Gordie Howe hat trick.” Then thank Google and pervasive intelligence for that bit of information that you probably couldn’t live without.
Feature image via Pixabay.
The New Stack is a wholly owned subsidiary of Insight Partners. TNS owner Insight Partners is an investor in the following companies: Real, Bit.