Modal Title
Culture / IoT Edge Computing / Observability

How to Fish Value From Your Data Lake

May 5th, 2016 11:37am by
Featued image for: How to Fish Value From Your Data Lake

Raymie Stata
Raymie Stata is the CEO and founder of Altiscale. Before founding Altiscale, Raymie was at Yahoo, where he was chief technical officer. At Yahoo, he played an instrumental role in algorithmic search, display advertising, and cloud computing. He also helped set Yahoo’s open-source strategy and initiated its participation in the Apache Hadoop project. Raymie received his PhD in computer science from MIT.

Your next big product may be submerged within your data lake.

According to market researcher Wikibon, data analytics remains one of the fastest growing IT segments. It predicts that the worldwide market for Big Data analytics hardware, software and services will reach $50 billion in 2017, up from $28.5 billion in 2014.

While many see the opportunity of Big Data deriving solely from the benefits of better decisions on behalf of the existing business, the implications of data, especially data from the emerging Internet of Things (IoT), are far greater than that.

We’ve already seen businesses born in the digital era harness customer data, social media insights, transaction data, geo-location data and device data to create new, data-driven product and service offerings. For example, companies like Facebook, Twitter and YouTube leverage user data to provide analytic reports about how consumers behave with content, enabling media companies to optimize ads and boost responses.

However, this ability to provide data products is not limited to native internet businesses. We are starting to see it expand to companies across a wide range of industries. For many of these companies, it is industrial and machine information, not just social or consumer information, that will be a major value driver.

Many established enterprises are sitting on vast stores of information that can be leveraged in new ways to create valuable data services. The payments processing company, First Data, for example, leveraged its troves of financial transaction data to provide consumer analytics to small-and-medium-sized businesses, companies that would otherwise lack the resources to make use of data. With First Data’s Insightics, companies can better understand purchase trends, the best time of day to make special offers and the ideal target customers for new products or services.

Data is a new kind of capital, vital to the development of new products and services.

PG&E, as another example, is making use of new information from digital meters, combining it with data about their enterprise customers, and providing dashboards to large companies to help them better understand their energy usage and how to manage it. Their customers can better control their energy spending while PG&E is better able to comply with state energy conservation mandates.

As more companies create data-driven products like this, here are five things to keep in mind:

1. Get an understanding of what you have: Complete a quick assessment of the data captured in your data lake. Consider how detailed the information is, how far back it goes, how it was collected, how accurate it is and how current it is. If you need to collect more information to fill in any gaps, consider the resources required and the possibilities of any additional details that you can capture along the way. For many IoT use cases, some of the data might be very novel – digital electrical smart meters, for example, capture far more information than an analog meter ever did.

2. Consider how third-party data amplifies the value of your company data: Augmenting your proprietary information with demographic, geographic, industrial, or weather data, for example, allows you to offer your customers new analyses that they can’t access anywhere else, or that would be too difficult to create on their own. For example, you can combine third party weather data with purchase history to better understand buying trends. With that information, businesses can strategically time specials and promotions that match consumer buying habits.

3. Adapt the user experience requirements to target market needs: Small businesses require easy, intuitive experiences so that those who aren’t data specialists can quickly benefit from the service. Analytics need to be quickly and easily understood to limit the need for additional data drilling and to mask the complexities involved. On the other hand, enterprises expect and demand greater access to the data details. Enterprise solutions serve multiple users, some of whom are highly technical and solely dedicated to data analysis. These expert users will want the option to drill down on data, run ad hoc analyses, or blend your information with their data.

4. Emphasize speed and service levels: Whether a small business or large enterprise, all users expect dashboards that are quick to load and update. Data must be up-to-date and responsive enough for fast analysis.

5. Make scalability a priority: Data volume and sources will inevitably grow over time, and you will need to be prepared to maintain service levels as they do so. In addition to scaling to accommodate more data, these services must also scale in terms of users and analytical sophistication. Scaling can be expensive and painful, so pick the right tech partner that is on the same page with you regarding performance, scalability, and operational support.

Data is a new kind of capital, vital to the development of new products and services. Connected devices and other data sources from IoT are providing, even more, fodder for enterprises. As the tools and infrastructure required to build and offer these data products become easier to acquire, access and use, more and more businesses are discovering the new business opportunities just waiting beneath the surface of their data lakes.

Feature image: Cancun street art by Senkoe.

Group Created with Sketch.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.