How Engineering Teams Will Evolve for Real-Time Product Data
If you’ve logged into Gmail recently, you may have noticed a new feature that generates suggestions for how to finish your sentences. Called Smart Compose, it’s an example of how product teams are using machine learning and real-time data to enhance the features and functionality of products. It’s a big change that can help businesses delight users and increase stickiness for their products, but it also means we need to rethink how engineering teams are structured.
Until now, data has served primarily a business function in most organizations, helping managers and executives make smarter decisions about how to increase sales and optimize processes. As such, data analysts and data scientists typically report to the business side of the house as part of corporate IT.
But data is now being used more in product development. Besides Google, Amazon has long used data to determine which merchandise to display as users move through its site, and many apps, such as Facebook and Instagram, select which content and ads to show in real time based on past activity. As cloud platforms evolve and make it easier to support such functionality, more mainstream businesses will start to do the same to maximize the in-app experience and improve business results.
To achieve this, data analysts and data scientists need to work much more closely with product engineers to provide the models and algorithms that enable these capabilities. Data in this context is typically acted on in real-time, which means engineering teams must be more closely aligned with data experts than in the past.
So what does this mean in practice? How should a CIO build an engineering team to enable tighter integration of data and product? There are two main areas to address: structure and processes.
Team Structure
For a feature like Gmail’s Smart Compose, engineers need access to a near-real-time flow of data to and from the product. That means infrastructure teams must work closely with product engineers to build and maintain the storage and data pipeline mechanisms to enable that. That’s a marked change from traditional business analytics, where the infrastructure team resides in corporate IT. In this new world, infrastructure needs to work alongside product engineers, allowing the two teams to collaborate in lock-step to achieve the desired result.
Data analysts and data scientists also need to become part of the engineering team structure. Rather than creating data reports for business leaders, they are now building the data models and pipelines to support product design and delivery. They must work closely with engineers to deliver the real-time analysis that drives the product.
Process
Besides these structural changes, data experts must be incorporated into the product delivery process to keep development running quickly and smoothly. That means they become part of the agile workflow that characterizes modern product teams, working alongside product managers, engineers and designers. This includes attending meetings for sprint planning, daily standups and retrospectives. Of course, data teams can be invited to these meetings, but making them a formal part of the process ensures they’ll be available and aligned with the engineering team’s objectives and priorities.
The New Normal
These changes should complement, rather than replace, the existing structures and processes within organizations. Data analytics will continue to be a vital part of business decision making, and data analysts will continue to play an essential role in corporate IT. But the new structure and processes are essential to support the expanding role data is playing in product delivery.
The use of real-time data in products was once restricted to bleeding edge tech firms like Google, Facebook and Amazon. But as with continuous delivery and integration, the pressure to constantly deliver new features and functionality means new modes of working are trickling down to a wider set of businesses. This includes smaller companies, which can now implement real-time data in products without needing such vast engineering resources.
These structural and process changes are only one aspect of supporting increased data use in products. The complexity of modern applications means other processes, such as QA testing, must also evolve to keep pace with the new demands. But rethinking how data teams and product engineers work together is a critical piece of the puzzle.
Feature image via Pixabay.