How DataOps Can Make Your Data More Actionable
The mantra that all companies are software companies continues to hold true. At the same time, in today’s digital world, integrating data in all facets of the organization has become an essential part of becoming an autonomous digital enterprise.
An autonomous digital enterprise is “customer-centric, agile and derives actionable insights from data,” according to a report by 451 Research sponsored by BMC. These “actionable insights” are derived from analyzing data across an organization’s entire operations, covering CI/CD production pipelines, internal communications and, of course, customer data.
Data management has thus become a top priority for organizations’ autonomous digital enterprise journeys. According to the results of a survey included in 451 Research’s study of 1,200 business and IT decision-makers, representing organizations with annual revenues of over $100 million, 43% of participants cited a “data-driven business” as their preferred business models and technology tenets.
A data-driven business treats data throughout the organization as an “asset,” as “organizations use analytics and artificial intelligence,” according to the research paper. The challenge is how to integrate data-driven processes and the requisite culture to become a true autonomous digital enterprise.
This is where DataOps — or “DevOps for data,” for which the creator of a data product takes ownership of it throughout its lifecycle — enters the picture.
The promise of DataOps is about “increasing the likelihood of success of data and analytics initiatives — traditionally, these initiatives have achieved less than expected business benefits, for a variety of reasons,” Ram Chakravarti, chief technology officer for BMC, told The New Stack in an email interview.
Ram Chakravarti, chief technology officer of BMC.
“DataOps, if done right, will enforce extensive collaboration across all the stakeholder groups in data and analytics use cases and end-to-end automation of data pipelines, both of which will lead to faster realization of desired business outcomes,” Chakravarti said.
“In turn, enterprises can adjust, respond, predict and act autonomously based on applicable scenarios. By implementing DataOps processes, organizations can gain insights that drive decision making in near real-time and become autonomous digital enterprises.”
The New Stack asked Chakravarti more about DataOps, including what organizations must do to adopt that approach and achieve an autonomous digital enterprise status. This interview has been edited for brevity and clarity.
The New Stack: What are the tools and cultural changes required to achieve DataOps?
Ram Chakravarti: In order to successfully implement DataOps, organizations should establish a multihorizon initiative with a set of deliberate steps. Introduce DataOps in one or two analytics use cases and start small. Through these use cases, learn, refine and then scale DataOps across additional business functions.
As this process continues, keep applying specific best practices until reaching a steady state. Across these horizons, it is fundamentally important to have executive buy-in and to deliver quick wins for the business stakeholders, so that they become the change agents to help scale DataOps across the enterprise.
What role does automation play?
Through pervasive automation that encompasses orchestration, provisioning, configuration and self-service, DataOps helps organizations capitalize and evolve their data. Automation technologies within DataOps, as well as extensive cross-business collaboration, can ensure the delivery discipline required for successful data and analytics transformations.
The “backbone” of automation as a tenet of DataOps is data-pipeline orchestration. Orchestration is a must-have, to coordinate the discrete data management tools and functions and to ultimately automate from source to fully operationalized insights. This is especially pertinent for any organization striving to become a data-driven business and autonomous digital enterprise.
What benefits should DataOps offer an organization? For example, what about speed?
DataOps is about incorporating DevOps best practices to the data management ecosystem. While it offers speed, it’s not necessarily at risk of quality. As with any best-in-class DevOps practice, it’s a balance between speed and quality that is at the heart of DataOps.
Can it offer better data analytics — as in, more real-time results?
It’s not just about DataOps, there are some disciplines underneath that need to be addressed as well.
Just implementing DataOps does not automatically guarantee better data analytics results. This is where we introduce MLOps with advanced analytics. MLops, which is a cousin of DataOps, is an agile approach to training and deploying [machine learning] models. That can basically be a potential solution to successfully operationalizing new ML models, thereby providing better data-analytics results.
Does DataOps make it easier to scale?
DataOps is not infrastructure. It can help you scale more easily because you’re going to introduce systematic techniques that help you deploy new use cases to production in a fairly systematic way. So from that point of view, if you have a repeatable set of best practices, it helps you scale from one to two use cases to an entire business function and eventually across the enterprise. So yes, it’s easier to scale.
Can it create more awareness throughout the team of data’s role?
Yes, absolutely. That’s at the heart of what we call extensive collaboration across the stakeholders. DataOps allows for extensive collaboration across the data management ecosystem — the personas spanning business and IT. With DataOps, there are clear success measurements across the organization.
Can DataOps make it easier to fix problems, like broken dashboards and so on, before they reach the end user?
This will help provide high-quality data but there are complementary capabilities and dashboarding that need to also be put in place, that are not necessarily DataOps. We shouldn’t overreach in terms of claiming what DataOps sets out to do.
What are the main challenges of automating DataOps?
Cultural change is really at the heart of it. It’s not automating DataOps.
I’d say the challenges are really adopting DataOps at scale in an enterprise. And those challenges are primarily around exec buy-in and support and collaboration across the ecosystem in what has traditionally been a relationship based on mistrust. Overcoming those barriers is going to be important.
The engineering of automation technologies itself requires a certain amount of discipline. That means having the right horse for the right course and not just embracing technology. If you are pragmatic in your technology adoption, then embracing the technology and institutionalizing it is not as big of a deal. It’s really all about collaboration and the exec buy-in.
How is BMC tackling these issues?
BMC’s Control-M solution is the leading enterprise-scale orchestration solution and it is finding significant use in the management of complex data pipelines in the hybrid IT landscape. The centrality of orchestration as an important tenet of automation gives BMC a pretty solid foundation in support of the DataOps movement.
Additionally, BMC is investing heavily in capabilities to complement data pipeline orchestration.
With orchestration as the central foundation, we aim to eliminate challenges faced by data personas in the development community (i.e. the data engineers) and in the operations community (i.e. data infrastructure engineers), as well as among the consumers, such as data analysts.
