Dimensional Data: Making the Leap from Monitoring to Observing
IT performance monitoring is undergoing a major transformation thanks to advances in artificial intelligence and real-time analytics. Many IT monitoring platforms promise new capabilities that will move them beyond monitoring to observability.
But before we get too far, let’s first define the goal of observability. Systems engineer and author Cindy Sridharan offers one of the clearest definitions to date in an essay on Medium. She describes how observability, a superset of monitoring, combines alerting/visualization, distributed systems tracing infrastructure and log aggregation/analytics to provide better visibility into IT systems health.
In order to achieve such a lofty goal, the industry must shift operational data collection from a focus on flat metrics to what we call “dimensional data.” Achieving observability starts with internal instrumentation and external data collection, the output of which is then passed to various analytical or graphing engines within a monitoring platform.
Why Current Metrics Fall “Flat”
To understand the need for dimensional data, it’s important to first understand the challenges
associated with the data collection status quo, something we refer to as “flat metrics.” Flat metrics provide only surface-level analysis, without segmenting a technology into its various roles and components.
This was not the case a decade ago, when most environments could be integrated using a handful of scripts or APIs. But since that time, the number of APIs has exploded, hybrid and even multi-cloud environments have mainstreamed and newer technologies like containers and serverless have clouded relationship visibility. The very abstraction techniques that make them so user-friendly to DevOps make them more challenging to monitor and analyze.
Simply put, it’s no longer scalable for an organization to be able to manually consider how relationships between different layers in their IT stack may be impacting alerts, performance problems and outages. Without the context of the complex relationships in the modern IT environment, flat metrics can create false-positives in monitoring platforms.
Key Elements of Dimensional Data for Observability
Dimensional data refers to the stream of information a next-generation monitoring
integration strategy delivers, such as monitoring-integration-as-a-service (MIaaS). MIaaS is an emerging set of tools for connecting monitoring platforms that are deployed in different IT environments, so they can analyze the health and performance of the entire ecosystem. MIaaS is often used by large business-to-business enterprises that need to integrate on-premise infrastructure performance metrics with cloud service-performance metrics or in multicloud environments.
A dimensional data stream will include highly granular behavioral detail — beyond what a single endpoint API connection might include — as well as rich relational context. The term dimensional data is specific to IT system health and performance information and should not be confused with dimensional models used in data warehousing or low- and high-dimensional data sets in business analytics.
In contrast to flat metrics, dimensional data combines the standardization, relational visibility and super metrics that move performance monitoring and analytics platforms closer to observability.
Some traditional approaches to monitoring integration may cover one or two dimensions of data — but the only way to access all four dimensions of operational data is with a MIaaS.
Here are the four elements of dimensional data that observability requires:
- Universal Data Language
A MIaaS provider translates all the metrics from a given endpoint into a universal data language, which makes it compatible with any monitoring platform and every use case, such as alerts, dashboards, reports, etc. The result is a standard process for integration — an integration layer if you will — that delivers universally accessible insights to every analytics platform within the organization. - Internal Relationship Links
Dimensional data includes data from multiple and varied types of APIs, ranging from REST to SOAP and SNMP to ODBC. MIaaS links them together, so you can analyze performance by endpoints, such as a PostgreSQL DBMS or cluster, or drill down into a specific component, such as a database instance or node. - External Relational Metadata
Internal relationship links can provide the highly granular detail you need to achieve observability, while external relationships provide another key requirement: rich context. As a MIaaS data provider processes new metrics, every metric with potential external relationships is flagged with a tiny piece of metadata. When shared with an analytics platform through a dimensional data stream, relationship metadata can significantly reduce alert fatigue and clear the way to observability. - Super Metrics
A dimensional data stream can deliver raw and synthetic metrics, or more precisely, super metrics. These calculated metrics combine multiple raw metrics to calculate rates or ratios that have more meaning than the original flat metric.
The Path to Observability Requires Dimensional Data
Monitoring purists such as Sridharan — cited above — might argue the best route to observability is to look for a way to bring together insights from your monitoring, log analytics, tracing and other tools. That approach works best when each of the tools draws from the same well of information, like a dimensional data stream.
However, in practice, some organizations try to achieve observability by expanding the capabilities of their existing monitoring tools to take on more observability-like functions. Indeed, many monitoring, log analytics and tracing tools are designed as an attempt to become a single platform for observability.
Organizations also tend to deploy multiple monitoring tools. Many of the platforms are best-in-breed at certain functions such as APM, log analytics and tracing, which is why teams often deploy more than one based on the use case.
While it is unlikely any platform will become the single source for observability, dimensional data deployed through a MIaaS can offer any platform the highly granular intelligence and the relational context they need to achieve observability.
Blue Medora sponsored this story.
Feature image via Pixabay.