Why Grafana Needs OpenTelemetry
Grafana is often considered to be the hands-down most popular observability platform for data visualization. OpenTelemetry has emerged as a key open source component to lend compatibility among the different observability platforms. It was just a matter of time before OpenTelemetry became a key staple for Grafana users. Grafana Labs has inevitably sought to accommodate OpenTelemetry for its famous panel as well as for its open source alternative, including Grafana open source tools such as Loki (log aggregation), Mimir (for Prometheus metrics) and Tempo (a distributed tracing backend), as well as staple observability tools such as Prometheus (for metrics and alerting) and Graphite (platform for gathering and storing time-series data).
As mentioned above, Grafana and OpenTelemetry share a mutually beneficial relationship. From both a technological perspective Grafana has a lot of expertise in the field of observability and monitoring, and “they also have a pretty brisk commercial business in selling their solutions to large and small organizations,” Austin Parker, head of developer relations, Lightstep, said. “What we’re seeing from the telemetry side is organizations standardize on open source standards like OpenTelemetry and Grafana helping OpenTelemetry become successful helps not only OpenTelemetry but it also helps customers and it provides more opportunities for them,” Parker said.
On the user front, a lot has been recently happening in the OpenTelemetry camp. Before discussing Grafana’s user-friendly OpenTelemetry accommodations, there have been more than a few developments in the OpenTelemetry camp.
“Observability requires high-quality, portable, and accurate telemetry data. OpenTelemetry’s mission is to make this telemetry a built-in feature of cloud native software. Over 2022, the project has continued to gain momentum and is the second-highest velocity open source project in the CNCF. This velocity is reflected in the increasing adoption of OpenTelemetry by open source and commercial monitoring tools, as well as recent releases of the project focusing on improved metrics, traces, and log collection and processing,” Parker said. As OpenTelemetry continues to gain adoption, it will be increasingly important for monitoring and observability solutions to embrace it and offer their users an analysis tooling built with OpenTelemetry in mind. OpenTelemetry’s standard metadata for resources such as Kubernetes clusters, database transactions, message queues, and other common architectural components allows developers to easily transfer and share knowledge between monitoring tools, and make sense of complex systems.”
OpenTelemetry enables organizations to understand how their services and infrastructure interact and their current and past performance characteristics to more quickly develop and deploy the software that powers their business, and to rapidly predict, identify, isolate and fix issues, Morgan McLean, director of product management, Splunk, said.
OpenTelemetry makes this deep observability more potent and available to developers, DevOps personnel and organizations that create and run software, by the following, McLean, said:
- Streamlining observability — OpenTelemetry makes it easy to adopt observability tools and best practices, which few have been able to fully achieve until now, as it is so difficult to extract properly-shaped telemetry signals from existing services and infrastructure.
- Improving accessibility — By using OpenTelemetry’s huge and continually-maintained cohort of instrumentation, organizations and DevOps teams can capture essential data like spans, metrics, logs, metadata and more from almost any piece of infrastructure or any service.
- Creating greater consistency and standardization — OpenTelemetry provides a consistent set of semantic conventions across all data types, meaning that different signals from different sources can be correlated. For example, slow request performance can be traced to a specific service and its underlying infrastructure without any guessing or indirection.
- Fostering data ownership — OpenTelemetry can pre-process and route data to multiple destinations so an organization isn’t locked into a single vendor. In addition, developers can trivially create data or annotations that are specific to their business using native SDKs for every language, and OpenTelemetry can receive and transform data to and from most sources of metrics, traces, logs, and other telemetry. As a whole, this frees up data for people and the organizations who created it.
OpenTelemetry is also important for observability in that it merges and represents a shift by users from the reliance on proprietary observability tools to open source options while also helping to bridge the gaps between the needs of developers and operations team members.
“The current monitoring landscape is divided between monitoring solutions targeting infrastructure and platform engineers, and application observability solutions targeting application developers, Fabian Stäber, Grafana Labs senior engineering manager, said. “Infrastructure monitoring is often based on open standards like Prometheus’ OpenMetrics, while application monitoring is often based on proprietary tools. OpenTelemetry allows the application observability landscape to transition away from proprietary tools towards open standards, and this is leading to closer integration of traditional infrastructure monitoring with application monitoring,” Stäber said. “The core idea behind DevOps is to break the barrier between developers and operations. OpenTelemetry is playing a major role in breaking down those barriers.”
The Grafana Piece of the Equation
The need for Grafana to accommodate OpenTelemetry users is multifold. In the big-picture sense, OpenTelemetry and Grafana “share a common ‘big tent’ philosophy,” Stäber said. “Both are open source, integrate well with the broader ecosystem and are easily extensible for new usage scenarios,” he said. “Grafana users value OpenTelemetry’s open standards and tools for instrumenting applications, and find OpenTelemetry a natural fit for implementing application observability with Grafana.”
Collectively, Grafana engineers contribute to the OpenTelemetry collector to ensure it supports Grafana’s metrics database Mimir, Grafana’s logs database Loki and Grafana’s trace database Tempo, while the Grafana agent provides a native OpenTelemetry endpoint as well, Stäber explained. “For scenarios where you don’t want any middleware you can send OpenTelemetry data directly to the Grafana cloud’s OTLP endpoint,” Stäber said. “In a nutshell, Grafana provides native OpenTelemetry endpoints everywhere.”
As Richard Hartmann, director of community at Grafana Labs, noted, Grafana Labs “only publishes software we are using ourselves” and its support for OpenTelemetry is threefold: “For any of our projects and products, we are usually one of the most intense users. The second aspect is being where users and customers need to be — spearheading the observability space and buildings tools and technologies which will become relevant,” Hartmann said. “The third aspect is being where users and customers want to be, and given the industry focus on OpenTelemetry, it’s only natural for us to support it in all our projects and products. We also actively help shape the project only by both using and extending OTel can we be experts in the field our users and customers rightfully expect us to be.”
