Work Commences on the Observability Query Language Standard
The year 2023 marked a significant stride in the realm of observability, with key developments such as the OpenTelemetry project reaching general availability and the merger of the Elastic Common Schema into OpenTelemetry.
However, amid these advancements, a critical challenge persisted — the lack of standardized query languages for observability databases.
A notable revelation came to light during my reporting for observability articles in 2023 and 2024. A new working group named the Observability Query Language Standard working group had emerged, operating under the Cloud Native Computing Foundation Technical Advisory Group for observability.
This group aims to address the existing incompatibility among various observability database query languages by working towards a common standard — a standard aligned with the principles of the OpenTelemetry project.
Databases have long been using different query languages for various observability aspects, such as Lucene for log querying and PROMQL for metrics. This diversity has resulted in a lack of interoperability and compatibility between systems, hindering seamless communication and integration.
The Observability Query Language Standard working group seeks to bridge this gap by establishing a unified standard for observability languages, fostering cohesion and collaboration within the observability ecosystem.
Dotan Horovits principal developer advocate of Logz.io and a Cloud Native Ambassador for CNCF, noted how in 2023, the OpenTelemetry project made remarkable progress by achieving general availability across all three pillars of observability.
This included the integration of the Elastic Common Schema into OpenTelemetry and the decision to sunset OpenCensus. Additionally, efforts were underway to formalize the Prometheus protocol as an IETF open standard (RFC2119), he said.
“One of the key challenges addressed by these initiatives is the lack of standardization in Continuous Integration/Continuous Deployment (CI/CD) pipelines. While many CI/CD tools emit telemetry, the absence of a standardized approach, specification, or semantic conventions poses a significant hurdle for effective monitoring using observability tools,” Horovits said. “Recognizing this challenge, a new initiative was launched under the CNCF in 2023 to extend OpenTelemetry’s capabilities to encompass the CI/CD use case.”
OpenTelemetry, already recognized for its effectiveness in DevOps scenarios for monitoring production systems and reducing Mean Time to Identify (MTTI) and Mean Time to Recover (MTTR), is poised to play a pivotal role in CI/CD observability, Horovits said. The initiative aims to leverage OpenTelemetry as an open unified specification, providing a foundation for standardized CI/CD telemetry. This approach not only ensures consistency but also facilitates the development of specialized tooling for collecting and transmitting telemetry in CI/CD pipelines.
The efforts to standardize observability query languages, as exemplified by the Observability Query Language Standard working group and the extension of OpenTelemetry to support CI/CD use cases, definitely show progress. These initiatives not only pave the way for enhanced interoperability but also lay the groundwork for a more streamlined and efficient observability ecosystem — one that is better equipped to meet the evolving needs of modern technology and development practices.