TNS
VOXPOP
Where are you using WebAssembly?
Wasm promises to let developers build once and run anywhere. Are you using it yet?
At work, for production apps
0%
At work, but not for production apps
0%
I don’t use WebAssembly but expect to when the technology matures
0%
I have no plans to use WebAssembly
0%
No plans and I get mad whenever I see the buzzword
0%
Data / Observability / Software Development

The Combined Power of Chronosphere and Calyptia

This merger is particularly significant in its potential to streamline developer workflows and optimize data pipeline management, underpinned by the robust capabilities of Fluent Bit and OpenTelemetry.
Feb 6th, 2024 5:00am by
Featued image for: The Combined Power of Chronosphere and Calyptia

Chronosphere’s acquisition of Calyptia is a shrewd strategic move that adds strong data pipeline management capabilities to Chronosphere’s cloud native observability platform. Both Chronosphere and Calyptia follow a federated management paradigm that includes a central SaaS-based control plane that remains out of stream, in combination with multiple data planes that can be located close to the customer’s application clusters.

This acquisition is impressive alone based on the fact that Chronosphere found a like-minded company focusing on the highest possible degree of scalability. Like Chronosphere, Calyptia is the result of a seasoned founder team seeking to figure out a solution to the rapid growth of cloud native infrastructure in terms of quantity, complexity, and constant change.

The Chronosphere observability platform resulted from the rich experience the founder team had made at Uber, while Calyptia launched in 2020 with the idea of providing commercial services around Fluentd and Fluent Bit, to open source data collector projects the company helped found. But not only do both companies follow very similar philosophies in their approach toward establishing cloud native observability, but Calyptia has the final puzzle piece Chronosphere needed to become a truly formidable contender in the cloud native observability space: a platform for data pipeline management.

Why Developers Should Care

After the introduction of Lens, Chronosphere’s capability to automatically surface telemetry data relevant to any specific developer, the acquisition of Calyptia will allow Chronosphere to add more developer goodies to its platform. This radically developer-centric approach is so impressive as all of the traditional observability platforms have tried this same strategy without much success, creating space for Chronosphere in the market. The key strategic value propositions of this acquisition are:

  1. Enhancing Developer Productivity: Calyptia Core, built on the open source Fluent Bit data collection agent, is tailored to streamline developer workflows, thereby boosting productivity in observability tasks. Developers can simply connect telemetry data sources with transformation steps and export targets to optimize observability for their application via a drag-and-drop interface and without having to worry about complex agent management procedures.

Examples:

Automated Log Aggregation: A developer working on a microservices architecture needs to aggregate logs from various services for debugging. Fluent Bit can be configured to automatically collect logs from all services and forward them to Calyptia Core. This eliminates the need for manual log collection, allowing the developer to focus on analyzing the aggregated logs to quickly identify and resolve issues.

Streamlined Monitoring for Kubernetes: In a Kubernetes environment, a developer can use Fluent Bit as a DaemonSet to collect metrics and logs from each node. Calyptia Core can then process this data, applying filters and transformations before sending it to a monitoring solution like Prometheus or Grafana. This setup simplifies the monitoring pipeline and reduces the time spent on configuring and managing multiple monitoring agents. 

  1. Optimizing Observability Costs: A key focus of Calyptia Core is to reduce the costs associated with observability. This is achieved through more efficient data management, which directly benefits Chronosphere’s clientele by simplifying telemetry data ingestion and management.

Examples:

Reducing Data Ingestion Costs: Calyptia Core’s First Mile Observability strategy optimizes observability costs by focusing on efficient data management at the source. This approach reduces backend and egress costs, particularly beneficial for organizations dealing with IoT and edge applications spread across disparate public and private infrastructure. By filtering out noisy data before it is ingested into systems like Splunk, organizations can reduce costs and improve data quality, with some of Calyptia’s customers reporting substantial savings on their Splunk bills.

Additionally, by routing data to more cost-effective storage locations, organizations can significantly reduce the costs associated with data growth. This strategy enables organizations to gain immediate insights, improve data quality, and significantly reduce their observability costs.

Routing Non-Essential Data to Cheaper Storage: Calyptia Core, powered by Fluent Bit, allows organizations to redirect non-essential data to cheaper storage. This can result in substantial savings, especially when using costly storage and analysis applications.

  1. Boosting Performance and Security: By leveraging Calyptia’s technologies, Chronosphere aims to enhance the performance of its observability platforms and fortify application security, addressing two critical concerns in modern computing environments. 

Examples:

Data Security: Calyptia Core provides features to identify sensitive data and redact, remove, or hash it midstream before it is stored. This capability can enhance the security of Chronosphere’s observability platforms by ensuring sensitive data is not unnecessarily exposed or stored.

Performance Measurement: Calyptia emphasizes the importance of effective performance measurement, which can help Chronosphere to establish performance benchmarks and goals, ensuring the quality of its application improves over time. By tracking metrics over time, teams can detect changes in behavior that may point to an underlying issue, thereby enhancing the performance of the observability platforms. 

  1. Addressing Microservices Architecture Challenges: The growing trend toward microservices architecture, characterized by loosely coupled services owned by different teams and residing in varied cloud environments, poses unique challenges in observability, particularly in cost management and risk mitigation. The Chronosphere-Calyptia combination aims to offer solutions to these emerging challenges.

Examples:

Enhanced Observability: Microservices architecture can be complex and difficult to monitor due to the distributed nature of services. Chronosphere’s observability platform, enhanced by Calyptia’s log processing capabilities, can provide comprehensive visibility into the system. This allows for efficient troubleshooting and debugging, which are crucial in a microservices environment where services are loosely coupled and owned by different teams.

Cost Management: The cost of observability can be a significant challenge in a microservices environment due to the high volume of data generated. Calyptia’s platform can reduce the volume of data at the source using filters, making it more cost-effective to analyze. This can help Chronosphere manage the cost of observability in a microservices architecture.

Pipelines Simplified: Low Code/No-Code for Data Collection, Processing and Export

Calyptia Core simplifies data pipeline construction and management in cloud native environments through a low-code/no-code approach. Its drag-and-drop interface allows developers to build data pipelines visually, reducing complexity and making it accessible to those with limited coding expertise. With over 100 integrations, Calyptia Core standardizes data collection across various applications and environments. It also offers built-in data processing transformations for data enrichment, aggregation, filtering, and reduction. Developers can test their pipelines in a sandbox environment before deployment, reducing risks.

Calyptia leverages Fluent Bit’s pluggable architecture, eliminating the need for SDKs and APIs for custom integration code. Fluent Bit’s input, filter, and output plugins collect, process, and route data respectively. It also integrates well with Kubernetes and Prometheus, enriching logs with Kubernetes metadata and integrating with Prometheus Grafana dashboards.

WebAssembly Integration

The Fluent Bit WebAssembly (WASM) plugin introduces a significant shift in data processing capabilities within observability frameworks, moving beyond traditional methods like Lua scripting or regular expressions. WASM allows developers to use comprehensive programming languages like C, Golang, and Rust for custom data processing tasks, compiled into a portable and efficient WASM binary for high performance and low overhead. This enables sophisticated log processing routines in complex cloud native environments, where logs can be filtered, enriched, and classified. For example, a routine could parse log entries, extract key information, and enrich the logs with additional context from an external API.

The WASM plugin can also leverage advanced technologies like large language models (LLMs) for tasks such as sentiment analysis, trend prediction, or anomaly detection. For instance, an LLM could categorize logs based on their urgency or potential impact on system performance, enabling a more focused and efficient observability strategy.

Additionally, the WASM plugin can facilitate sophisticated routing mechanisms, allowing log data to be routed to different destinations based on the analysis performed. In essence, the Fluent Bit WASM plugin enhances data observability and analysis capabilities in complex and dynamic cloud native environments by allowing the use of full-fledged programming languages and the integration of state-of-the-art AI models.

Small Footprint

Fluent Bit’s small footprint is a key feature for data collection efficiency in cloud native microservices architectures. Its implementation entirely in C results in a dramatically reduced memory footprint, operating with just around 1MB compared to FluentD’s 40MB and an average OpenTelemetry collector’s 50MB. This minimal resource usage is crucial in distributed environments with numerous microservices, where it leads to cost reductions and performance improvements.

Fluent Bit’s lean architecture allows for widespread deployment across application containers without significant resource consumption, especially important in Kubernetes environments. Additionally, the compiled C code enhances Fluent Bit’s data processing speed, essential for real-time observability and monitoring, making it a highly suitable solution for modern cloud native architectures.

In-Stream Processing

Fluent Bit’s in-stream processing, particularly through tail reading, enhances real-time data handling in cloud native environments by continuously monitoring and capturing changes in log files. This capability is beneficial in dynamic cloud native architectures, where the volume and velocity of data can be substantial.

Fluent Bit’s in-stream processing reduces latency and resource consumption, supports advanced operations such as applying new data schemata, parsing logs, and enriching data streams. For example, it can dynamically parse and structure log data, convert it from unstructured to structured formats, and enrich the data stream by applying tags or integrating additional metadata. This process is crucial for context-aware monitoring and analysis, providing deeper insight into the behavior and performance of applications and services.

In essence, Fluent Bit’s in-stream processing and tail reading capabilities enable efficient, real-time observability in cloud native environments, ensuring immediate data availability and facilitating advanced data manipulation.

 Fluent Bit and OpenTelemetry: Better Together

OpenTelemetry instruments the application code and receives telemetry data. It can then simply export that data to different backends for tracing, logging, and metrics. Alternatively, OpenTelemetry can export the data to Fluent Bit to feed a data pipeline in Calyptia Core. The data pipeline filters, enriches and transforms the data for optimal consumption by monitoring systems, analytics tools, and storage platforms.

The integration of Fluent Bit and OpenTelemetry offers a comprehensive data collection and analysis solution for modern observability, combining Fluent Bit’s efficiency with OpenTelemetry’s extensive instrumentation capabilities. Fluent Bit, with its lightweight footprint, can be deployed as a sidecar container in each pod of a Kubernetes cluster, using the OpenTelemetry input plugin to collect telemetry data directly from the application and infrastructure for real-time data collection and processing.

Fluent Bit can also collect logs and metrics from multiple microservices and send this data to observability tools like Jaeger and AWS X-Ray using the OpenTelemetry output plugin, enabling a unified view of distributed systems and aiding in tracing requests through microservices. Furthermore, Fluent Bit’s customizable nature allows for the collection of targeted telemetry data, which can be enriched, processed, and exported to OpenTelemetry-compatible backends for detailed analysis and visualization, providing bespoke insights into applications. In essence, the combination of Fluent Bit and OpenTelemetry provides a flexible, efficient, and comprehensive observability framework for diverse monitoring needs.

Looking at the number of GitHub stars we can observe that OpenTelemetry, FluentD and Fluent Bit are currently growing at roughly the same pace. While the FluentD GitHub repo started in 2011, Fluent Bit and OpenTelemetry were “born” a lot later.

Conclusion: A New Era in Cloud Native Observability

The acquisition of Calyptia by Chronosphere marks a pivotal advancement in cloud native observability. This strategic alignment not only enhances Chronosphere’s observability platform but also signifies a commitment to scalability and developer-centric solutions. The integration of Calyptia’s efficient data management tools with Chronosphere’s existing capabilities paves the way for more intuitive, cost-effective, and performance-oriented observability in complex cloud native ecosystems.

This merger is particularly significant in its potential to streamline developer workflows and optimize data pipeline management, underpinned by the robust capabilities of Fluent Bit and OpenTelemetry. The future of observability in the cloud native domain looks promising, with this partnership poised to deliver more refined, intelligent, and adaptive solutions, addressing the intricate demands of modern application development and operations.

In short, Chronosphere’s acquisition of Calyptia is not just a merger of two companies but a fusion of shared visions and technological prowess, setting a new benchmark in the field of cloud native observability.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Bit.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.