LogDNA Moves to Rust-Based Agents for Speedier Logging
When we first spoke to logging management company LogDNA in 2016, the company was just pivoting from e-commerce to logging, with a focus on making logging more predictive with machine learning. Nowadays, LogDNA has turned its eye toward logging for Kubernetes on “any infrastructure, any architecture, any language.” To speed things along, this week the company has launched a number of performance and usability updates, most notably the move to writing the agents in Rust, rather than using a Node.js/JavaScript approach. The new feature set helps developers to more easily query, filter and gain insight from their log data and more quickly troubleshoot and fix problems, the company said in a statement.
Peter Cho, vice president of product management at LogDNA, explained that machine learning simply wasn’t the core problem that needed to be tackled. Instead, providing logging at scale for Kubernetes was a much more pressing issue.
“We realized that the problem that our customers were seeing at the time was more about scalability. A lot of customers were coming to us, using other tools and basically saying our stuff is crawling to a halt. If we go from sending a hundred megs to a hundred gigs or a terabyte, things are just really slowing down,” said Cho. “We ended up pivoting away from machine learning — not to say that we won’t go into it in the future, but we’ve definitely been more focused on scalability and making a good Kubernetes use case and experience.”
Beyond Kubernetes, Cho also said that LogDNA provides logging for other systems at scale, such as when with Internet of Things (IOT) devices and edge devices. In its efforts to tackle this issue of logging at scale, LogDNA realized that using a more system-level language might help in its effort, and this latest release comes with a new, smaller and more performant version of its agent written in Rust. Beyond being rewritten in Rust, the company says that the new agent “leverages the Linux kernel to monitor log files and directories for changes, freeing up CPU utilization, improving stability and accuracy, and removing duplicate lines with symbolic linked log files.”
“CPU consumption, RAM consumption, and overall footprint have all been greatly minimized from making this shift from the Node.js-based agent to a Rust-based agent. We were running into an issue where with a certain level of scale customers like IBM, as well as others, if the agent is using up too much system resources, it’s going to miss things,” said Cho. “We ended up realizing that we have to use a more systems-level language like Rust so that it could be much more performant, stay on top of fast-rotating logs, and other quirks that come with high production Kubernetes use cases, which is why we rewrote it from scratch.”
In addition to its new, more performant agent, LogDNA now provides hourly archiving, making it easier for users to get just the data they need, rather than an entire day’s worth, as well as the ability to extract and aggregate fields, which Cho explained as allowing users to “look through your current retention window of log lines, establish a pattern, and then aggregate those fields for whatever kind of data work that you needed to do.”
Finally, LogDNA also added custom webhooks, bringing users the ability to more easily integrate with non-supported third-party services, such as JIRA or Microsoft Teams, where they might automatically trigger a task or send a message, for example.
As for what’s next for LogDNA, Cho said that the company is putting its focus on the developer experience, explaining that complexity comes quickly for today’s startups, with even small startups dealing with larger amounts of data than giants like Cisco might have 25 years ago.
“I’m not going to say that we’re not going to go down the other paths of things like machine learning and whatnot, but I will say, at least in the near to mid-term, we’re really focused on customer experience. I think the problems that people are facing today, they’re not really focused on things like intelligence or new age things. They’re focused on dealing with the chaos of legacy systems while introducing things like Kubernetes,” said Cho.
LogDNA is a sponsor of The New Stack.
Feature image by Mildly Useful on Unsplash.
