30 Pull Requests Later, Prometheus Memory Use Is Cut in Half
Prometheus memory consumption is one of the numerous ways monitoring for observability can crash your system.
Bryan Boreham, distinguished engineer at Grafana Labs, detailed how he experimented with various methods to ultimately reduce the memory usage of Prometheus during his talk at KubeCon+CloudNativeCon. The title of his presentation, “How Prometheus Halved Its Memory Usage” was about his research into Prometheus, in particular the memory consumption of labels, which revealed ways to reduce memory consumption.
According to Prometheus documentation, labels are used to differentiate the characteristics of the thing that is being measured:
- api_http_requests_total – differentiate request types: operation=”create|update|delete”
- api_request_duration_seconds – differentiate request stages: stage=”extract|transform|load”
Boreham’s work also was in the spirit of open source, showing how contributions can lead to real results. After making 30 pull requests to the monitoring system project with over 2,500 lines of code changes during the course of two years, Boreham’s work helped the latest version of Prometheus to use half the amount of previous versions.
“It was a long road, but ultimately very satisfying,” Boreham told The New Stack after KubeCon+CloudNativeCon. “There are hundreds of thousands of Prometheus servers running, and by bringing down the memory requirements, we reduce the cost of running them and their carbon footprint.”
Go’s Memory Profiler
The Go programming language has in its runtime a built-in profiler that can offer a memory-consumption breakdown of Prometheus’ memory — as well as CPU — usage, Boreham explained in his talk. It provides what is called a flame graph view for the visualization. The width of the blocks is the proportion of how much memory is being used. The top of the chart above shows 100% and a total of 6.7 GB of memory consumption, Boreham said.
The so-called sawtooth effect comes into play in the chart. This is when garbage builds up over time, then is collected, so the memory falls sharply, then it builds up again, Boreham told The New Stack. “That’s the sawtooth,” Boreham said.
“The Go memory profiler reports the memory usage as at the last garbage collection, so, you never see garbage in this picture. A lot of people think, ‘oh, it’s probably mostly garbage and I don’t need to think about it,’” Boreham said. “But this is never garbage when you look at a profile from Go. This is the bottom of the sawtooth and what could not be discarded.”
The process of making memory consumption smaller “starts with saying, ‘well, what is it that makes it so big?’” Boreham said. “But the winner overall — and this was about two years ago — pretty much a third of all the memory inside Prometheus was taken up in labels (31% in this diagram),” he said.
The Problem with Prometheus Labels
Every series in Prometheus is uniquely identified by this name-value pair set, Boreham explained. “If you have another series which is kind of related, and the only difference there is between the method, you actually get a whole new set of strings, and on and on and on,” Boreham said. “So, you look at this and ‘you say, well, that’s dumb. I just have one copy of the strings,’ but it’s not as simple as that.”
The above diagram shows the kind of data structure inside. The slice header that’s pointing to all the labels is 24 bytes and every string has a string header, which is 16 bytes. “It’s a pointer to the contents and a length,” Boreham said. “And if you add all them up, it turns out that all of these pointers in the data structure are way bigger than the strings themselves.”
With Prometheus PR 10991, Boreham then put all the strings in a single string and encoded them with a length:
“It took a year and 2,500 lines of code changed because there was a huge amount of code that just assumed that it knew what that data structure was like,” Boreham said.
With Prometheus 2.74.2 — while the previous version would crash at 17 GB of memory consumption — Boreham ran 2.47.2 with a memory consumption of 13.1 GB without incident:
While a handling of samples and native histograms features had been added to 2.47.2, “they didn’t really use up all the memory,” Boreham said and while memory consumption was significantly reduced, it was not yet exactly at the 50% mark yet.
Boreham then found and fixed a bug in 2.39: the transaction isolation ring, which “used to get enormous under certain conditions,” Boreham said. “But I did the math, and memory consumption was still not cut completely in half”:
The bug fix reduced the memory consumption to 10 GB:
Boreham continued to study the Go profiler to target the biggest memory-consumption culprit.
“You pick the biggest number, you work on that, find some inefficiency in it if you can and do it again. And now what was the second biggest number is now the biggest number,” Boreham said. “Numbers that were really not that big to begin with are now big numbers. So it’s a nice self-reinforcing process.”
This is 2.47 plus all of the PRs in the diagrams above for a total of 8.6 GB of memory consumption, which is nearly at the 50% reduction mark:
As Boreham explained, There is a parameter in the Go runtime called GoGC, and it defaults to 100. The size the sawtooth grows to is 100% of the size of the bottom of the sawtooth, which is 7 GB. “For those of you who have 100 GB Prometheus, it’s growing by 50 GB, but for housekeeping purposes, you don’t need 50 GB of garbage to run an effective heap,” Boreham said. “You can tune that number — it’s an environment variable you can set and it will grow to whatever percentage you set it to over what it went down to as the minimum and it will garbage collect a bit faster.”
Prometheus has since been running with a memory of 8 GB — the 50% mark of memory consumption had been reached. Boreham had reached his goal:
Prometheus users will inevitably appreciate the lower memory consumption, while most will probably not be that interested in how it was achieved.
But for those who like to give back to the open source community, Boreham’s travail shows how, with a lot of work and patience, it is possible to make pull requests (PRs) that can have a real impact. While Boreham’s work may look simple in hindsight, it obviously was not — which is so often the case in math and scientific research. Prometheus, and open source projects in general, offer users and those interested in computing an opportunity to make a difference.
“Really it’s a labor of love,” Boreham told The New Stack. “Making computer programs smaller and faster is something of an obsession for me, so it’s great to be able to put that to use with such a popular and widespread project. And, at Grafana Labs, ‘open source is in our DNA.’”
The full video of Boreham KubeCon presentation can presentation can be enjoyed here: