How has the recent turmoil within the OpenAI offices changed your plans to use GPT in a business process or product in 2024?
Increased uncertainty means we are more likely to evaluate alternative AI chatbots and LLMs.
No change in plans, though we will keep an eye on the situation.
With Sam Altman back in charge, we are more likely to go all-in with GPT and LLMs.
What recent turmoil?
Containers / Observability

How to Monitor Containers with cAdvisor and Wavefront

Apr 19th, 2016 1:20pm by
Featued image for: How to Monitor Containers with cAdvisor and Wavefront

Evan Pease
Evan Pease is an engineer focused on real-time analytics, micro-service architectures, and big data. His past work was focused on search, log analytics, and e-commerce. He is a graduate of Syracuse University and has lived in Southern California since 2008. Evan works for Wavefront as an engineer in the field helping companies instrument their systems for operational and business intelligence.

With Docker adoption reportedly doubling last year alone, it should be no surprise that the need for container monitoring is increasingly pressing. While containers have numerous benefits and can help simplify many tasks, they also introduce an extra layer of granularity and interdependency. This is why it is important to have visibility into running containers.

Surprisingly, until somewhat recently, there were no straightforward methods for monitoring resource metrics of Docker containers.


The Docker Stats API, which was introduced with Docker 1.5, introduced the ability to stream resource metrics from containers such as CPU and memory usage. And there has also been numerous other open source efforts to address container monitoring, including Google’s cAdvisor project. The tool, cAdvisor, is “a running daemon that collects, aggregates, processes, and exports information about running containers. Specifically, for each container it keeps resource isolation parameters, historical resource usage, histograms of complete historical resource usage and network statistics. This data is exported by container and machine-wide,” according to the project’s GitHub page,

Running cAdvisor

One of the best attributes of cAdvisor is how easy it is to run. Since it runs in a container itself, you can spin it up just like any other Docker container with predictable and consistent results. It is also packaged with several “storage drivers” out of the box. When storage driver parameters are passed, it will automatically export resource metrics to the storage driver. For example, the command below will create an instance of cAdvisor on any Docker host, and export resource metrics to a StatsD server.

Viewing Container Metrics in Wavefront

Recently, we consolidated several services we had running for demos and related functions onto new servers. When we performed this task, we decided that we wanted to redeploy using Docker. By isolating each service in a container, we’re able to simplify the deployment and scaling process in the future. Using cAdvisor, we’re also able to isolate resource metrics for each container. This is especially useful when you have multiple containers running on the same host – which we have, in this case.

Using the command shown above, cAdvisor immediately starts emitting metrics to a StatsD server. The metrics are created with the following naming scheme:

Storage_driver_db is an arbitrary name for your docker host that is passed to the run command for cAdvisor. In this case, we used docker001


CAdvisor also provides storage metrics, but that can be trickier to monitor if you’ve mounted volumes to your containers.

Comparing Container Resource Usage

One of the services we moved was a Twitter streaming application. The Twitter streaming application (“twittermentionstream” in the chart below) was only streaming tweets from a handful of support accounts. The total rate of tweets being ingested was less than 0.2 per second even during normal peak Twitter hours. Still, it was the largest user of CPU amongst our containers:


This chart is showing the percentage of CPU being used by each container.

Shortly after the migration occurred, we added several Twitter accounts to the app. This increased the total rate of tweets to over eight tweets per second, a 40-fold increase. Not surprisingly, a 40-times increase of tweet ingestion rate does have a large impact on resource usage.


Container CPU Load after increasing tweet ingestion rate by 40 times.

Getting this information required zero instrumentation other than running cAdvisor. If you isolate your apps in containers, you can see exactly what the impact of your changes are in CPU, memory, and throughput. We’re only looking at CPU here, but other metrics displayed similar behavior.

Capacity Planning Using Container Metrics

Let’s say we wanted to add some more Twitter accounts to the streaming app. We know from our previous chart that the Twitter streaming app is using close to 40 percent of the server’s CPU at ~8 tweets per second. We want to measure how much of a correlation there is between CPU usage and Tweet rate.

Correlation between CPU Usage and Tweet ingestion rate.

Correlation between CPU Usage and tweet ingestion rate.

This chart shows a linear correlation between tweet ingestion rate and CPU Usage. We could use this line to predict how many tweets this server could handle before reaching 100 percent CPU usage.

In the old days, exploring data relationships meant having a solid grasp of statistics and perhaps a little code. In Wavefront, it takes two clicks (see our Data Exploration example).


Container adoption is a mega-trend that we don’t see slowing down anytime soon. We will continue to evaluate and blog about developments and trends as more people (and ourselves) ask how to get better visibility into containers. Google’s cAdvisor provides an extremely fast, easy, and effective method to get resource metrics from your containers into Wavefront.

Docker is a sponsor of The New Stack.

Feature Image via Pixabay.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack, Docker.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.