How To Detect, Map and Monitor Docker Containers with Weave Scope from Weaveworks

If you use Docker, you probably already know about Weave’s Docker network. Weave creates a virtual, software-defined network, or SDN, across every Docker host in your infrastructure. Each container gets its own IP, allowing you to design your application topology without changing your application’s behavior. Weave Run, a set of companion tools for service discovery and address management, also contributes to an easier development experience.

Weave Scope, a new product from Weaveworks, continues in the company’s tradition of making tools that reduce complexity. Scope automatically detects and monitors every host, container and process in your infrastructure, and builds a map of their communication. Scope presents an always-up-to-date view of your infrastructure in a modern web interface, allowing you to deeply understand, monitor and explore your complex application topology. Scope does this without requiring a single change to your code or configuration, and without requiring you to make declarations about your infrastructure that will immediately become out-of-date and stale.

In this tutorial we’ll demonstrate how to deploy Scope alongside a typical three-tier web application stack, consisting of a pool of data services, a set of custom application servers and a load-balancing layer. You’ll learn how Scope works out-of-the-box with no configuration, and how it can immediately start returning useful intelligence.

While Weave Scope works brilliantly with Docker and the Weave network, neither is required. Scope can be deployed in any infrastructure, and works well in all cloud and bare-metal environments.

Getting Started

We’ll get started by deploying an example application using Docker Compose. For the demo we’ll work on a single host, but remember that Weave works equally well on multiple hosts, or even across data centers and cloud providers.

First, let’s install Docker and Docker Compose:

Now, let’s use Docker Compose to launch all of the components:

Take a look at the output of Docker ps, to make sure that everything is running:

Verify everything is working by curling one of the tns_lb instances. By default, they’re listening on ports 8001 and 8002:

Great! In just a few commands, we’ve deployed a three-tier application stack. Now, we want to get a picture of our infrastructure, so let’s install and launch Weave Scope:

Now, you can point your web browser to localhost:4040. (On a Mac, that might be `boot2docker ip’:4040.)

Seeing Your Infrastructure

Now that Scope is up and running, it’s time to see your infrastructure. The Scope user interface is loaded in your browser and gives you an immediate overview of your network.

A simple map of your containers

Each circle of this map represents a container in your network. The circles that are connected by a line are containers communicating with each other. Use your mouse to explore the network. When you hover over a container or a connection Scope highlights the connected containers and helps you understand the topology quicker. This is especially helpful for infrastructures with lots of containers.

Let’s say we’re looking for more information about one of the database (db) containers. Click on a container with the label tns_db1_1 to find more information about it.

Details and metrics of the DB container

The panel on the right shows basic information about the container, like the image and process name, as well as network metrics, like ingress/egress byte rates and number of TCP connections.

More Advanced Use Cases

Scope works great in topologies of any size. When you get too many containers on the screen at once, you can use Scope’s grouping features to make things easier to understand. For example, in the containers view, you can click the grouping button to merge all the containers that are running the same container image together.

Grouped container view

In our stack, that means all of the db, app, and lb instances get merged together into single nodes, even when they exist across hosts. The result is a logical, rather than physical, lens on the infrastructure — something that application developers should intuitively understand. And detailed statistics are merged, too.

These features are great for application developers, but those with a more operations-oriented background will feel right at home in the hosts view. Here, we see the physical layout of the infrastructure, as you might find in a tool like Ganglia or Squid.

The Hosts view

A click on applications changes the map again, showing all processes communicating in your network. This is the most granular view of Scope.

The Applications view

Scope collects information in a flexible data model, so it’s possible to visualize nearly any possible transformation of your network.

Taking Integration a Step Further: The API

Now that you have seen how to easy it was to get visibility on an application network, it is time to integrate it into your system. There are multiple ways for you to set up Weave Scope for your environment. It is open to connect with other platforms and data sources. Out-of-the-box the user interface gives you a good overview of your application network. Use it as a starting point to troubleshoot problems or simply explore the topology. Beyond that, you can directly query its API and get a feed of container connections and metrics. This feed can act as an input to your own monitoring solution and provide topology information.

Let’s use the API to get to the same information about the database container from above (remember that this works on any host that runs Scope):

First, we see which topologies are available:

We want the container topology. To get all containers we query:

Here we see that the container topology is represented as a node collection with adjacency lists. In the example we see that node “123…” has a connection to node “456…”. Iterating over all nodes allows you to derive your own topology.

Let’s assume that node “123…” is already the database container we are looking for. To get more information on that container we can query:

Try it out to see what information is available in the response.

More Visibility

Use Weave Scope now to automatically generate maps and feeds to enable visibility for your Docker apps. It is especially easy to add when you are already using Weave Net to set up your Docker cluster. There is a collection of helpful guides on how to get started with Weave.

Weave Scope is open source and continuously being developed. The code is available on GitHub and is easy to set up if you like to contribute. The probes and the backend are written in Go. The user interface is a React single-page application. Follow Weaveworks on Twitter to stay in the know about latest releases. You can expect more container metrics and features to be added soon.

He is a distributed systems and infrastructure engineer, with a focus on coördination-free systems and observability. I'm the creator of Go kit, a toolkit for microservices in Go; and OK Log, a coördination-free log management system.

Read more from Peter Bourgon

He creates data visualizations and analytics. Been working on the frontend side of things for more than a decade. Extensive React/D3/Angular experience. Occasional dabblings in Python/Ruby/Go/Java/Swift. Domain knowledge of networking, VoIP, WebRTC, distributed systems monitoring.

Read more from David Kaltschmidt