Video Tutorial: Weave Scope, the Missing Piece to Show What You’ve Built with Docker

A few weeks ago on The New Stack, Weaveworks software engineers Peter Bourgon and David Kaltschmidt published a tutorial on Weave Scope, which provides for automatic detection and monitoring of every host, container and process in your infrastructure, and builds a map of their communication.

Recently, The New Stack Founder Alex Williams connected with Peter and the Weaveworks team to capture this video tutorial of Scope, and to discuss the development of this new product.

“What motivated us when we were developing Scope was the new context that a lot of developers and tech organizations are working in these days,” says Peter, “and that context is one that’s built on a lot of new concepts and technologies like Docker, like containers, like microservices. It’s a new way of thinking about writing software, and deploying software, and running software in production.”

It was their opinion that some necessary tools were missing. “In this new world,” Peter says, “you piece together your application from off-the-shelf data services, like Elasticsearch, or PostgreSQL, or Redis, you combine that with application services that you’ve written yourself in-house, and you use standard load-balancing and traffic-shaping layers at the top, and you end up with an infrastructure that makes sense when you’re building it, but maybe is a little bit tricky to keep track of over time.

We didn’t see that there was a good way to get a look at what you’ve actually built, and to keep an eye on it as time goes on. That’s the context of Scope.

“We wanted something that you could deploy just like an application service into your infrastructure. Crucially — with no configuration, no declaration — we want to be able to show you the infrastructure that you’ve built, let you explore it, let you see the services that you’ve deployed as they exist, click around, and be able to monitor the things that you’ve deployed — your containers — troubleshoot them when they go wrong, and understand them fundamentally.”

In the tutorial, Peter starts with the Host view, which shows a simple single-host set-up. He then moves to Containers by Image, which shows a typical three-layer stack he has deployed for the demo: database server at the bottom, application servers in the middle (where business logic would be), and a load-balancing layer at the top (such as Nginx or HAProxy).

He stresses that Scope has gathered this information from the infrastructure without having to change or instrument any of these applications. No matter the database, load balancers or application origin, “it’s all going to work fine.”

“Scope is looking at the host operating system and making all sorts of intelligent correlations,” says Peter, “to figure out the data communication paths without having to instrument any code.”

The Container view displays three instances of the database, showing how each is talking to two app servers, each of which are in contact with respective load balancers.

The Applications view displays the individual processes on each host. Clicking on the node representing an application server brings up information about that process, including the number of TCP connections, the process name, and statistics about the host on which it’s running. Weaveworks intends to expand this resource in the coming months to include bandwidth and deep packet inspection.

“The magic sauce of Scope is that all of this information can be semantically combined,” he says, showing how the data for two application servers is merged in the Containers by Image view.

“This way you can really drill down and expose problems, and see which instances are perhaps misbehaving, perhaps seeing more traffic,” says Peter, “all of this happening without any configuration, any declaration.”

Peter says that while they think this tool is critical in a production environment, he emphasizes that it’s also important to have in a typical developer workflow — the build-run-test loop.

“It works equally well in that environment,” says Peter, “and it helps new developers and old hands on your infrastructure to figure out what exactly they’re building, and how it’s interacting — and from the perspective of what’s actually happening on the network, rather than what you think is happening, or what you expect should be happening. This kind of transparency is very important when you’re building software, and something that we think is lacking in the current ecosystem.”

Weaveworks is a sponsor of The New Stack.