Meet Spinnaker: The Ultimate Multicloud Deployment Tool

Video streaming service Netflix certainly knows a thing or two about web-scale deployments. Since the time of the creation of Chaos Monkey in 2011, the company has been regularly contributing to the open source ecosystem.

In 2012, Netflix developed Asgard to automate Amazon Web Services’ deployments involving Amazon Machine Images (AMIs), Elastic Cloud Compute (EC2) instances, auto scaling groups, launch configurations, and elastic load balancers. Netflix built Asgard to expose a consistent deployment experience for internal teams targeting AWS. It abstracted the underpinnings of EC2 infrastructure through a standard nomenclature and a set of APIs.

By 2015, Asgard developers started to move the software beyond AWS, to support Google Cloud and Microsoft Azure as well. The multicloud avatar of Asgard evolved into a brand new tool called Spinnaker. In its current version, Spinnaker supports a variety of deployment targets including mainstream cloud and container platforms, including OpenStack, DC/OS, Kubernetes, and Cloud Foundry.

Irrespective of the target environment, the deployment experience is always consistent, offering the following capabilities:

• Enable repeatable automated deployments captured as flexible pipelines and configurable pipeline stages.
• Provide a global view across all the environments that an application passes through in its deployment pipeline.
• Offer programmatic configuration and execution via a consistent and reliable API.
• Be easy to configure, maintain, and extend.
• Be operationally resilient.
• Provide the existing benefits of Asgard without a migration.

Spinnaker is not just about automating multicloud deployments. It emphasizes continuous deployment through powerful pipeline capabilities. Spinnaker can package application artifacts such as binaries and container images for each target environment. For example, it can build an AMI to deploy in EC2 while packaging the same application as a set of Pods that can run in Kubernetes. Spinnaker is tightly integrated with existing tools such as Jenkins, Docker Hub, Travis CI, and Circle CI. Operations teams get ultimate control in managing the deployments across staging and production environments.

Spinnaker encourages the use of immutable infrastructure. Operations teams will never have to use SSH to manage a VM deployed through Spinnaker. If the health of a VM is not green, which means the health check has failed, Spinnaker automatically spins up a new VM, and stops routing traffic to the unhealthy VM.

The best thing about using Spinnaker is that the teams will never have to manage individual resources of each target environment. They can register target platforms with Spinnaker and use the interface to define deployment strategies and pipeline stages. Organizations dealing with contemporary, cloud-native applications will find Spinnaker highly effective for managing end-to-end application lifecycle.

Setting up Spinnaker

Spinnaker can be easily setup in AWS, Azure, and GCP through templates. It’s available as a CloudFormation template in AWS, Azure Resource Manager template in Azure, and as a Deployment Manager template in GCP.

If you don’t want to use existing templates, you can use a tool called Halyard to setup and configure Spinnaker on Ubuntu 14.04. Halyard simplifies the installation by abstracting the details involved in installing Spinnaker.

In its current form, Spinnaker is not secure enough to be exposed on the public Internet. The integration with LDAP and support for standard authentication mechanisms is still in works. When deployed in public cloud environments, it is best accessed through SSH-based port forwarding or VPN.

Since Spinnaker is built as a microservices application based on a set of Docker containers, it can be deployed in Docker Swarm or Kubernetes clusters. We will explore this deployment model in one of the upcoming articles.

Managing Deployments with Spinnaker

Think of Spinnaker as the middleware that deploys your application artifacts in the target deployment. At a high level, it is a tool that connects the dot between the two — source code and production environments.

You start by registering the target deployment platforms — VMs, PaaS, or container orchestration engines. You will then define the pipeline that may have multiple stages to deploy the application in one or more target deployments.

Once the target deployment environment is registered with Spinnaker, you create the following resources that map to the exact counterparts in the target environment.

• Load Balancer — This will define the endpoint where the application would eventually become available. In a Kubernetes world, this translates to a Service. In AWS, it may be an Elastic Load Balancer.
• Security Group — This resource defines a set of firewall rules that are translated into network policies in respective environments.
• Applications — Applications are logical groupings of clusters.
• Cluster — A Spinnaker cluster defines a set of compute resources associated with a deployment. It typically represents a distinct, isolated environment such as development, staging, testing, or production. Each cluster may have one or more server groups. The cluster can be configured to set the desired state of each server group.
• Server Group — A server group is an elastic deployment unit consisting of fine-grained compute resources such as VMs or Pods exposed via a single load balancer, and controlled by a security group. Each server group has a minimum and a maximum number of compute resources that define the desired state. In AWS, the server group is mapped to an Auto Scaling Group (ASG). In Kubernetes, it translates to a Replica Set.

Though the terminology seems to be highly influenced by EC2, it is used to configure PaaS environments such as Google App Engine or Cloud Foundry, and container orchestration tools such as Kubernetes and DC/OS.

Spinnaker uses the above-defined resources to manage application deployments. Once the registration and configuration are done, it is possible to achieve various patterns such as blue/green deployment and canary deployments.

After configuring the target environments, DevOps teams create pipelines to deploy artifacts into these environments.

Pipelines — Pipelines are key to defining the deployment strategies in Spinnaker. They are made of a sequence of actions, known as stages. It is possible to pass parameters from stage to stage along the pipeline. Pipelines are started manually, or they can be configured to be started by automatic triggering events, such as a Jenkins job completing, a new Docker image appearing in your registry, a cron schedule, or at the completion of a stage in another pipeline. Pipelines emit notifications to interested parties at various points during pipeline execution (such as on pipeline start/complete/fail) via email, SMS or HipChat.

Stages — A Stage in Spinnaker is an action that forms the fundamental building block of the pipeline. DevOps teams can sequence stages in a Pipeline in any order, though some stage sequences may be more common than others. Spinnaker provides a number of stages such as Deploy, Resize, Disable, Manual Judgment, and many more. For example, a deploy stage can be configured to target one of the server groups while another stage can be used to manually trigger the rolling upgrade of the latest version in production cluster.

Strategies — Strategies in Spinnaker define how a specific deployment takes place. The Highlander deployment strategy will push the new version into a server group, and when it becomes healthy, it automatically terminates the server group running the previous version. The Red/Black strategy provides a mechanism to perform blue/green deployment by enabling both the versions to run side by side. The rolling push performs a rolling upgrade by gracefully deleting the previous version and replacing it with the new version.

It is also possible to define custom strategies that are aligned with specific deployment patterns defined by the DevOps teams.

Spinnaker Community

The Spinnaker user base is steadily growing outside of Netflix. There are some commercial implementations and professional service offerings available from companies such as Armory, Kenzan, and OpsMx.

With sparse documentation and lack of tutorials, not many users are adopting Spinnaker. If you are targeting multicloud deployments or deploying cloud-native applications, Spinnaker may be the tool that you are missing.

In one of the upcoming articles, I will walk you through the process of setting up Spinnaker in a local Kubernetes cluster and using it to deploy containerized applications. Stay tuned.

Additional editions of this series on learning Spinnaker will be found here.