Ditch the Pitfalls of Layer 2 Networks by Routing on the Host
If you’re operating on the IT side, you’re likely all too familiar with troubleshooting an unexpected outage caused by what should be a simple upgrade to a switch, or bringing a new server onto the network. While modern data center design for private and public clouds has been rapidly evolving from traditional Layer 2 networks to Layer 3 centric leaf-spine architecture, too many of us are still dealing with the headaches that come from L2 technologies such as Multichassis Link Aggregation (MLAG), First Hop Redundancy Protocol (FHRP), and Spanning Tree Protocol (STP).
What if there was a way to provide the redundancy MLAG can offer without its fragility and risk to maintenance windows?
The History of Data Center Design: From Bridging to Routing
In the early days of data center architecture, the entry point to the data center consisted of a pair of aggregation switches which ran a routing protocol with the backbone and was the center-of-the-universe for an L2 domain that connected the servers. Initially, the servers were wired directly to these aggregation switches; however, increases in data center scale lead to a large L2 switching fabric. At the time, many applications were very “L2-aware”; they relied on L2 for things like application/service discovery and heartbeats to the point that technologies like vMotion migrated L2 identity and could only work in an L2 domain. As the scale increased the industry tweaked on Spanning Tree Protocol (per-VLAN spanning tree, Rapid Spanning Tree, etc) and created technologies like bonding, MLAG, and virtual chassis. Today, the state-of-the-art L2 data center design consists of an MLAG based leaf-spine.
While this was going on, services providers and webscale companies realized that these L2 networks had bad failure characteristics, were hard to troubleshoot, and didn’t scale. They looked at the research in computer science to realize that interconnects like Fibre Channel and Infiniband rely on highly parallel meshed fabrics; just like the L3 networks you can build with IP routing. They started reducing the scale of their L2 domains to stay within a rack of servers. At the same time, applications have evolved to use IP for service discovery and housekeeping in part to broaden their use case and because the “IP socket” is a bread-and-butter communication tool in all application frameworks.
Over the course of my career, I’ve seen a few companies route directly to their hosts. However, those were few and far between – in part due to comfort with the available software as well as due to the configuration complexity that existed with routing. Maturation of routing suites has addressed this first issue and the introduction of Open Shortest Path Available (OSPF) and Border Gateway Protocol (BGP) unnumbered to deploy routed networks has made configuration extremely simple.
Routing on the Host for the Win
Routing on the host is a natural extension of the adoption of industry standard L3 protocols as part of modern data center design. Expanding the L3 footprint further into your data center by leveraging the routing protocols OSPF or BGP, all the way to the server means you can resolve that MLAG and STP problems once and for all. It lets you finally realize more resilient networks running at higher speeds.
By Routing on the Host, you gain two huge bonuses to data center operations: the ability to gracefully remove a switch from the fabric for maintenance and increased bandwidth redundancy by having more than two ToRs.
L2 networking operates such that there is no way to influence data flow without being disruptive and incurring some traffic loss. The most sophisticated deployments with MLAG and LACP suffer from this fate while relying on non-standard protocol interactions along the way. Routing protocols like OSPF and BGP allow a node to gracefully remove itself from the data flow by notifying the rest of the network to route around them; enabling safe and structured rolling upgrades of networking equipment.
L2 networking is also founded on the premise of a single path between any two points; that is the foundation of the spanning tree protocol. Through the years, the industry has come up with mechanisms such as bonding and MLAG to improve upon this; however, they are all still fighting nature.
In contrast L3 networking was designed to allow and encourage multiple paths. American computer scientist Van Jacobson once told me that the Internet was designed so that a major city could take a direct hit and the system would still function. This characteristic, when applied via Routing on the Host, allows data center architects to build in excess bandwidth and redundancy into their networks using simple, standards-based tools.
Routing on the Host provides an elegant architecture when building infrastructure to support container-based applications. Architects can use Routing on the Host to either advertise the overlay endpoint when using native Docker networking or use a network plugin to dynamically advertise individual containers as they are provisioned.
In the Field: The New Gold Standard in Modern Data Center Design
Routing on the Host is available for download by anyone in the community. We have several customers that are operating their data centers with L3 all the way down.
Email marketing software provider Campaign Monitor has used Routing on the Host to greatly improve the availability of critical services across their network. The biggest benefits are the addition of Quagga BGP unnumbered and the elimination of L2 technologies such as MLAG, FHRP and STP.
SWITCH, a Switzerland-based non-profit internet service provider, is another company that uses Routing on the Host to eliminate MLAG and spanning tree in their environment, while still providing redundancy to the host. Cumulus Quagga’s OSPF unnumbered gave the organization network agility, making it a core functionality for them. Deploying Routing on the Host with Cumulus Quagga improved their overall system availability while allowing simpler operation and troubleshooting.
Finally, as an enabler of online 3D printing services, Shapeways deployed Routing on the Host architecture to achieve network reliability, scalability and availability in order to meet its resiliency and simplicity goals. The resulting system allows Shapeways to serve their demanding customers and partners both now and as their business continues to grow.
Routing on the Host powered by Cumulus enhanced Quagga packages is a way to achieve power, flexibility, and web-scale reliability throughout the entire data center. Declaring independence from late night troubleshooting and L2 data centers may be more possible than you could have ever imagined.