A Guide for Running Multiple Controllers in Software-Defined Networks
Illustrations by Sridhar Rao.
In this article, we will take a bird’s eye view on multiple controller scenarios for software-defined networks (SDNs). In this article, when we discuss the multi-controller scenarios, we will not restrict to those controllers that use standard protocols such as OpenFlow. Instead, we’ll use the broader meaning of the term “controllers.”
I would like to highlight that, with the growth of NFV, the multiple-SDN-controllers’ use-cases/scenarios are taking a lot of importance, mainly due to the inherent nature of the NFV-based services. NFV-based service realization may require orchestration of management of heterogeneous resources and at multiple geographical locations and involve multiple administrative domains.
The term “multiple” covers both multiple homogeneous controllers, where all the controllers have same capabilities and possibly different roles, and multiple heterogeneous controllers, where different controllers with different capabilities are used. Typically, in the case of homogeneous controllers, the communications between the controllers are mostly limited to, what is commonly termed as “east-west communications,” whereas, in the case of heterogeneous controllers, the hierarchically-arranged “north-south” communications are common.
The multiplicity — whether hierarchical or flat — of SDN controllers is typically driven by a few key technical and business drivers such as performance improvements, scalability and reliability, multiple administrative domains and their interactions, multiple network layers and corresponding fault management.
In hierarchical arrangements, typically, the controllers have specific ‘roles’ performing specific tasks. There are no standardized ways of communications among the hierarchy of controllers. In flat (non-hierarchical) deployment scenarios, standard distributed system solutions have played signification role in achieving the communication between the controllers.
The below table summarizes various use-case attributes that may need multiple SDN controller deployments. I would like to highlight there may be overlaps, i.e., a single use-case may include one or all of these attributes. For example, a transport-SDN use case would include multiple domains, multiple network types, clustering and heterogeneous network elements.
| Use-Case Category | Example |
| Different Network Types (end-to-end provisioning, multi-layer optimization) | Optical, IP/MPLS, Heterogeneous-networks. |
| Provisioning of High Availability (Clustering) | Active-Passive an N-Active |
| Multiple Domains (Cross-domain optimization and deployments) | Data Center Interconnects, Multi-domains within single data center and combination of Packet and Optical Transport.
|
| Heterogeneous Network Elements Management | Network Connectivity for Virtual machines,
Physical Network, Virtual Machines (when they act as data plane elements – ex: NFV’s VNF) |
If we consider the typical ‘Transport SDN’ use-case, we typically find multiples technologies, such as IP/MPLS, Optical, and so on. In such cases, there could be more than one SDN controller managing switches in data centers, aggregation switches, and optical switches.
In addition, to achieve end-to-end service provisioning and resource management in such multi-layer networks, there would be a need for interaction and coordination between Optical and IP/MPLS networks. Such coordination can be typically achieved by deploying multiple SDN controllers and realizing efficient interactions among them to achieve multilayer optimization and intelligent traffic engineering. Multiple-controller deployments are also expected to handle ‘restorations and optimizations’ at both IP and Optical levels, in the case of failures.
At the Open Networking Summit 2015, Huawei’s Peter Ashwood-Smith presented the solution of unified management in Transport SDN. He began by highlighting typical carrier customer’s problems such as cost of network operation, cost of equipment, and the integration of operation control and management. He then showed how three NFV-based technologies could help: Automatic IP Link deployment, service-driven IP topology optimization, and Multi-layer restoration.
The SDN Controller cluster can be seen as a system that manages a large-scale network, handling control plane communications, consisting of a large number of network elements. Typically, such systems consist of multiple controllers and expose themselves as a single logical entity.
In a majority of the real-time deployments, the controller plane is realized as a cluster of SDN controllers. Multiple controller nodes are to be deployed in different cluster configurations for high availability. Such configurations may come in different forms, typically active-passive or multiple-active.
The SDN cluster, apart from the address common challenges such as availability, scalability, reliability, consistency, and security, should also help with other issues as well, such as efficient topology management and corresponding path/route computations, consistent data store (data that is shared across controller instances), and efficient northbound APIs.
Many commercial vendors of SDN controllers, such as NEC, Fujitsu, Cisco, BigSwitch, insist on deploying their controllers as distributed cluster. In fact, all practical deployments of SDN controllers rely on some form of cluster deployment, typically, to achieve either availability or scalability or both. Even the open source projects such as OpenDaylight and the Open Network Operating System (ONOS) supports clustering, with the latter (ONOS) being built as distributed architecture.
Multi-Domains
Multi-domain environments are those networks that interconnect data centers, enterprise networks, customer sites and mobile entities. Such networks are typically decomposed into administrative/geographical domains, which are interconnected using variety of network technologies.
Multiple SDN-controller deployment scenarios, including multiple domains, typically target end-to-end service provisioning and optimization. The below figure summarizes various domains – data center, provider network, internet exchange points and mobile backhauls. Some real-world deployments consider domains such as cloud, metro, and core networks, where the service deployments typically span these multiple domains. In addition, orchestration (a major challenge in multi-domain deployments) usually occurs both within a domain and across multiple domains.
Resources Managed
Multiple SDN controllers can be used to manage different types of network elements:
Physical, Virtual and Virtualized-Network Elements
The controller(s) may manage physical switches (Ex: NEC’s programmable flow switch), or virtual switches (ex: Cisco’s 1000v) or virtualized network element (Ex: VNFs). There may be cases where different controllers can be used to manage each of these switches. I strongly believe that NFV-deployments are the strong candidates to adopt such multi-controller deployments.
Open and Proprietary
The network elements can be: (a) proprietary and use ‘open and standard’ protocols (ex: NEC’s Programmable Flow switch), (b) White-box switches with proprietary but open and standards-based software (ex: model used by pica-8, big switch, cumulus networks, etc) and (c) Proprietary and closed software (ex: Cisco’s hardware switches). In a deployment involving such open/proprietary switches, different controllers may be used to manage such heterogeneous switches.
Different Network Types
The cloud infrastructure may be composed of an underlay and overlay networks that are managed by different SDN controllers. For example, OpenStack and OpenDaylight may be used in realizing and managing the overlay network, and have an NEC’s programmable-flow SDN controller to manage the underlay.
Mixed OpenFlow and Non-OpenFlow Deployments
An increasingly common scenario is one in a centrally-managed network support both OpenFlow and non-OpenFlow resources. For example, a data center network may have a core network that is purely OpenFlow-based, over which VXLAN-based runs, including a VXLAN-gateway, which could be managed by a controller and not support OpenFlow.
In summary, multiplicity of SDN controllers – hierarchical or flat – can be seen in various cases such as data center network with underlay and overlay networks, telecom networks with layers of network technologies, networks including heterogeneous network elements, and networks involving multiple administrative domains. Such multiple controllers help in improving performance, scalability, reliability, and provide efficient end-to-end services.
