Azure Arc Is Developing into a Full Hybrid Infrastructure System
When Microsoft announced Azure Arc as a new option for hybrid and multicloud at its Ignite conference in 2019, the vision was much broader than the initial preview, which covered only managing VMs through Azure Resource Manager. Now that more of the features are available, like a public preview of the Kubernetes support, organizations can try out the wider scope.
That progression is deliberate, director of Azure management Jeremy Winter told The New Stack; starting with a minimum viable product and developing features in the order prompted by customer feedback (for example from the private preview of Kubernetes support for some very large customers last year) “instead of unpacking it and showing one big thing that’s globally available and all done.”
“The whole hybrid space, as well as the Kubernetes space and the path we’re taking with the Arc, enabled services — they’re all so new and out on the cutting edge that is shifting so much faster, that we’re taking a much iterative approach on it.”
Hybrid cloud isn’t just about infrastructure or even services; often, management is what enterprises are thinking most about.
Arc is about extending the Azure control plane to manage resources beyond Azure, like VMs and Kubernetes clusters wherever they are, whether they’re Windows or Linux or any Cloud Native Computing Foundation-Certified Kubernetes distro, even if they’re not always connected to the internet; first to apply common governance polices — whether that’s about compliance, cost management or patching orchestration — and then to deploy PaaS services (initially Azure Data Services but others further down the line) onto those managed resources.
This isn’t just about operations though: it’s also a chance to get some of the benefits of cloud, like agility, scaling and always being up to date, from existing on-premises hardware that isn’t uniform enough to run cloud services on directly the way Azure Stack hardware can.
Arc Is ARM Everywhere
While there are many ways to manage infrastructure on Azure, from the portal to third-party tools like Terrraform and Pulumi, Azure Resource Manager (ARM) templates (which represent resource schema in JSON declaratively and with parameters and can be created in Visual Studio Code) are the native infrastructure as code solution for Azure. You can use ARM templates through the Azure portal, the Azure command line or the Azure SDK (which tools like Azure Data Studio are built with), so you can use any of those to work with Arc.
Arc uses ARM, local agents and resource providers (created by Microsoft or third parties) to manage VMs (on your own servers or in cloud IaaS), Kubernetes clusters and the new Azure Stack HCI (where Arc is built into the OS), as well as the services — initially Azure Data Services — that you can run on top of them.
On Kubernetes, the agents run in the Kubernetes namespace, Winter explained. “They’re responsible for the connectivity to Azure, they collect Arc logs and metrics, they watch for configuration requests, and they marshal calls back and forth between the Azure ARM control plane and Kubernetes.”
Arc ties into ARM features like tags, policy and RBAC, and to Azure Management logs and Azure Policy. You can also install OMS, Desired State Configuration and custom script extensions in both Windows and Linux VMs, which let you apply Azure Update Management, inventory, change tracking and Azure Monitor insights to Arc-enabled servers. That way you can see what updates a server is missing and deploy them, track the software inventory on Windows and Linux services (down to registry keys and daemons) or track server and cluster health and performance — whether the hardware is under your desk, in a data center, in Azure or in AWS.
You still use native tools for provisioning VMs and clusters, doing upgrades and lifecycle management, and for monitoring though, so you’ll be using tools like kubectl, Helm charts, CRDs, Grafana and the GitOps workflow that’s become common for Kubernetes, or PowerShell and Windows Admin Center.
If you use a tool like System Center or SUSE Manager or the AWS management console to manage your VMs, you’ll still manage deployment and lifecycle with that, but Arc will handle policy compliance. “You can use Azure Arc to manage at scale, providing one central view and manage all of it in one place. SUSE Manager gives you a control plane to manage systems running with SUSE (but not only),” Thomas Di Giacomo, SUSE president of engineering and innovation, explained to us.
GitOps is central to Arc because so much Kubernetes adoption is led by developers, Winter explained. “We wanted to ensure you could use your existing DevOps pipelines, use your existing Kubernetes manifests, you could use kubectl to work with it and then Helm charts if that’s how you’re wrapping up your deployments But because they have it-Arc enabled and they’ve created policies, with GitOps configurations and ARM policies, now whether the developers or whoever is doing administration through the command line with kubectl or doing deployments through their pipelines, that environment ensures they stay within policy or configures it to policy immediately.”
You do need to have connectivity to Azure to register a Kubernetes cluster with Arc for management because it needs to pull configuration information from Azure; but you can still manage the cluster without a permanent connection to Azure as long as the agents can access the local git repo.
Microsoft customers are already using that kind of Kubernetes DevOps pipeline, but at Fortune 500 customers, that may still be in pilots and labs rather than deployed in their mainline business, Winter noted. “I still think we have a way to go in the industry with Kubernetes becoming the mainline and the practices around it.” Often, they’re modernizing existing apps.
In fact, while customers using the Arc server preview liked the “single pane of glass” for management, for Kubernetes they want Arc to tell them what Kubernetes infrastructure they already have. “They’re taking an inventory of what’s out in that Kubernetes environment, as well as using the GitOps configuration to simplify the configuration and governance. I think there’s just a lack of Kubernetes management tools and cluster management tools.”
Although Arc can manage servers and Kubernetes clusters in any cloud, organizations who have chosen Azure are often the ones trying out the Arc previews. “What they’re saying is ‘we’re Azure-centric customers; we believe in the path you’ve given us with ARM templates and the ARM control plane with management groups and tagging and policies in the resource graph for inventory and real-time insight within the environment. I struggle with that in a server world and I don’t have it in a Kubernetes world and I’d like to apply what you give me in the Azure world back [to those environments]’,” Winter explained.
The current situation with remote work adds to that pressure and organizations are using the Arc GitOps workflow to get governance and consistency with the increasing amount of remote development. “You can ensure that the corporate policies you need or your general policies for those dev environments are enforced.”
Although you can use Arc to manage any CNCF-certified Kubernetes distribution, the main ones Microsoft is focusing on initially are RedHat OpenShift, Canonical’s Charmed Kubernetes, Rancher Kubernetes Engine (RKE) and GKS (as well as Microsoft’s own Azure Kubernetes Service). Microsoft has also done some work on integrating Arc with SUSE SLES and RedHat RHEL; for instance, you can use Arc to manage SQL Server on RHEL or Azure SQL Managed Instance and Azure PostgreSQL Hyperscale on OpenShift.
“We’re starting to do some work with Pacific on VMware,” Winter noted (that’s the project to integrate Kubernetes into VMware vSphere) but the focus is on more on getting Arc ready for general availability than on deep integrations with partners.
The core capabilities of Arc are ready, Winter said. Private previews with customers already proved out the concept and feedback from those customers led to prioritizing features like extensions. Public previews will validate the quality with customers and preparing for GA will be more about getting the Azure regions (included the government cloud regions) and security controls ready. “We’re seeing customers already start to push Arc at quite a bit of scale and they’re already starting to talk with us about shifting into production environments.”
Sophisticated Services
The enthusiasm for extensions gives some clues about how customers want to use Arc. “They’re wanting to have all these management services we already built-in Azure come down to Arc a lot faster than we expected them to,” Winter noted. “We started to build out the management services from the ground up over the past five or six years in Azure and now they’re mature enough that we can not only bring them down to bring a Kubernetes and server control and governance plane to Arc but start to bring more than management services. Security Center is one that customers wanted for the extensions; you’ll see backup, patching, monitoring and all those other follow suit.”
One of the goals for hybrid cloud is to bring cloud services to places where you can’t use cloud, whether that’s because of connectivity, data gravity or regulations, and these higher-level services are what a lot of organizations are interested in Arc for.
The first cloud service on Arc is Arc Data Services; still in private preview, initially for the Azure SQL Managed Instance and Azure PostgreSQL Hyperscale services, offering the same elastic scaling, always up to date, secure, cloud-billed database service you’d get in Azure, with a single management portal for multiple database services, but on the infrastructure of your choice (by using CRDs to deploy those data services onto Kubernetes clusters).
“Any application needs to have its data layer and the ability to have SQL run in any of those Kubernetes environments is key,” Winter noted. But Arc Data Services is also a way of understanding how to bring an Azure cloud service to Arc infrastructure. “We’re packaging what I’ll call a collection of services that brings in the ARM control plane, the Kubernetes environment and the data services so it can run on any Kubernetes environment and we’re taking the time to get the right architecture on that. It’s more than just deploying an agent and getting a connection back with the ARM control plane; there’s more that we’re doing to create Arc-enabled services we can run as very portable edge services.”
That means that although Arc may be generally available later this year, Arc Data Services may be released later — with other services to follow. The first is likely to be Arc App Services, which will combine Azure Functions and Event Grid into a PaaS deployment platform you can run on Arc, and other Azure services will be packaged in the same way in the future.
Having an app service will help when customers compare Arc to Anthos (which supports Knative and Cloud Run), which Winter says is already happening even though Anthos had something of a head start. “There’s a lot of conversations where we’re pulled in now [to compete with Anthos] and Anthos customers are reaching out to us because, as they look at Anthos, then they turn right to Arc.”
“I think we will see a lot of bake-offs between Arc and Anthos over the next 12 months,” CCS Insight vice president for enterprise research Nicholas McQuire agreed.
“Arc initially has been positioned as a broader play, governing a wider set of resources including bare metal, VMs and Kubernetes etc., in addition to layering in monitoring and security. Whereas up until now, Anthos has been much more about doing container development in Kubernetes at scale across infrastructure environments, most notably AWS. But both are moving rapidly — from adding bare metal, VMs, container clusters and multicloud orchestration and governance to new security features.”
“We are also now seeing next frontier services coming into the picture as well: ML, edge and 5G for example. All will become core parts of these environments over the next year.”
Other cloud providers view hybrid is a way to put their cloud services on-premises (Microsoft has Azure Stack Hub for that), to offer their cloud services outside of hyperscale regions (Microsoft has Azure Stack Edge and Azure Edge Zones for that) or to wrap their arms around usage on other clouds. Arc does address multicloud management but it’s much broader than that. Arc is an attempt to unify very different kinds of infrastructure — whether that’s VMs, bare metal, private cloud with Azure Stack Hub, hyperconverged infrastructure with Azure Stack HCI, Kubernetes, Azure or other clouds — into a single platform that can deliver applications and services, so organizations can use cloud practices like DevOps and monitoring across all of them.
You could use Arc to manage Kubernetes running on multiple Azure Stack Hub instances in different data centers from a single place; or you could use it to manage the physical servers running your own Kubernetes cluster as well as the cloud Kubernetes service you scale out to. The beauty of Arc is that you can do all of that, in the same single place, and then use that mix of infrastructure to run cloud service on.
That’s not quite private cloud, multicloud or any of the familiar versions of hybrid cloud; but it’s an ambitious approach that covers more of the large heterogeneous environments many Microsoft customers have.