How the U.S. Air Force Deployed Kubernetes and Istio on an F-16 in 45 days
As hybrid cloud strategies go, the U.S. military certainly is taking a unique approach.
Just like almost everything else, military organizations increasingly depend on software, and they are turning to an array of open source cloud tools like Kubernetes and Istio to get the job done, according to a presentation delivered by Nicholas Chaillan, chief software officer for the U.S. Air Force, at KubeCon 2019 in San Diego. Those tools have to be deployed in some very interesting places, from weapons systems to even fighter planes. Yes, F-16s are running Kubernetes on the legacy hardware built into those jets.
“One point for the team was to demonstrate that it could be done,” Chaillan said. He challenged the Air Force and its partners to get Kubernetes up and running on a jet in 45 days, and while that was as difficult as it sounds, the team met the goal and F-16s are now running three concurrent Kubernetes clusters, he said.
The natural follow-up question is, of course, why? During his packed presentation following the opening-day keynotes at KubeCon, Chaillan explained how the Air Force — and now at his direction, the rest of the Department of Defense — is making a big bet on containers, Istio and Kubernetes. It’s a flexible but universal development platform for software teams across the military and prevents vendor-lock in.
Before the Air Force embarked on this project about 18 months ago, most military software teams were building software using the old-fashioned waterfall process, and it could take years for new code to make it all the way through to production, Chaillan said. And even then, updates, testing and even security reviews relied heavily on human labor to complete tasks that the commercial sector had long ago decided should be automated.
The military can’t really afford to fall too far behind the mainstream when it comes to the technology it needs for its mission. As rivals invest in their own software capabilities, they could get an edge on the battlefield. And the security implications of a glacial process required to update some of the military’s most critical applications are obvious.
“It’s very important for us to reduce the attack surface and be able to mitigate threats,” Chaillan said.
Department of Defense Enterprise DevSecOps
Chaillan and his team decided to embrace open source software as the foundation of the new development platform, which they called the DoD Enterprise DevSecOps Initiative. This initiative specified a combination of Kubernetes, Istio, knative and an internally developed specification for “hardening” containers with a strict set of security requirements as the default software development platform across the military.
Software teams in different branches or regions have some discretion over how they use the tools at their disposal, but everything must be constructed on the layer provided by the Air Force’s Platform One team, and there are several things teams are not allowed to change, Chaillan said. And, interestingly, given all the hubbub that has surrounded the Department of Defense’s 10-year JEDI cloud contract awarded to Microsoft last month, the entire stack was designed to be run on either Amazon Web Services’ GovCloud or Microsoft Azure.
Chaillan expanded on that during a press conference hosted by the Cloud Native Computing Foundation following his talk, explaining that “we don’t want to get locked into any one thing.” The team chose to use Kubernetes specifically for this reason, with other projects like Istio providing security for the networking layer of the DoD stack: “We want to make sure Istio is running constantly across the entire stack,” Chaillan said.
There were, of course, lots of challenges along the way. Kubernetes was not designed for the disconnected environments that the military must use in many situations involving data that should not ever reach the internet, Chaillan said. He hinted that DoD will have a lot of suggestions for Kubernetes maintainers as they address this portion of the project, which could help pave the way for Kubernetes to be used in other sensitive operating environments.
“We’re very used to updating using the internet, and having connectivity to the internet, getting the updates directly from the internet. For us, we have to bring the entire stack with us” on jets or weapons systems that are disconnected from the internet by design, he said.
An Open Source Stack at DoD Scale
The scale at which the DoD operates is unlike almost all commercial operations; Chaillan had to train 100,000 people on the principles of DevSecOps, not to mention the new tools. “The culture shift has been interesting,” he said during the press conference, seemingly with some restraint.
That scale is a reflection that the DoD will be using this new development platform for lots of mundane and unclassified applications: More than 2 million people work for the military, and most of them are not flying F-16s running Kubernetes.
“The jet is interesting, but it’s a tiny piece of the rest of the work we’re doing. We have a lot of business systems moving to cloud native environments, moving to microservices, being built right from the get-go,” Chaillan said.
The entire DoD Enterprise DevSecOps Initiative stack is open source and available for anyone to check out, Chaillan said. Military agencies are “getting better” about releasing more of their work to the open source community, he said, also noting that “we’re not going to fork (open source) software.”