Fugue Strives to Bring Management Harmony to Amazon Web Services
Frederick, Maryland-based Fugue hopes to make Amazon Web Services cloud deployments run at scale as harmoniously as a symphony.
It’s drawing heavily on its namesake musical form, in which a theme repeats and evolves over time. The company offers a “cloud operating system,” called Conductor, serves to manage cloud resources much like Linux or Windows do on individual machines. (Its logo is animated to Beethoven’s Fifth Symphony.)
Companies are trying to manage the cloud like they would a traditional data center while the problems are different, according to CEO and co-founder Josh Stella.
“Pretty much anybody who’s operating at scale on AWS has somebody who’s writing bags of scripts that they’re bailing wiring together to get some kind of automation” — Josh Stella
“We know that under the hood, but cloud is not a remote data center, it’s really a big distributed computer,” said Stella. As a result, the cloud needs to be managed as such.
Stella previously was principal software architect for AWS. He says his experience and that of other AWS veterans on staff give them unique insight into the problems that arise with cloud deployments at scale.
At any one time, Netflix, which he calls “the poster child for success on cloud,” operates between 50,000 and 70,000 compute instances on AWS. Fugue has been working with Netflix to replicate that success for a more general customer base.
“Pretty much anybody who’s operating at scale on AWS has somebody who’s writing bags of scripts that they’re bailing wiring together to get some kind of automation,” he said. “We wrote Fugue as a systems-level answer to this problem of exploding complexity and dynamic and short-lived infrastructure that’s driven by the application.”
Fugue provides a way to define infrastructure upfront and a runtime component to ensure that all cloud resources comply with that definition. It corrects automatically should resources “drift” away from the original definition.
“A very common problem in building things on cloud that use lots of service types is that there’s no one place to describe the system as a whole,” said Stella. “You might deploy half a dozen Lambda functions, an API gateway, an ECS [EMC Elastic Cloud Storage] or Kubernetes cluster, you have a VPC [virtual private cloud]. How do you coordinate all these things? Fugue can be the overarching description and declaration that defines that system.”
Gaining Simplicity
An immediate use case is to use Fugue as the basis of an infrastructure-as-code architecture, as well as policy-as-code, he said. You can have composition libraries that assert company policies, which is important to regulated companies, such as HIPAA-covered entities. It’s isolated within your inside your own cloud account and does not “call home” with any information.
While Stella wrote an O’Reilly ebook on immutable infrastructure — infrastructure that is replaced rather than maintained — he says Fugue is equally applicable to traditional workloads without it.
It can be used in scenarios like these:
- As part of a continuous integration toolchain with the likes of Jenkins, CircleCI, or Travis to automate cloud infrastructure lifecycle management.
- To preview changes to your infrastructure.
- To implement an enforceable change-control process using your code repository tool, such as Git or GitHub, to provide a single, trusted view of your cloud.
- Building and managing the underlying infrastructure to support containerized applications that use tools like Docker and Mesos.
Ludwig, its proprietary language, is used to write compositions, the defined programs that the Conductor runs. (Stella says it’s “keeping options open” for open-sourcing this technology.)
It doesn’t require programmers to learn a new language — it looks just like a YAML file, Stella said. It’s a compiled language using a strong type system.
“Ludwig is key to getting simplicity. You need to be able to express what you want to happen on cloud. Right now, the choices are pretty grim. You can do that manually through the console, you can learn a JSON-template-based domain-specific language,” he says, adding that Ludwig was designed to make common cloud tasks easy.
“Setting up a network in a VPC takes about eight lines of code for an Amazon Virtual Private Cloud (VPC), subnets, routing rules, Internet gateways, everything,” he said.
But if you get anything wrong in that code, Ludwig has a compiler to give programmers meaningful errors while they’re writing compositions rather than after the application runs.
Conductor, an IaaS API, acts as an operating system “kernel” for automating cloud infrastructure operations. It’s not meant to replace Linux or Windows or any other tools for that matter, Stella said.
To manage policy with Fugue, you embed best practices into libraries used in the composition of the VPC.
“We have an example of this that enforces the HIPAA rule. It’s just one function call, then Fugue will build it. But it will also monitor it and enforce it. If anything goes in and changes it, other than through the Conductor through a sanctioned command, it will put it back how it’s supposed to be,” Stella said.
It’s all managed as processes.
“Because you told the Conductor how it should be, and because it’s constantly monitoring and enforcing that — and it can modify it and also tear it down — rather than having to shape the hairball that winds up on AWS, you can just look at your processes and make sure they’re correct or choose to run new ones,” he said.
Continued Evolution
Fugue, previously known as Luminal, was founded in 2013. It changed its name to its flagship product in January when it announced a $20 million series C round of investment. It has raised $33.8 million total.
“We think cloud is going to continue to innovate and evolve probably for another decade at least. So Fugue is designed to incorporate very quickly any new cloud services that come out,” he said.
Few people would have predicted the success of Lambda “when it looked like containers were going to be the next big answer for a very long time,” he said. Fugue supports containers and its flexibility allowed the company to add Lambda functions in two weeks.
Containerized solutions and tools for provisioning hardware will usher in an era of operations that don’t require human management, Jamie Dobson, CEO of Container Solutions, a company specializing in programmable infrastructure, wrote recently for The New Stack.
Fugue joins the ranks of new companies focused on automating infrastructure and enforcing business policy.
Billing itself as a “DevOps automation platform,” Calm.io tackles the challenges of developing, provisioning and deploying applications, automating compute resources as well as the underlying storage and network and applying business policies to it all.
CloudFabrix, meanwhile, focuses on governance to ensure applications meet key business goals.
CloudFabrix and Docker are sponsors of The New Stack.
Feature Image: “charles bowling conducts westland orchestra” by woodleywonderworks, licensed under CC BY-SA 2.0.
