So the tools team for the company had to work to build out a pipeline that could accommodate these languages, and others as well, noted Mike McGarr, Netflix developer productivity manager, speaking at the O’Reilly Velocity systems engineering conference in New York last week.
The result is a new workflow developer tool called NEWT (Netflix Workflow Toolkit), a programming language-independent application that provides a way for developers to specify their own chosen Build tools to package their applications according to the official Netflix specifications.
“I can tell NEWT to run pretty much anything I want and NEWT doesn’t care. It is unopinionated about builds. It is un-opinionated about dependency management for your application,” McGarr said.
When it started as a company that shipped DVDs by mail, Netflix was a Java shop running on Oracle software in a private data center. As the company moved to streaming, however, it experienced some “challenges of scale” that this monolithic Java-based setup was ill-equipped to handle. “We started to reassess what our architecture should look like,” he said, noting that this re-engineering had led the company, like many others, down the path of cloud computing, microservices and NoSQL.
This path proved to be problematic in that most of the developer tools in its pipeline were built around the Java stack. “Building” the apps that the developers created is overseen by the Java-centered Gradle. They were “baked” in a virtual machine where it can be shipped to the cloud as an “immutable artifact” (meaning it is not tampered with once deployed), using Gradle plugin developed by Netflix called Nebula. The company developed its own continuous integration and deployment software, called Spinnaker, to automate the deployments.
The deployment side of operations is fairly un-opinionated — it doesn’t need to know programming languages were used. The build side of operations was pretty wired into Java, however. But what if a dev would want to build a Node app? At first, the tools team adjusted Gradle for these use cases, asking developers to set the configuration to use Apache instead of Tomcat, the default Java application server.
The tipping point came with a new initiative to re-architect a major Netflix component, using Node.js. “We couldn’t ignore this growing trend of non-Java,” McGarr said. “At Netflix, we don’t generally mandate tools. Tools have to win out,” among developers, McGarr said.
In short, the company needed a way to package non-Java apps in its preferred .deb packaging format. Though finding the non-Java Build tool to package up all the dependencies, however, proved to be problematic. So many choices! Webpack, Brunch, Browserifly, Grunt, Gulp, npm’s Bower, and Broccoli.js, for starters.
So the team started building NEWT, a command line tool, written in the Go programming language. Go dependencies have no binaries, so they freed developers to work on any machine they preferred.
The NEWT package command checks to see what dependencies an application needs, and installs them, along with the application itself, in a Docker container. This ensures that each developer gets the correct support tools and system dependencies for each application. The team developed profiles for different applications, called “app types,” that specifies the required configuration and conventions in YAML files for Node.js apps, Python apps, or other types as well. Developers can specify in the YAML file which build tools they want to use to build out their apps.
Why build a developer workflow tool in-house when other options, such as Chef’s Habitat are just arriving on the marketplace? Netflix does try to buy rather than build anywhere possible, McGarr said, after the presentation. But other workflow tools were, for Netflix’s purposes, were opinionated with the wrong opinions. In this case, it was better to build from scratch.
Currently, NEWT is not open source, though it may be in the future. The project is only managed by a handful of developers. As more are hired, McGarr will have the sufficient manpower to prepare the software for external release, he said.
Chef is a sponsor of The New Stack.