For Facebook and Instagram, it’s All About Automating the Workflow
When running a shop with many talented (and highly-paid) software engineers and developers, you want to keep their innovations rolling out as quickly as possible.
So it is not surprising that Facebook, and the Facebook-owned Instagram, create tools to expedite the process of moving new services and features from the workbench to the front-line servers.
In separate blog items posted this week, Facebook software engineers describe a new workflow tool they created that automates the optimization of bytecode for the company’s Android app, and Instagram touts a new programming language for specifying how updates get propagated across its system.
Both fit squarely in practice of DevOps, which seeks to tighten the feedback loop between developers and operations folk in order to hasten the process of getting new features and services to users.
The Facebook work came about in an effort to boost performance of its Android application, according to the post, which was jointly authored by Facebook software engineers Marty Greenia, Bert Maher and Shane Nay, who worked on the project primarily at the company’s Menlo Park, California headquarters.
The post details the workings of an internal application, called Redex, that acts like a production assembly line of bytecode optimizations.
With Redex, Facebook developers can plug their own optimizing transformations into the process for compiling the Android code files, which are known as .dex files (short for Dalvik Executable). Every time Facebook updates its Android application, the code is run through this toolchain, which hammers out the inefficiencies.
“The Java compiler produces the Java bytecode, the .dex compiler takes the Java bytecode and produces .dex bytecode. Redex takes the .dex bytecode and optimizes it as needed,” explained Maher, in a follow-up e-mail.
The Redex pipeline
The work on optimizing bytecode is, in itself, pretty unusual, though not unprecedented. When it comes to optimizing code, most developers focus on tweaking the source code.
Bytecode itself was designed to be extremely compact, offering byte-sized abbreviations for each machine instruction. Compiled to be read by a Java Virtual Machine, bytecode is the intermediary step between the programmer’s source code and the final machine language run by the processor.
Nonetheless, there are gains to be had in squeezing the bytecode, Facebook has found. Less bytecode means the app runs faster, with fewer instructions to execute. The app itself takes up less room on the device, and can be downloaded more quickly.
“Facebook engineers tend to move fast, so we wanted to architect something that would benefit from multiple engineers working on lots of optimizations,” the post’s authors wrote of Redex.
They list a few of the optimizations.
One plug-in does minification. Developers may make class paths names or function names very descriptive — that is to say wordy — to ease in debugging and understanding the code at some later date. The minification plugin replaces long strings with shorter placeholders. It also generates a backup map that developers can user later when debugging.
Minification
Another plugin does inlining. Again, the idea behind inlining is that the best practices for the programmer aren’t necessary the most efficient ones for the machine.
For writing code, good software engineering practices dictate the separation, or encapsulation, of code into discrete chunks of functionality.
Inlining simply involves moves a function being called closer to the function that is doing the calling. This could involve placing a child function right next to its parent function. Or it could involve removing an intermediate wrapper designed to smooth the process of writing a function call.
“Whenever we inline one function into another, we can reduce the overhead (and bytecode) associated with a function jump,” the developers wrote.
Inlining
While Facebook has automated its process of optimizing bytecode, Instagram has made several refinements in its process of rolling out new features.
Like most web-scale companies, Instagram likes to test new features by trying them out on a subset of its users at a time. If the feature works, it can then be rolled out in stages to successively larger sets of users. If it breaks something else in the system, or if the initial users hate it, it can be pulled back before it stinks up the whole joint.
Initially, Instagram controlled these cascading releases through software called IG Gateway.
IG Gateway had its limitations. Developers had to write the deployment instructions directly into the feature code itself. This meant when it came time to deploy the feature in full production, updated code would have to be reshipped to all the servers worldwide. It also didn’t offer much nuance in how a new feature could be rolled out or pulled back. It wasn’t extensible, as IT architects are wont to say.
So Instagram engineer Chenyang Wu, also working in Menlo Park, created a Python-based domain specific language (DSL) that Instagram developers could use to more tightly control how their updates went out on Instagram.
Based on Python, the Gate Logic language is simple enough that even program managers and sales folk can add their own conditions, Instagram claimed.
This language is designed to express the conditions for rolling out a new feature in simple Boolean logic. The engineer can specify, for instance, a single country or region to introduce a feature in, or only to run the feature on a certain OS, or on even a specific version of an OS.
Instagram’s Gate Logic
After the engineers write the rollout conditions, they are compiled into native Python bytecode, and added to a database, where they can be picked up by Apache Zookeeper, the open source software Instagram uses to schedule operations on its servers worldwide.
“With Gate Logic, we’re able to control feature rollout in a safe and flexible way, with no performance loss compared to hard-coded Python,” Wu wrote.
Neither Facebook nor Instagram have immediate plans to open source the software.
Feature image via the Facebook blog.
