The Need for Speed: LinkedIn’s Rapid Release Pipeline for Mobile App Development
Last December, LinkedIn released Voyager, the service’s mobile app. But for engineers, the intriguing part of this release came out this month on the LinkedIn engineering blog. The hidden secret of success, it turns out, is all about the testing. And automating the testing.
They’ve rebuilt the new Flagship app from the ground up. “The new app also fully realizes our multi-app strategy by bringing more focus to the main flagship app,” announced Joff Redfern, vice president of product, in a statement. “And with the new app launcher feature, it is incredibly seamless to launch into other LinkedIn apps for dedicated and deeper experiences, like Pulse, Groups, Job Search, etc.”
That’s a lot of integration. Voyager needs to be in synch with each of these LinkedIn apps, the other teams of engineers working behind the scenes, as well all of the testing necessary for each change in the mobile app itself. “If that sounds like a lot of testing, it is,” said Jens Larsen, Director of Engineering, Developer Tools.
Even when mobile app code isn’t changing, other things are. Voyager has “multi-dimension connections to lots of pieces across the LinkedIn universe,” according to Larsen.
There are numerous dependencies, multiple APIs being called, data migrations on the backend, and other big data offline changes. In addition, crunching big data to provide individualized content fuels many of the app’s features. Lots of things are constantly changing, with new functionality being deployed on a constant basis.
Testing the Android UX across multiple emulators
“This is a big part of the loop,” explained Larsen in a phone interview. “The output, schema and form may change and can affect Voyager. All the worlds are interconnected through the user experience.”
In order to make Voyager work with all of the moving parts that make LinkedIn a seamless user experience, testing had to be automated.
With so many connection points, it looked like testing was going to chew up most of the engineer’s time, explained Drew Hannay, Staff Software Engineer, on the LinkedIn engineering blog. If engineers were going to be spending 80 percent of their time fixing UI bugs, then the product could not be considered a success.
Three by Three
The solution, according to Hannay, is “a complete automation pipeline for every step, from code commit to production release. This pipeline needed to be fast enough to fit inside our three-hour window, and reliable enough for us to maintain confidence in our releases.”
“We settled on a rule that we call “3×3”,” said Hannay. “Release three times per day, with no more than three hours between when code is committed and when that code is available to members.”
So the LinkedIn engineers automated testing across the board, coding processes for static analysis, code style checks, and even generating production binary and running multiple UI tests on different mobile versions. And then another set of test to make sure that the user-upgrade experience is seamless.
Not all areas fit neatly into the 3×3 philosophy, and those needed extensive work to achieve automation. One example is the work done with the localization team, who has the task of “translating and localizing the application for different parts of the world,” said Larsen.
For example, do search results display the same way on the UI whether the user language is English, Arabic, or Chinese? Or languages with special characters, or names that were unusually long or short. Arabic text, which reads right to left, was especially tricky.
The Voyager Engineers worked with the localization team to identify bugs commonly found, then coded automated testing processes to catch those bugs.
Android UI layout test
“For UI tests,” said Hannay, “we took advantage of Google’s Espresso tool for Android and Square’s KIF utility for iOS. Both of these are already quite fast, but we wanted to be able to run a large number of tests on different Android/iOS API levels and in different configurations (left-to-right vs right-to-left UI layout, for example). Similar to our distributed build concept, we developed a system for distributing our test suites across multiple machines in our CI pool and collecting the results when all of them had completed. On Android, we were also able to take advantage of running multiple emulators (up to 15!) on a single build machine, which greatly increased our capacity and scalability.”
Details of the iOS layout testing methodology can be found on this post.
Once all of the language tests were automated, the next step was to bring the testing time into the 3×3 window. This was accomplished by spreading the testing across multiple machines. Hannay writes, “The parent node builds the minimum set of binaries needed to run tests, and fires off “child jobs” to other nodes that build the remaining binaries while the UI tests run in parallel.”
To read here to find out more about LinkedIn’s testing platform XLNT.
All of this brought the testing within the 3×3 model. Once the new functionality has passed all of the automated testing, it is put into a state called “known good.” Then every 3 hours, yet another automated process sweeps all of the “known good” code into the next launch. In reality, the sweep happens 3 times per business day, as the engineers found that the optimal internal release schedule.
There are many advantages to keeping all testing in the 3×3 model, in addition to freeing up engineers to write new code. “If something breaks, you can re-stabilize very easily,” said Larsen. “The time to recovery is much smaller.”
“The testing model picks up all dependencies, API calls, etc. So when something breaks, you know it’s probably dependencies or an API call that changed in someone else’s project.”
Voyager engineers have released some of their tools in open source libraries on GitHub. including a testing frameworks in Espresso and KIF.
So as an end user, just how often are you going to have to update your LinkedIn app?
Obviously, we don’t push updates out to our customers three times a day,” said Larsen. Besides not wanting users to be continually bugged to update the app, there are technical issues to consider. New binaries can be very large and are accessed by customers worldwide, many of whom will be downloading the updates via wireless. Requiring frequent updates would be unduly burdensome for those customers, explained Larsen.
The decision about when to push out a new release through the Google and Apple App Stores is made by the product marketing team and is out of the hands of the engineers. The product team balances the need for the improved functionality against the frequency of the update schedule so as not to annoy their customers.
Larsen isn’t so lucky. “Every time I open Voyager, I get a message to update the app — but that’s because I get all of the internal updates before they are released.”
