How Drone Solved ownCloud’s Continuous Integration Woes
Before adopting Drone, the cloud storage provider ownCloud was in the “dark ages” when it came to continuous integration and delivery, asserted Thomas Boerger, an ownCloud DevOps engineer.
All too many times before then, forward momentum for the company had ground to stop as the development team cleaned up old Jenkins jobs, or waited for days for a Travis CI set of tests to complete.
“It was frustrating,” said Patrick Jahns, solution architecture for ownCloud, aggrieved at all the times a feature release had to be pushed back because the tests weren’t completed. The duo presented their experience with using Drone at the ContainerDays conference held in Hamburg, Germany this week.
Moving to Drone allowed the organization to clean up many of the CI issues that plagued the company, they noted. And the software’s auto scaling feature allowed the company to dramatically increase the amount of unit and integrations tests that can be done on new and upgraded features, without increasing the amount of time each job takes — permitting the company to roll out more features while keeping costs steady.
With a mission of facilitating “open source collaboration,” ownCloud is an online service for sharing and syncing files. It is built upon an open source stack of software, including PHP, JavaScript, JavaScript, Elastic, memcached and Redis.
The cloud service is built from 1,800 repositories, residing on GitHub. Until recently the company ran its development pipeline through a combination of the community (free edition) Travis CI hosted service and both public and on-prem versions of the Jenkins continuous integration (CI) software.
Travis CI ran the DAV-based tests such as Litmus, as well as PHP-specific tests and Selenium tests. Multiple Jenkins instances, both public and in the cloud, ran unit tests of code against what the users may be deploying, including different versions of PHP, various databases, and storage interfaces such as Swift, Ceph, and Samba. Jenkins also ran the Smashbox tests.
Beset on All Sides
A growing company still on a slim budget, ownCloud encountered limitations with both of these CI tools, noted Boerger, who introduced Drone to ownCloud (he now manages the Drone namespace).
With Travis CI, developers ran into cases where something would work in development but not in production. Since there is no way to extend Travis CI with plug-ins, it was difficult for Travis CI to meet the company’s own specific needs. Developers also experienced time-outs, often due to the complicated nature of the tests. And due to the vast amount of jobs they needed to run, it could take days before a job could finish.
Admittedly, ownCloud was only using the free version of Travis CI, which doesn’t offer doesn’t offer the ability to pay for additional throughput. But ownCloud didn’t want to pay for the premium version, given the amount of work needed to done would make it cost prohibitive.
⚡️ “Container Days, Hamburg 2018” #CDS18 https://t.co/TejtZEZUgJ
— The New Stack (@thenewstack) June 20, 2018
Jenkins had its own set of issues, Boerger noted. Upgrading Jenkins is a chore, given that each upgrade destroys the job configurations. Jenkins is a hog on the disk space. One instance used ran on a dedicated machine with 32 cores and 128GB of RAM, and it still needed periodic resetting. The problem got worse as ownCloud moved to containers: Jenkins doesn’t clean up the containers after they’ve been used. “Jenkins was not made for Docker payloads,” Boerger said.
And like Travis CI, Jenkins was too slow for ownCloud’s fast-paced development cycle.
Enter the Drone
Drone is an open source container-native CI system, explained Jahns. The workspace is bootstrapped from a Docker Volume, and services, such as databases, are created from Docker Compose files. The workspace and services are then available for every step of the pipeline. “It’s all self-contained,” Jahns said: The clone step is done in a container, the build step is done in a container, and the publish step is done in a container.
Because it is packaged in containers, Drone is easy to upgrade. Because it is pulled as a static Go language binary, it was quite compact. The pipeline itself is described as a YAML file, so it can versioned. And the whole setup can be run locally or in the cloud in an identical configuration. For each test, ownCloud could simply pull the container, or create one when none was available, with the suitable environment. “Whatever environment you are testing for, you can have it as a defined stack you can test, and give to your developers to test with,” Jahns said.
For an interface, Drone builds on Docker Compose, so it is immediately familiar with container-savvy developers. It also offers a lot of extensibility: It integrates easily with GitHub (as well as other repositories such as GitLab), and, through plug-ins, can publish directly to cloud providers such as Amazon Web Services and the Google Cloud Platform.
Ready Steady Go
Initially, ownCloud rolled out Drone as an alongside its other CI tools, using Ansible to manage the set-up. The company started with a few test workloads, but quickly expanded operations. As the team learned how to work with Drone, they put more and more loads on the software. Eventually, they stopped using Travis CI and Jenkins entirely.
The best part about Drone was its ability to be easily scaled up. Because offers a public service, the company must test its code against every possible user configuration, a number that continues to grow dramatically. The creator of Drone, Bradley Rydzewski, is also working on software that auto-scales Drone automatically, based on demand. It can provision more nodes depending on demand, and when the work is finished the nodes are decommissioned. Still an experimental feature, it only supports a few cloud providers, however.
The benefits that ownCloud enjoyed with this technology are best shown through a chart that Boerger and Jahns presented to the audience. It shows how the amount of testing time per build increased dramatically — 30,000 minutes per build — increased with the adoption of Drone, even as the wait time for each build has largely remained the same, about nine hours for build time, and 45 minutes for the test results.
While continuous integration tools paved the way for faster development cycles, containerization and cloud-native computing are showing ways of further automating operations. And their gains are being folded back into the velocity that CI/CD tools are generating. Software such as Drone show the immense possibilities of this synergy that is still on tap.
Feature image: Thomas Boerger (left) and Patrick Jahns, at ContainerDays.
