Sauce Labs: Pipeline Automation Key for Competitive Advantage
Steve Hazel of Sauce Labs Says Pipeline Automation is Key for Competitive Advantage
With the move from waterfall to agile to continuous delivery, software testing has undergone several necessary transformations, the most important of which is testing automation. For this latest episode of The New Stack Makers podcast, Steve Hazel, co-founder and principal architect at Sauce Labs joins us to talk about the move to automated testing, which Sauce Labs has been at the forefront.
The best advice Hazel has is to aggressively invest in automating your deployment pipeline from start to finish. Watching their customers over the last 10 years, he’s seen that companies who have completely automated their pipeline, start to finish, are going much faster than their competitors.
“You should have push button deployment, from ‘I wrote that code on my laptop’ to that code goes into production, fully tested, if there’s a problem detected in production, it gets rolled back, making that 100 percent automatic, including how you spin up the infrastructure that’s going to run it, absolutely everything.”You just see how much faster origination are moving who are at 100 percent over the companies that are at 40 or 60 percent automated. It’s huge.”
Founded in 2008 to provide functional, continuous testing for apps, based on using Selenium, Sauce Labs has been on the forefront of automated testing. There’s been a lot of change since then, Hazel said.
In the beginning, there were automated tools, Hazel said, but they automated the waterfall method of testing. Then, the focus was on speeding up the manual testing.
While they were successful in that, the move to agile changed the testing needs. Once the shipping moved ti every couple of months or weeks, the need for testing did not correspondingly drop. Instead, needs became stricter, and Quality Assurance (QA) testing became a huge bottleneck. The whole idea of a QA phase before you can ship just didn’t work anymore, he said. Out of that, Selenium emerged as the clear leader in automated testing for this faster world.
As the world moved to mobile, many of those ideas transferred over into mobile testing. And with the addition of microservices, things changed again. Because of the flexibility of the deployment environment, and you have full automation, rolling deploy, canary deployments, automated rollbacks.
All of which opens up new issues, and the likelihood of shipping software with errors.
“If you look at not like ‘are we saving our customers from any errors?’ because you’re never saving your customers from all errors,” he said, “if you look at instead as ‘how well are we protecting our customers from errors?’ What percentage of errors are we letting get through?’”
Testing in Production
While you always need to do testing prior to production, there’s nothing like production traffic, Hazel said. “You’re exposing your app to a greater variety of behaviors than you’re going to be able to replicate in testing.”
But testing in production is best thought of as an additional layer of testing, on top of what you’re used to doing, not a replacement.
Hazel is seeing more companies doing more automated testing, and more types of testing. As software becomes more and more mission critical for more and more businesses, continuous automated testing is critical.
What Sauce Labs Does
Testing is a big issue in an agile world. Software is being built in small packages, needs quick but comprehensive testing, and requires quick testing results.
Sauce Labs provides the ability to run comprehensive tests in parallel, so the time it takes to get through the comprehensive testing suite is quick.
“What we provide is a cloud-based test execution environment that will run your test massively in parallel,” he said. “Our biggest customers run thousands of tests simultaneously across different browsers, different mobile platforms and mobile versions to get the feedback in a short amount of time.”
This automation is just the basics, he said. “As you’re running these tests, we’re collecting data from the logs, the browsers, the mobile devices, screen shots, videos of the tests, test results and so on. From this massive amount of data, they are starting to build analytics features on top of that.”
The software can tell, for example, that there was a JavaScript error just before your test failed, and return the console error. It can see the failure started with a specific test run and they can show what’s different and suggest what might be broken. “When we highlight some of those things, we shorten the amount of time it takes to understand what’s wrong and what it takes to get the problem fixed,” he said. “It’s an accelerator for a lot of teams.”
When you can shorten the amount of time that it takes to find out what’s wrong, the better and the faster you’re going to move.
Security in Automation
In short, the more automated your pipeline is, the more secure your system, said Hazel. Since most pipeline automation includes automatic upgrades and running tests with every browser or mobile device configuration, so the old versioning and control issues are just not a problem. It also means that all of your code is running through the same tests before it’s shipped.
This is good, he said, because “you’re only as secure as your the least-checked software.” So having the pipeline fully automated is a huge benefit for security.
Listen in to find out testing with open source components, how to manage security when testing across distributed systems, what’s different about the Sauce Labs stack, and how Sauce Labs got named.
In this Edition:
2:41: Can you discuss the move from Selenium and waterfall testing, and how that path has changed over the last several years?
9:45: Testing in new environments
12:45: Can you talk a bit about testing, the integration of open source services, and what organizations need to think about when using them?
16:56: Are there parts of Sauce that are open source?
20:30: Exploring new layers of testing in production
22:12: If you could impart one piece of wisdom to listeners, what would it be?
Feature image via Pixabay.
