CI/CD / Culture

A CIO’s Guide to Automated Testing

4 Apr 2019 9:00am, by

Derek Choy, CIO, Rainforest QA
At Rainforest QA, Derek Choy is responsible for driving technical innovation and scaling a globally distributed engineering team, as well as the Product and Professional Services teams. Choy joins Rainforest QA from Aria Systems, where he was VP of engineering, scaling a similarly distributed engineering team by 10x and supported revenue growth of more than 20x. Prior to Aria Systems, Choy was director of software development at eBay, where he led product development for its billing platform.

Barely a month goes by these days without another dire prediction about automation. This time it was the Brookings Institute, which estimates that a quarter of all U.S. jobs are at high risk of automation in the coming decades. That may sound a long way out, but virtually all businesses are looking for ways to automate today, primarily to cut costs, reduce human error and speed time to market.

Software is no exception, and if you’re a CIO worth your salt you should be looking for ways to automate at least some aspects of your organization’s software testing. Testing accounts for as much as 60 percent of total development costs, so automating at least part of it could yield significant savings. But automation doesn’t work in all situations, and trying to eliminate manual testing for all software development can lead to higher costs and poor results. Human QA teams still play an important role.

So how do you decide which tests to automate? In this post, I’ll look at where automation works best, then explore a few areas where it doesn’t. Hopefully, you’ll end up with a good idea of where to apply automation in your own organization to get the best results.

Stepping back a moment, the overall goal is to build an automated test once that can be used repeatedly in the future with minimal extra work. The repeatability aspect is key: automated tests aren’t simple to write and the initial investment can be high, but sufficient use over time ensures the benefits will outweigh the costs.

Stability and Simplicity

If your dev team is working on an app that will evolve quickly, or if you anticipate frequent significant changes to the app, it’s probably not a good candidate for automated testing. The cost of making updates each time the app evolves will outweigh the benefits because you’ve lost that crucial repeatability aspect.

Complexity is another issue, and this comes into play especially when UI elements are involved. Automating tests for backend apps is generally easier because processes are more logical; less depends on human interaction. Front-end apps can be more difficult. None of this means you shouldn’t automate testing for a complex app, but understand that the investment will be greater, which makes stability over time even more vital.

Talent and Resources

Along with the technical considerations, CIOs must think about the human element. Is your team ready for automation or will it distract from their core job of delivering reliable code on time? Startups, for example, tend to have smaller dev teams that are waging a constant battle to get new features out the door quickly. Adding an automation project at the wrong stage can throw a wrench in the whole process. Timing is important.

So are skills. Building an automation framework isn’t necessarily harder than other types of development, but it’s a different skill and it helps to have people with experience. Do you have the budget or the bandwidth to hire them right now? If not, that’s another reason to hold off.

An effective QA strategy means striking the right balance between automated and manual testing. The right QA processes allow us to ship better code more frequently and meet a company’s goals for agile development and continuous delivery. But there’s some art with the science. Choosing the right tests to automate, and implementing automation at the right time, takes some judgment. Hopefully, the considerations above will help you make the right decision.

Feature image via Pixabay.

A newsletter digest of the week’s most important stories & analyses.

View / Add Comments

Please stay on topic and be respectful of others. Review our Terms of Use.