How to Choose the Right Test Automation Tools (And Stay Out of Trouble)
Tricentis sponsored this post.
Figuring out which of the hundreds of available test automation tools is right for your organization may not be fun — but it is critical to get it right. Make one bad decision or tool choice and you will most likely take the blame for negating the speed gains otherwise achieved by automation and modernization across development and operations. However, if you choose wisely, you can take credit for accelerating delivery speed, minimizing business risks and even lowering testing costs.
How do you move past the flashy product demos and decide which tool will really work for your culture, your existing team members and your application architectures?
One respected rubric for evaluating tools is “Gartner’s Critical Capabilities for Software Test Automation.” Each year, Gartner analysts assess how the software testing industry is evolving, then develop a set of criteria for evaluating and comparing software testing tools in light of the latest trends. The report analyzes how the most popular commercial software tools compare versus these criteria — but you can use their framework as a vendor-neutral guide for evaluating and comparing any software testing tools you might be considering.
Top Considerations for Evaluating Test-Automation Tools
Here are the top 10 test automation capabilities outlined in the latest version of the report:
- Match to nontechnical role skills: Suitability for business analysts, subject matter experts and other non-programmers.
- Match to technical role skills: Suitability for developers and test automation experts.
- Templates and accelerators: Template libraries and wizards that facilitate testing for specific verticals and applications (e.g., SAP S/4HANA, Salesforce…).
- Change impact analysis: Identification of tests to prioritize based on code changes and risk.
- Cross-platform/browser support: Testing across operating systems, browser types/versions and mobile devices.
- Intelligent automation: Intelligent automated testing with AI, machine learning, self-healing technologies, bots and automated test generation.
- Breadth of technology support: Support for various UI and API formats/protocols across desktop, web, mobile, AR/VR, IoT and full-stack cloud application testing.
- DevOps tool integrations: Support for DevOps, agile planning and ALM tools.
- Open source support: Support for and extension of, open source testing tools and frameworks.
- Dashboards and analytics: Tracking and analysis of overall test progress, sprint-level test activities and individual tests.
Aligning Test Automation Criteria with Your Expected Use Cases
The report encourages evaluators to customize the criteria weights based on their unique environment and requirements, but offers the following four “use cases” as a starting point:
- Enterprise End-to-End Testing: Testing end-to-end business processes that might cross SAP, other packaged apps (Salesforce, Oracle EBS…), APIs, ESBs, web/mobile UIs, etc. The primary consideration for this use case is Breadth of Technology Support (18%). Change Impact Analysis and Templates/Accelerators are also important.
- Progressive web apps/response web/native apps: Automated testing on a broad array of end-user browsers and devices. With a weighting of 34%, cross-browser testing dominates this use case. DevOps Tool Integrations and Match to Technical Role Skills are the runners up.
- Continuous Testing: The ability to mitigate risks at every stage of the application delivery pipeline. For Continuous Testing, Gartner places the heaviest emphasis on match to technical role skills (22%) vs non-technical role skills (7%). Open Source Support, DevOps Tool Integrations and Intelligent Automation are also heavily weighted.
- Intelligent testing: Testing capabilities that take advantage of AI, advanced analytics, machine learning and similar technologies. Not surprisingly, Intelligent Automation is the primary factor here (25%). Match to non-technical role skills and change impact analysis are also key.
Download Your Testing-Tool Comparison Matrix
To help you compare commercial and open source testing tools, we’ve prepared a fully customizable testing tool comparison matrix. You can enter your own scoring for each tool (or use Gartner’s), specify how you want to weight various criteria and even factor in additional criteria that are important to your team/organization:
[Download the testing-tool comparison matrix]
More Resources for Selecting and Comparing Test Automation Tools
If you’re still not sure which test automation tools to evaluate, here are two more resources that can point you to a variety of viable options, both open source and commercial.
The Top Software Testing Tools: The Ultimate List of Lists
There are now 100+ software testing tool lists making the rounds on blogs and software testing community sites. To help DevTest professionals sort through them all, this article reviews and compares the various lists of lists — so you can rapidly research the available options and make your own decision on what software testing tools you want to evaluate. Each software testing tools list is rated on three criteria:
- Scope: The number of software testing tools covered.
- Details: The level of detail provided in the list.
- Analysis: The depth of analysis that it offers.
[Read the ultimate list of top software testing tool lists]
Gartner MQ for Software Test Automation
This annual report on software testing industry trends has become “must-read” research for software testing professionals. In addition to exploring how Agile and DevOps are altering the expectations for software testing, Gartner also assesses how the top vendors compare versus these changing expectations. Their assessment is largely based on 700+ client inquiries as well as extensive (50+ question) customer surveys and interviews. [Read the Gartner Magic Quadrant for Software Test Automation.]
Feature image via Pixabay.
