A Workflow for Component-Based Accessibility Testing

Components are a significant force in software development now, but can you use them in conjunction with accessibility testing? The answer is probably yes. Here’s an approach that’s worth trying.

Before You Start: The Five-Minute Rule

Before we walk you through the workflow, let’s address a particular digital-accessibility elephant in the room: the need for speed. One common concern about building accessibility tests into a development process is that adding accessibility scans will bog things down and make it harder to reach release goals.

A solid strategy to avoid that is to create a company policy requiring feedback within five minutes. For a product or workflow to pass muster, developers must receive feedback from unit, functional and accessibility tests within five minutes of submitting a pull request. This time frame allows the developer to meet release velocity needs, easily maintain context and keep frustration low.

Benefits of Component Libraries and Design Tools

We believe that a component-based approach is intrinsically faster. These five benefits make components and component libraries an ideal match for testing:

1. Feedback clarity. Tests created for individual components are inherently both short and atomic by design. Atomic tests validate a single feature or function. For example, if an atomically designed test for, say, a website’s search functionality fails, one can be confident that it is the actual search functionality that is broken, not a problem in the setup or teardown step of a longer flow.
2. Short and stable tests. If we’re trying to get developers near-immediate feedback, it helps to have short tests that are more stable and run quickly. Should a commit introduce a bug, you’d want to know immediately, not a few hours later!
3. Less noise. Teams always strive to reduce noise, obtain clear results and make interpreting results easier. If component testing can be disconnected from needing a server-side API or any external resources, that will help.
4. Efficient error-state setup. With a component library, an error state can be loaded quickly and tested with no setup or teardown steps required, dramatically reducing test execution times. Whereas within a full staging environment a user must automate a flow to drive the application to a particular state. For example, to verify that an error message appears correctly, a tester must load the site, navigate to the appropriate page and trigger the error before checking the assertion criteria. This process takes too long.
5. Fast local connections. When using a component library for testing, all connections are local, which means no more waiting to log in, or worrying about internet connection or back-end bugs.

What’s more, a component-based approach to testing is especially powerful when considering digital accessibility. Why? Because if each component is accessible, then the site as a whole will be pretty close to accessible as well.

Workflow Details

Consider using something like Storybook to manage components and their various states. That way, you can create a static website that contains all different components and is rendered in isolation, offering a distinct advantage over traditional staging environments.

Now, let’s look at workflow details with respect to test design, build process, gating and interpreting results.

Test Design

For functional and accessibility testing at the same time, you’ll need a test framework, like Cypress, and some kind of accessibility test SDK. Once installed, it’s easy to add accessibility scans to your functional test. For example, using Cypress and the Evinced Automation SDK, you would simply add the cy.evStart() and cy.evStop() commands to the before and after hooks of each test:

As an extra benefit, some accessibility automation SDKs use a kind of continuous flow technology that scans state changes and dynamically loads elements and DOM changes for any accessibility issues resulting from the test’s interaction with the component.

Build Process

Consider integrating a CI/CD system, like CircleCI. Then each time a developer pushes code, the CI/CD system listens for new branches and runs several checks before marking a pull request as safe to merge. These include linting, unit tests, functional tests and accessibility checks. Make sure all of these checks are kicked off in parallel to minimize execution time and ensure developers get feedback as quickly as possible.

Gating and Assertions

It’s very helpful if your accessibility testing SDK supports gating. For example, a test could fail if a new critical accessibility issue is detected, or whatever logic your organization finds most appropriate. In the example below, from the Evinced Automation SDK, the assertion is expecting to see no critical issues after the scan has been completed.

Now that we have access to the results object, any number of assertions is possible, depending on the flexibility of your automation SDK. You could potentially assert based on the number of issues, issue type, issue severity, Web Content Accessibility Guideline (WCAG) tag or even specific element.

Interpreting Results

When an issue is discovered, the report from your automation SDK should contain everything needed to resolve the issue quickly. This includes the issue description, element locator, broken DOM snippet, WCAG success criteria and recommended fixes. The atomic nature of component testing ensures that issue reports are easily tracked directly to the component being tested.

Summary

Your mileage may vary, but components might be an essential way to build accessibility into your development flow. It’s a way to “shift left” on accessibility, resulting in a faster, cleaner, more accessible product, especially between audits. That’s a win everybody should have.