Simple Load Testing with GitHub Actions
In this article, we show how to use GitHub Actions to load-test, benchmark and validate HTTP and gRPC services with service-level objectives (SLOs).
When developing a new version of an HTTP or gRPC service, it is desirable to benchmark its performance and to validate that it satisfies desired service-level objectives (SLOs) before upgrading the current version. We describe a no-code approach based on GitHub Actions that can be used to automate such testing at any point in a continuous integration/continuous delivery (CI/CD) pipeline. For example, at build time it can be used to validate the new version as soon as possible. Alternatively, at deployment time it can be used to validate SLOs in the production environment.
HTTP Load Testing with the Iter8 GitHub Action
The Iter8 GitHub Action, iter8-tools/iter8-action@v1, enables automated Iter8 experiments in a GitHub workflow. To use the action, specify an experiment chart and its configuration via a Helm valuesFile. No programming is necessary — all configuration is declarative. Typical use is to define the test configuration in a values.yaml file and call the Iter8 action passing the configuration. For example, to run a load test against an HTTP service, use the built-in load-test-http chart. For example:
Example use of Iter8 action to run a load test against HTTP endpoint http://localhost:8080 and validate three service-level objectives: Error rate must be 0, mean latency must be below 100 milliseconds and the 97.5th percentile latency must be below 200 milliseconds.
All available options for the load-test-http experiment chart are documented in the chart readme.
Additional examples of the Iter8 action for testing HTTP services can be found in the Iter8 end-to-end test cases. To run these test cases:
- Fork the Iter8 docs repo: https://github.com/iter8-tools/docs to your organization, myorg.
- If necessary, enable workflows on your forked repository (by navigating to https://github.com/myorg/docs/actions).
- Select the workflow end-to-end tests.
- Run the workflow using the Run workflow button.
- When the workflow is completed, you can inspect the test definition (
test.yaml) and logs for the load tests against a local httpbin service,local httpbin tests.
Benchmarking
When the Iter8 action executes, it logs a list of benchmarks that can be inspected later. A sample benchmarking report is:
SLO Validation
By default, the Iter8 action validates any specified SLOs and the action succeeds only if the SLOs are validated. In this way, a workflow using this action can be certain that it is safe to promote the new service. However, if the goal is only to benchmark the service, the optional flag validateSLOs can be set to false. In this case, the action will not validate SLOs and will succeed when the experiment completes.
gRPC Load Testing with the Iter8 GitHub Action
gRPC services can be tested using the built-in load-test-grpc chart. The Iter8 end-to-end test cases contain an example. Run as described above, but inspect the logs for local grpc tests. The full range of configuration options for this chart is documented in the chart readme for the chart.
Conclusion
The Iter8 GitHub action simplifies the execution of performance testing for HTTP and gRPC services. Configuration is declarative; no programming is necessary. The action generates and applies load to a target service. It measures and records performance metrics that are used to benchmark the service and to validate desired SLOs. In a GitHub workflow, it can be used to ensure that new versions are promoted only when it is safe to do so.