How To Do Microservices Integration Testing in the Cloud
Testing applications inside of enterprises has long been a largely lab-based affair. Twenty years ago, a room full of Mercury licenses and server replicas would have been expected from a testing team. Today, it’s almost impossible to replicate data center environments, let alone cloud services-based architectures.
What’s a lowly tester to do when he or she cannot build their own testing systems to be as complete as the production systems? There are multiple types of solutions, but each has its own holistic bent on the job of integration testing. One solution long espoused by CA Technologies is service virtualization.
Alex Martins is chief technology officer for continuous quality testing at CA Technologies. He said of integration testing that, “There’s a lot of experimentation going on because it is a real challenge. What I see in most enterprise customers is people trying to bring some level of control into their pipelines, and the low hanging fruit towards that is to use service virtualization as one technique that is quick and dirty to isolate what you’re testing. If you’re testing on-premise and it needs some services in the cloud, then you isolate the services and don’t worry about the cloud,” said Martins.
That’s just the beginning, however. “The next challenge they hit is that they need diversity in those virtual scenarios. I need a lot of different types of data and different data correlations I can test against. That’s where test data starts getting into the conversation. Pretty quickly they realized its costly to replicate a real test environment with all the connectivity and all the dependencies in a cloud-based setting,” said Martins.
“Their systems — although some are cloud-based — they’re not set up in a way that they can roll out changes to multiple regions or to multiple types of audiences or users. It requires a different type of mindset and technology to do A/B testing in production. Expecting a bank to do that… it takes a lot for them to do that for obvious security reasons,” asked Martins.
Martins feels that, primarily, this problem will be solved by management and process rather than tools. “I don’t think we need more tooling. We need a different way to manage production. If we look at the more modern company, and their movement of code from production environments to the hands of users: They don’t look at production as this big event where we cut things over and another team handles it and really doesn’t understand what the code does. It’s really more of a mindset change, and a process and culture change. The tools are out there today, building environments into cloud. Leveraging containers and proper orchestration is one, start but it takes a lot of effort to make it happen and to make it get to any sort of ROI, but it’s possible. The technology allows you to do that today. It’s not trivial, but the tech is there,” said Martins.
Real World Testing
Matt Swann, CTO at StubHub, said that testing in production is part of his company’s daily routine. He had some tips and tricks to share around the idea of integration testing. We chatted with him via email to discuss just how his teams are able to test in production.
How does a team move from standard testing, to testing in production?
Testing is an overloaded term so let’s start with some quick definitions. We think about testing and learning in the following ways:
“Pre-build hypothesis testing”: We are implementing a process based on Lean UX, focusing on obtaining feedback as early in the cycle as possible, much of which will happen “pre-build.”
“Light-weight evidence-based optimization” [uses] a tool such as Optimizely, where we can experiment both on front end and server side modifications.
Each of these processes has the following elements: Observation, Hypothesis, Experiment Parameters, Metrics (to determine how we will know if we were right/wrong), and Next Steps (what we will do if we are right, and what we will do if we are wrong).
Moving teams from “standard testing” to new methodologies requires clear communication of the new process and helping teams understand the roles and benefits. Start small and collect some wins. Start with one or two teams where you invest time training and supporting, collect evidence (the wins and learnings) and adapt the process to fit the organization. As your initial teams begin to thrive, roll out the changes to other teams.
How do you ensure production-based testing doesn’t damage the line of business?
The important part here is making sure that the team is focused on getting “signals” and learning as quickly as possible. Rather than implementing large, time-consuming features and then learning (large effort — high risk) the team needs to learn to get data as early as possible that our hypothesis is correct (small efforts — small risk). When you are right, keep going and perform another test to learn more. By ensuring keeping tests small and focused you minimize the overall risk to the company.
Why test in production?
Our testing philosophy is based on the importance of always keeping the customer at the center. As technologists, we can fall into the trap of assuming that we know what is best for the product, but that kind of thinking can be dangerous. Only by getting customer feedback and by falling in love with the customer’s problems will we be able to create the space for innovation and build remarkable customer experiences.
What aspects of traditional testing still remain while testing in production?
One thing that remains consistent with both types of testing is clear communication of the process and ensuring teams understand the roles and benefits.
