Down the Right Corridor: Dynatrace Jump-Starts Cloud Foundry Unit Testing
If you’re a Cloud Foundry applications developer, how well do you test the performance of your work before you deploy it to production? A CI/CD framework may help you run a battery of automated tests on your code in a sandboxed system. But does it enable you to do unit testing?
“The process of testing in Cloud Foundry,” reads the platform’s documentation, “starts with running unit tests against the smallest points of contract in the system (i.e., method calls in each system component). Following successful execution of unit tests, integration tests are run to validate the behavior of interacting components as part of a single coherent software system running on a single box (e.g., a VM or bare metal).”
If you’re testing your software the way Cloud Foundry’s principal contributors would prefer you did, you’re testing individual functions in isolation. Or, as many developer shops are discovering, they think they are. Or, perhaps they’re performing a series of steps that seem second-nature enough that they may as well be called unit tests, for all anyone knows.
During the Cloud Foundry Summit last month, applications performance management services provider Dynatrace announced a partnership with Pivotal, Cloud Foundry’s steward. The object? Open the floodgates for performance metrics from the PaaS platform.
So What’s a Unit?
Unit testing is a methodology that some advocates of modern CI/CD frameworks don’t even mention, although long-time proponents such as Martin Fowler consider it something of a science. It is, almost, exactly what it sounds like: the process of thoroughly evaluating the behavior of a small piece of code in isolation, preferably in an automated fashion, to ensure that the logical outcome of that process always falls within the boundaries of expectations. The space in-between those boundaries is what unit testers call a corridor.
In theory, unit testing would be a perfect match for the emerging microservices architecture.
As Fowler has stated in presentations, when a developer has trouble crafting a unit test for a code module, the trouble itself can be a positive signal that the module is too big — that it should be broken up into constituent parts, not just for the unit test’s sake, but for greater efficiency as microservices. “Thus, alongside being a useful testing strategy, unit testing is also a powerful design tool,” wrote Fowler, “especially when combined with test driven development.”
While Cloud Foundry developers have taken to microservices architectures, the platform itself was arguably designed for quick, and sometimes dirty, development. Performance has not always been a factor of consideration, especially in the case of public PaaS providers whose own admins are supposed to be the ones responsible for delivering performance. So how effective can unit testing on a PaaS platform be, which so many of those performance factors are, intentionally or not, being masked from the developer?
A unit testing regimen should not be so focused on individual elements of code that it forgets about the broader context of how those elements are implemented.
“When you start talking about the performance factors of how a unit test is executed in a build environment,” said newly-installed Dynatrace product evangelist Mike Villiger, in an interview with The New Stack, “the performance metrics that are in a ‘default’ Cloud Foundry environment, or perhaps even a more traditional environment – yes, there are going to be performance metrics that are will be different between the test environment and the production environment.
“That’s why, when you look at some of the simpler, more traditional performance metrics, like response time, you’re going to build a corridor for response time, to determine what is normal in that execution environment. And you’re going to be looking for failures when you are perhaps even orders of magnitude outside of that corridor. In a unit test environment, you’re not going to have that statistical basis, like you would have in a traditional performance test where you would have hundreds of thousands of data points, where you can accurately determine what’s the 90th percentile of performance, the 97th percentile, and so on. So the statistical significance that is on the order of milliseconds is really not what you’re looking for in the unit test. In the unit test, you’re looking for those response times that are exceeding the corridor.”
Inside Job
Developers who have tested code at a very granular level may be familiar with JUnit, a unit test framework for automating repeatable tests with Java. Many development shops that use Jenkins as their CI/CD platform are familiar with automating their unit tests with JUnit. As Dynatrace’s Villiger told us, his company’s new relationship with Pivotal is made even more interesting, by virtue of its existing relationship with Jenkins’ commercial steward, CloudBees.
“Pivotal has pursued a relationship with CloudBees Enterprise Jenkins to be able to do your unit test, and the entire lifecycle, self-contained in the Cloud Foundry environment,” said Villiger, “so that at no point in your pipeline do you leave the platform that you’re going to be running in production.” It’s an agreement like this that could bring JUnit and unit testing frameworks into a kind of seamless, automated encapsulation – a way for the platform to focus on the performance of the code, as it’s being generated.
Over the past two years, Cloud Foundry has provided a kind of container-based staging mechanism for Java applications. It’s the Cloud Foundry Garden project that provides Linux containers with the isolation they need to run in a distributed fashion over the PaaS platform at scale. As Villiger reminded us, each Garden container running within a cell in the Diego orchestrator includes an instance of Tomcat as part of its stack.
Running apps in the testing phase, with this particular setup, should not be significantly different from running apps in production, Villager believes. So injecting Dynatrace into the production phase with CloudBees and Pivotal, he said, has the benefit of bringing it into testing as well – and in so doing, integrating a necessary degree of Dynatrace performance metrics into unit testing, in a self-contained model.
The n + 1 Problem
Won’t the inclusion of more performance metrics than Cloud Foundry developers have had available to them before, change their perception of where the performance corridor boundaries lie?
“In my opinion, one of the interesting things about this is that we have a new stack replicating the old problem,” responded Dynatrace’s Mike Villager. “We’ve got this brand-new architecture — we’re talking about microservices, and we’re talking about one microservice for fetching data, and we’re splitting out the different elements: you’ve got Spring, you’ve got Spring Boot, you’ve got Spring Cloud, etc. And what ends up happening in the end, you’ve got that n + 1 problem.”
I wrote about the n + 1 problem in a book on Visual Basic 24 years ago. It’s a phenomenon that can crop up when you delegate the job of fetching a record from a database to a component that composes each query on the fly. If a data table has 1,000 records, and a loop clause cycles through the invocation of queries that fetch each record one at a time, then the component generating the SQL query constrains that query to each specific record, for a total of 1,001 queries (1 for the “over-the-cliff” fetch that reveals there’s no more table left). I argued back in the 1990s that this method was a misuse of SQL; that an intermediate component should be capable of iterating the results of a “SELECT * FROM table” directive.
Villiger makes the point that with microservices, the same problem can arise whenever 1,001 services are dispatched by an orchestration platform to iterate 1,000 records. Ordinary unit testing would apply to the function that fetches an individual record, making it a faster fetch. But it would not clue the tester into the fact that dealing with the records collectively would reduce the duplication of services, effectively improving the overall performance of the application.
“To a certain extent, we’re looking at the same old problems replicating themselves in a new stack,” remarked Villiger. “Then some of the new things that are going to happen will be additional architectural regressions where perhaps a developer has bypassed a data service and attached a record directly from a database when he should have iterated through a data service to get there.” That’s a situation, he explained, where the services don’t fall in-line with one another and follow each other’s procedures — where one service skips a step or two.
Architectural regressions tend to fall outside the boundaries of a performance regression, he said — of the typical focus of a unit test. “But they are regressions, nonetheless, that developers should be focusing on.”
His point: A unit testing regimen should not be so focused on individual elements of code that it forgets about the broader context of how those elements are implemented. Villiger and Dynatrace are looking for the opportunity to inject performance metrics into the Cloud Foundry development space, but in a way that defines the performance corridor in a context one level, at least, beyond the unit.
Cloud Foundry and Pivotal are sponsors of The New Stack.
Feature Image: The ultimate unit test: Alexander Rossi just crossing the yard of bricks in the 100th Indianapolis 500, May 29. Photo by Scott Fulton.
