Amazon CloudWatch Gets Feature Flags, User-Based Monitoring
Competitively aligning with other application observability platforms, Amazon Web Services has augmented its Amazon CloudWatch monitoring service with feature flags and A/B testing, the company announced at its annual user conference AWS Re:Invent, being held this week in Las Vegas.
In addition, CloudWatch has taken a step beyond synthetic monitoring of web applications, and can now offer performance data (anonymously collected) based on actual usage.
Feature Flags and A/B Testing
The new Amazon CloudWatch Evidently capability provides developers the ability to control what specific features get rolled out to an application’s users, either through feature flags and/or A/B testing.
With feature flags, a developer can test new features of an application by only rolling them out to a select set of users, to test how the feature operates with a small subset of users, rather than roll it out for the entire user base. Specific conditions for when a feature appears can be determined through a set of if-then statements. Then, as the developers grow confident with the feature, it can be rolled out over larger groups of users, monitoring the technical and business metrics along the way.
The A/B testing capabilities also offered by Evidently are similar to feature flags, but more statistically rigorous. With these services, features can be tested through randomized experiments with multiple variations, allowing the developer to test which feature works best. The success of each variation can then be statistically analyzed. Evidently provides some statistical tools to help do this: a built-in statistical engine can provide insight even while the experimentation runs, offering anytime p-value and confidence intervals.
An e-commerce website could be using A/B testing to evaluate different types of checkout buttons — different sizes, shapes or colors — to see which one brings generates more revenue, an AWS blog post on Evidently posited.
Although another AWS service, AWS AppConfig, part of AWS Systems Manager, also offers features flags, though the Evidently service offers more advanced capabilities, according to AWS. Other companies, such as LaunchDarkly and Optimizely, have long offered feature flag and A/B testing capabilities as well.
The AWS blog post listed above also provides walk-throughs of how both feature flags and A/B testing works on CloudWatch.
Real User Monitoring
In addition to feature flags and A/B testing, CloudWatch now users what AWS calls “Real-User Monitoring” or RUM. To date, CloudWatch has offered synthetic testing, or the ability to externally watch web applications for possible problems. In contrast, RUM provides a way to watch the performance of various components as they are deployed by actual end-users.
For end-users, a website’s performance can vary dramatically from the aggregated results offered by synthetic monitoring, pointed out AWS Chief Analyst Jeff Barr in a blog post explaining the new service. Factors such as browser type, browser configuration, user location, connectivity can all affect performance in different ways.
Deploying RUM appears to be quite easy: Register an application with AWS, then add a JavaScript snippet to each Web page, which will dispatch telemetry information back to AWS. You can choose what components you want to monitor.
The AWS console shows performance metrics, and how well they stand against industry standards. The data can also be fed into other Amazon CloudWatch monitoring services, such as CloudWatch ServiceLens or X-Ray.
RUM usage costs $1 for every 100,000 events collected.
