AppMap Releases Runtime Code Review as a GitHub Action
In its quest to help developers eliminate runtime code flaws earlier in the process, Boston-based AppMap has made its open source runtime code analysis tool available as a GitHub Action called Runtime Code Review.
Though the company previously focused on rooting out design flaws, security holes and performance issues as an extension for VS Code and JetBrains, this first CI release aims to more closely meet developers where they work: in the pull request.
“AppMap in CI [Runtime Code Review] is automatically reviewing incremental changes in PRs for behavioral changes, and it is incorporating all the other incremental changes that are occurring on other branches at the same time. It is fundamentally a different set of data than what is in the code editor alone, which is a branch of one person’s work,” said Elizabeth Lawler, AppMap CEO.
Though based on the eponymous previous tool, “We did a bunch of cool new things that were never available before because we can look over sequential pull requests to get new kinds of insights about behavioral changes over time, and drift,” she said.
Analyzing Runtime Code Behavior in a Pull Request
Pete Cheslock
Elizabeth Lawler
“The way that people are solving this now would be using production observability tools,” said Pete Cheslock, AppMap’s head of growth and community.
“And then you see people having million-dollar Datadog bills, because they’re just trying to catch [it] whenever it’s live. They’re kind of flying blind on the local development, and so they push it on production, but everyone knows it’s way too late.”
Runtime Code Review evaluates every code change to help developers understand the impact on the code’s runtime behavior.
Starting with a baseline GitHub workflow set up in a branch as a testing environment, it records code execution traces and compares runtime code behavior before and after a code change. Unlike static code analysis and AI-based code review, the review is dynamic.
Changes can be tested in a pull request before merging any code changes to the mainline branch. The analyzed traces can then root out the causes of issues that only appear at runtime, such as:
- Failed tests with direct links to the line of code that caused the error, as well as source and behavioral sequence diffs.
- API changes such as new, deleted or modified routes or revised response attributes.
- Security flaws such as deprecated cryptography algorithms, improper session management or unauthorized access.
- Performance issues like N+1 SQL queries, slow function calls and sluggish HTTP requests.
Those problems can then be displayed as interactive visual maps explaining the behavior causing the issue.
It leaves a comment in GitHub, including links to the problematic code. None of the data is sent to a third party; it’s stored as artifacts in your GitHub build system.
Changes are represented visually as a “code diff,” a comparison of the code before and after a change that highlights the difference.
In the IDE, you can see a sequence diagram for the branch you are working on, but it doesn’t provide you with a comparison of before and after and calculate the differences. It reflects the code you are working on the current state — and only your own work.
In creating the CI tool, the company had to figure out how to represent changes would affect the application’s behavior, and across branches.
“It isn’t only the instance of new containers or code appearing, although that is one kind of change,” said Lawler. “It includes function pathways appearing or disappearing, new API routes and removal of API routes, and changes to queries. So how do you reflect that to a user who needs to review the code, and then how do you zero in on changes that matter and surface those?
“In order to depict this for a code reviewer, we had to invent a new visualization to anchor the understanding — a behavior diagram “diff.” We show this in sequence diagrams, runtime dependency maps and other visualizations. These are now attached to each pull request to show a difference in behavior before and after the code change.”
Another example issue is looking at the incremental drift of performance.
“Often performance degradation is death by a thousand cuts — there is typically not one major issue, although sometimes there can be. Appmap’s performance analysis in CI is more sensitive to drift over sequential releases,” she said.
Based on Code Editor Extension
The core open source runtime code analysis tool AppMap, written in JavaScript, works with web applications and API frameworks, such as Rails, Django, Flask, Express and Spring. It has specialized features for mapping routing, view templates, HTTP client requests, caching, authentication and SQL. It is compatible with Ruby, Java, Python and Node.js.
It involves the use of lightweight, language-specific agents installed in the code to record how functions, web services, data stores, security, I/O and dependent services all work together when the code runs. The agents record how the code executes in the development environment without requiring custom instrumentation like OpenTelemetry or other types of platforms. The trace data is then saved as JSON files that are converted to interactive diagrams that users can navigate and search. It also produces flame graphs to help developers understand how the code performs.
It also includes an integration that enables developers to automatically create OpenAPI specifications for an app. AppMap also has included features in the code editor to work with AI coding assistants, including the ability to work with ChatGPT 3.5 and 4.0 directly.
Lawler said that when she started building the company, she called two people, one being Adam Jacob, co-founder and former CTO of Chef, who is working on his own project, System Initiative, for understanding the interconnected relationships in complex systems. The other was Grady Booch, who developed the Unified Modeling Language (UML) as a visual way to describe complex systems.
“I said, ‘Did you know, it’d be really great if UML was an on-demand concept that could explain to you what you were doing wrong and be a thinking partner?’” Lawler said.
“If I look at what Adam[Jacobs is] doing with System Initiative, it’s all about context, communication, understanding of complexity, understanding of interconnected relationships in complex systems. It’s what’s broken, it’s why stuff breaks down. … I think everybody kind of recognizes that’s the problem. It’s not the velocity of the code. It’s the velocity of understanding complex systems. That’s really what’s breaking down.”
