Our WebAssembly Experiment: Extending NGINX Agent
This is the second in a two-part series. Read Part 1 here.
At NGINX, we’re excited about what WebAssembly (Wasm) can offer the community, especially in regard to extensibility. We’ve built a variety of products that benefit from modularity and plugins, including NGINX Open Source and NGINX Plus. This also includes open source NGINX Agent, which is a companion daemon that enables remote management of NGINX configurations, alongside collection and reporting of real-time NGINX performance and operating system metrics.
NGINX Agent is designed with modularity in mind, and it’s written in a popular and Wasm-friendly language: Go. It also uses a publish-subscribe event system to push messages to cooperating plugins. Its current stage of development, however, limits plugin creation to the Go language and static linkage.
Seeing as NGINX Agent is designed with a powerful and flexible architecture, we wondered how we could improve the developer experience by experimenting with an external plugin model (caveat: not as a roadmap item, but to evaluate the ergonomics of using Wasm in a production-grade system).
The choices available to us are wide and varied. We could directly use one of the many runtime engines in development, build some bespoke tools and bindings, or adopt one of the burgeoning plugin software development kits (SDKs) developing in the community. Two such SDKs — Extism and waPC — are compelling, active, excellent examples of the growing ecosystem surrounding Wasm outside the browser.
The Extism and waPC projects take complementary but different approaches to embedding Wasm into an application. They provide server-side SDKs to simplify runtime interfaces, loading and executing Wasm binaries, life-cycle management and server function exports, while also expanding the language set available to the programmer.
Another project, Wasmtime, provides APIs for using Wasm from Rust, C, Python, .NET, Go, BASH and Ruby. Extism has expanded on that set with OCaml, Node, Erlang/Elixir, Haskell, Zig. It also provides an extensive collection of client-side APIs, referred to as plug-in development kits (PDKs). The waPC project takes a similar approach by providing server-side and client-side SDKs to ease the interaction with the underlying runtime engine.
However, some significant differences remain between Extism and waPC. Here is a basic comparison chart:
| Extism | waPC |
| Helper APIs (e.g., memory allocation, function exists) | Fewer client-side APIs (cannot access memory) |
| Direct runtime invocations | Abstracted runtime invocations, indirect server and client APIs |
| Single runtime engine | Multiple runtime engines |
| Host function exports | Host function exports |
| Complex routing input and output system | Simplified inputs and language native function output |
| High number of server languages | Limited server language support (Rust, Go, JavaScript) |
| High number of client languages | Limited client language support (Rust, Go, AssemblyScript, Zig) |
| Required C namespace code | C namespace and bindings hidden behind abstraction |
| Early, pre-GA development releases | Early, pre-GA development releases |
| Active | Active |
| Smaller backing group | Used by dapr with larger potential backing |
| Configurable state through supported APIs | Durable state must be passed via custom initialization stage |
| Basic hash validation | No bytecode custom validation |
| Host call user data supported | Host call user data unsupported |
Depending on your use cases, either Extism or waPC may be a better fit:
- Extism supports only one runtime engine — Wasmtime; waPC supports multiple runtime engines and is more configurable.
- Extism allows calls directly to the exported symbols from server and client sides. The waPC project builds an abstraction between the server and client sides by exporting specific call symbols and tracking user-registered functions in a lookup table.
- Extism defers data serialization entirely to the user. The waPC project integrates with an Interface Definition Language (IDL) to automate some of the serialization or deserialization chores.
We extended NGINX Agent with both projects and used Wasmtime as the exclusive engine to keep things simple. With our candidate SDKs and runtime chosen, it was generally a straightforward process shunting in an external plugin mechanism.
Our process of extending NGINX Agent followed these stages:
- Extended the NGINX Agent configuration semantics to define external plugins and their bytecode source.
- Created an adapter abstraction as a concrete Go structure to shim the Go function calls to their Wasm counterparts.
- Defined the client API (Guest) as expected client-side function exports.
- Defined the server API (Host) as expected server-side function exports.
- Defined data semantics for Host and Guest calls. (Wasm’s type system is strict but limited, and its memory model is a contiguous array of uninterpreted bytes, so passing complex data requires interface definitions and serialization and deserialization utilities.)
- Finally, we wired everything together by initializing our runtime, registering our expected server API exports, loading example plugins as bytecode, validating expected client APIs, and running the mostly unchanged NGINX Agent core code.
The diagram below shows the high-level data flow for the plugin components using Extism. It differs slightly from waPC, in that waPC brings its own abstraction between the Host and Guest systems. That said, the same conclusions can be drawn. Adding an external plugin system to a new or existing one does adopt some overhead and complexity but, for that cost, our plugins can also gain significant benefits from developer choice and portability. Compared to network latency, microservice complexity, distributed race conditions, increased security surface area and the need to protect data on wire and endpoints, the tradeoff is reasonable.
In this simplified view, you can see our shunt between the NGINX Agent core executable to the Wasm “Guest” (or client) code. We used “Go Runtime” as shorthand for the NGINX Agent system and executable. NGINX Agent, having already supported plugins, provided “Plugin Interface.” Then, we built a small shim structure to shunt between Go native calls and the respective SDK calls, such as a call to Plugin. Process simply generated a call to Extism.Plugin.Call (process). The SDK (for both Extism and waPC) does the rest of the work regarding memory, Wasmtime integration and function invocation until the client-side plugin execution. As shown in the diagram, plugins can also call back to “Host” through Wasm exports, in this case allowing for plugins to also publish new messages and events.
Wasm as a Universal Backend Control and Configuration Plane for Plugin Architectures
The Wasm landscape and ecosystem is rapidly advancing. Use outside of the browser is now more than science fiction — it’s a reality with increasingly extensive options for runtime engines, SDKs, utilities, tools and documentation at the developer’s disposal. We see further improvements coming fast on the horizon. The Wasm community is actively working on the component model, along with specifications like WIT and code-generation tools like wit-bindgen defining interoperable Wasm components, server and client APIs. Standardized interfaces could become commonplace, like we experience when writing protobuf files.
Without a doubt, there are more challenges ahead. To name one: higher-order language impedance, such as “What does a server-side Go context mean to a Haskell-sourced client bytecode?” Even so, we found our limited — and experimental — exercise of embedding Wasm into pre-existing projects exciting and illuminating. We plan to do more because Wasm clearly will play a major role in the future of running applications.
In theory, many other applications with plugin architectures could benefit from a similar Wasm stack. We will continue exploring more ways we can use Wasm at NGINX in our open source projects. It’s a brave new Wasm world for the server side, and we are only starting to get a glimpse of what’s possible. As the Wasm toolchain continues to mature and compatibility issues are ironed out, Wasm appears to be a promising path toward enhancing application performance while improving developer experience.
