The Warewulf 4 cluster-provisioning system was recently added to the OpenHPC collection of open source tools for high-performance computing (HPC), alongside (and, eventually, supplanting) legacy Warewulf 3. If you’ve been waiting for downstream adoption before switching to Warewulf 4, now is a great time to get to know what’s so exciting about next-generation HPC provisioning.

Simpler and Easier to Maintain

Warewulf 3 was a complex beast, with a codebase written mostly in Perl that had evolved over many years. Warewulf 4 revisits many of the fundamental design decisions, making a final product that is easier to administer and maintain:

• Warewulf 4 has been rewritten in Go, providing modern safety and usability features while enabling integration with a growing ecosystem of relevant libraries.
• Warewulf 4’s configuration is all plain-text/YAML now, and all data (e.g., files for deployment to cluster nodes) are stored as files on the Warewulf server, giving more transparency to the cluster administrator.
• Warewulf 4 is a stateless provisioning system throughout and has removed previous attempts to support stateful image deployment, simplifying the code base, feature set and interface. (That doesn’t mean your nodes have to be diskless, or even stateless, though!)
• Warewulf 4 no longer assumes the same tight integration between the “head node” and its server that, in previous versions, required keeping them in tight sync for kernel versions and driver modules.

Translation Guide

Warewulf 4’s interface and, in many ways, its underlying architecture have been redesigned. So here’s a translation guide to get you started if you’re coming from Warewulf 3:

• All of the previous Warewulf commands (including wwsh, wwbootstrap, wwmkchroot, wwmngchroot and wwvnfs) are replaced with a new command-line interface, wwctl. Unlike wwsh, wwctl does not operate as an interactive shell, but it does consolidate all of Warewulf 4’s administrative capabilities into a single command. (That single command does have multiple sub-commands, and each is documented with a man page and –help.)
• Like Warewulf 3, Warewulf 4 builds its configuration on the concept of a node, which consolidates the configuration for each individual cluster node in a cluster. Warewulf 4 adds node “profiles,” which contain the same configuration parameters as a node but can then be applied to multiple nodes. Multiple profiles may also be applied to a single node, allowing a node’s final configuration to be built from remixable parts.
• Like Warewulf 3, Warewulf 4 distributes pre-built node images via a network boot process. However, Warewulf 4 has renamed its images from vnfs (“virtual node file-system”) to containers, emphasizing Warewulf’s new integration with the OCI ecosystem. (More on that in a bit.)
• Warewulf 3 supported importing files into its database for provisioning to cluster nodes beyond the node image itself. Warewulf 4 replaces the management of discrete files with its new “overlay” system, which groups related files for application either during system boot or periodically at runtime.

Import from OCI containers

Warewulf has always been, at its core, an automation system for building, maintaining, and serving node images to cluster nodes. The process for building these node images has historically been somewhat cumbersome and quite manual:

1. Make a chroot directory
2. Use dnf or yum to install packages in the chroot
3. Copy files into the chroot environment
4. Shell into the chroot environment and make configuration changes

While much of this process borrows from more general system administration procedures, even in the best case, you’re left with a thick directory full of manually built, unversioned, tenuously maintained states. This process must be repeated for each new image, at each site, with only documentation serving to aid reproducibility.

Warewulf 4 replaces the entire image initialization process with wwctl container import, which initializes new node images by importing them from upstream OCI container registries.

Want a cluster node based on Rocky Linux? It’s as simple as:

Then you can shell into the image and make configuration changes as before, or you can extend your use of the OCI ecosystem by defining your customizations in your own Containerfile. This Containerfile is easily maintained, version-controlled and shared, allowing you to track changes to your node images over time and rebuild them reliably and trivially on-demand.

Manage Kernels Together with Node Images

There have also been improvements to kernel management. You no longer have to import kernels into Warewulf by hand: instead, Warewulf 4 automatically detects and uses the kernel from the node image itself.

This allows kernels to be managed within the image, using the same kernel packages that you would typically use when deploying the OS. This also has the side effect of simplifying the integration of out-of-tree drivers and other kernel extensions: modules built inside the container build against the kernel inside the container as usual, and that kernel is used by Warewulf during boot, with no manual synchronization required.

Customize Nodes with Overlays

As we mentioned before, Warewulf 4’s overlay system provides remixable, per-profile and per-node customization of provisioned cluster nodes at system boot time and periodically at runtime. But the overlay system goes beyond static files, supporting dynamic templates that are rendered with input data relevant to the target node or even the entire cluster.

With this functionality, it’s easy to write a single configuration file template that applies to all nodes in the cluster or even to write a configuration file template for a master node—say, a queueing system server—that draws on Warewulf metadata to automatically and dynamically define cluster-wide configuration.

The Future

Warewulf 4 is under active development and is growing quickly, adding significant features with each new release. You can download the latest release of Warewulf 4 from GitHub. We also hope you’ll join the community, either via our active Slack team or mailing list. And, of course, be sure to check out the Warewulf website, where you can find documentation and more, and the blog, where we’ll continue to post tips, tricks and information about new features in Warewulf as they are released.

Jonathon Anderson has 16+ years of experience in Information Technology, including high performance computing, high throughput computing, parallel, distributed, and local file systems, local area networks, and IaaS and SaaS cloud services. Jonathon previously has served at R1 research institutions...

Read more from Jonathon Anderson