Windows Subsystem for Linux Brings the Full 4.19 Kernel to Windows

Microsoft has been trying to woo full stack developers to Windows for some time. The Windows Subsystem for Linux (WSL) that runs Linux binaries natively on Windows has been a popular start. There are 3.5 million active developers using WSL and it’s been getting improvements with every six-monthly update of Windows 10, like being able to install and use multiple Linux distros at the same time and to edit Linux files from Windows without damaging them. But that hasn’t been enough to deliver full compatibility.

So for the next release, which will be available to Windows Insiders in June, Microsoft is switching to running a full Linux 4.19 kernel in Windows to support Linux binaries.

WSL 2 promises much-needed improvements to file IO performance, as well as native support for Docker, simplifying working with Linux containers. But it also enables fully Linux compatibility on Windows, making this a significant shift in what Windows can offer.

“WSL is where a lot of web developers spend their time [in Windows]. Now we have the pure Linux kernel, so you’re getting much higher compatibility,” Kevin Gallo, corporate vice president of the Windows Developer Platform told the New Stack.

“File performance has been a huge complaint; we’re fixing that. We’re delivering massive improvements to Linux Containers on Windows, with significantly better performance and compatibility,” Gallo said. “You won’t need a VM to run the Linux version of Docker so that reduces the memory footprint and improves execution speed and startup time. LCOW has been a bit clunky, but it’s going to be seamless once we get native Docker support. As a developer, you can be as close to production as possible, running Docker on WSL and debugging directly on top of that.”

How WSL 2 Works

“From an experience and user standpoint WSL 2 is, as the name implies, just a better WSL,” Taylor Brown, principal program manager for containers and virtualization on Windows, explained to The New Stack; but the way it works is very different.

“Where WSL translated Linux syscalls into corresponding Windows APIs, WSL 2 runs a very lightweight Hyper-V VM (the same type we use for Hyper-V isolated containers) with a full Linux kernel so there is no syscall translation occurring. It addresses many of the top feature requests (file system performance and ABI compatibility) and puts us on an architectural path to address many more.”

WSL 2 builds on the work Microsoft has done to make Linux containers on Windows (often known as LCOW) work well on Azure. “The lightweight Hyper-V VM, the 9P filesystem bridge and even the kernel is heavily borrowed from the kernel we use with LCOW in Azure,” a protocol developed for the Plan 9 distributed OS used to bridge the Windows and Linux filesystems for WSL.

If you use multiple Linux distros with WSL, each distro will run in its own VM. “However, these VMs use virtually allocated memory, start up in under a second and stop even faster so the experience and overhead is that of WSL (or even better),” Brown says.

Performance will be better than WSL, especially for files. “The IO pattern of Linux is quite different from Windows which is what NTFS was optimized for; NTFS is amazing at large file IO, so it’s great for databases where ETX4 is great for smaller IO. In WSL 2, the Linux filesystem is now just using EXT4 directly over a virtual pmem device, so we get Linux native filesystem performance on the Linux side and Windows native filesystem performance on the Windows side (best of both worlds). And with the 9P bridge, we can connect them together for a great integrated experience.”

Working with WSL 2

Despite the changes, the experience for developers using bash and other Linux tools will be the same as in WSL today. “The functionality developers love from WSL (like sharing files/folders) is fully maintained by use of a filesystem bridge using the 9P protocol, so WSL 2 sees all your Windows files and folders and you can see your Linux mounts from Windows as well,” Brown confirms.

Cron jobs and persistent daemons will work the same as with the current WSL, but there will be some changes to network sockets, which are still in development. “Since we are using a VM there is now a NIC in WSL 2. We used our NAT network mode so that NIC is fully managed and coordinated by the host, but it does have its own IP. What we are working on now is automatically mapping the sockets so that experience will be the same as WSL but we feel that for a preview the current experience is understandable and gets us really valuable feedback.”

Image courtesy of Microsoft.

WSL 2 gives Windows full compatibility with Linux, at least at the kernel ABI layer, but Microsoft will be looking to the developer community to find out what kernel modules should be included in WSL. “iSCSI targets which didn’t work in WSL do work in WSL 2 but required the proper module to be included,” Brown explains.

WSL 2 will initially be available for Windows 10 through the Insider program, so developers can try it out and give feedback about features like kernel modules. Brown notes that there’s also still work to be done around the 9P performance before it’s ready for general use, as well as the automatic mapping of network sockets.

The plan is to bring WSL 2 to Windows Server in time as well, and the Linux kernel on both will be automatically updated and serviced like other Windows components. You’ll be able to use WSL 2 in a VM on Azure thanks to Hyper-V’s nested virtualization; but while it’s based on the LCOW work done for Azure, WSL 2 itself isn’t being used by the Azure platform.

Visual Studio Code and WSL 2

Tools like Docker will be able to build on WSL 2. “Today Docker Desktop creates and manages their own Linux VM; with this experience, they will be able to just directly use WSL 2 improving performance and reliability,” Brown says.

Native Docker support will also simplify remote debugging with Visual Studio Code for WSL and Docker containers. The new Remote Development Extension Pack runs a set of development services in the environment you’re targeting that Visual Studio Code can connect to; those services install workspace extensions that run in the remote environment rather than the local Windows environment, so they can detect what languages and runtimes are available and provide the appropriate code completion and linting in the IDE.

“Visual Studio Code remote debugging allows you to connect directly to an existing container, and today Visual Studio Code on Windows can run against node.js in Windows but not against WSL,” Gallo explains. “If you launch Visual Studio Code in the Linux environment it will debug that Linux environment; before you had to launch a VM but now it connects directly into the environment it launches from.” Visual Studio Code commands and extensions run directly in the Linux distro you launch it from so developers don’t have to worry about path issues, and you can edit files in both the WSL and mounted

This is a feature Visual Studio Code users have been asking for; the third most commented issues in the Visual Studio Code repository is to support launching the IDE from bash in WSL and it works with the current version of WSL and with WSL 2.

Linux on the Windows Desktop

The container support that WSL 2 depends on wasn’t available when Microsoft first wanted to bring Linux binary support to Windows, so the team had to start by adding Linux syscall support to the Windows kernel for WSL. That was always going to have some disadvantages, Brown notes.

“One of the concerns we had with the syscall translation approach was being on a treadmill — always following the Linux kernel to create new translations — and more specifically that some syscalls were hard to get right and at some point in the future new ones could emerge that would be impossible. This architecture gets away from that since we are now just running the Linux kernel, but it took a lot of the innovation we built for Hyper-V isolated containers before we could fully realize this concept.”

If the new approach to WSL 2 delivers the full compatibility and significant performance improvements promised, then it will make Windows a very appealing platform for any developer whatever they’re working on.

In fact, Gallo believes WSL 2 will give developers better Docker support than they have on a Mac because they don’t need to think about running a Linux VM themselves. WSL 2 could make Windows an OS developers actively choose to work on for any cloud workload they need to build, not just when they’re coding for the Windows platform, he suggests.

“Along with all the improvements in the new terminal and the ability to develop Chromium-based web sites in the new Edge browser for front enterprise development, we think the end to end experience will be the most productive environment for developers, compared to any other environment.”