Microsoft Mulls Expanding Windows 10 to Support Multiple Linux Distributions

Microsoft’s latest update of Windows 10, called Creator’s Update, marks a new level of stability and support for the Windows Subsystem for Linux (WSL), the way to run Linux binaries on Windows 10. It also brings the promise of expanding beyond the current Ubuntu distro to support multiple Linux distributions in the future.

Microsoft’s Rich Turner told The New Stack that with the Creators Update, WSL is good enough to be a “daily driver” for developers who need to use Linux command line tools alongside Windows GUI tools. “We’re getting to the point where the maturity of the underlying platform is getting reasonably good, at least in terms of mainstream developer scenarios.”

But the WSL team still has plenty of improvements, some of which need changes in the underlying Windows systems.

“We know that file system performance needs to be improved,” Turner said. “We want to increase our disk I/O throughput; our disk IO performance right now is not where we want it to be. In almost every other aspect, anything to do with process or memory throughput, we’re actually as fast as Linux if not a little bit faster on the same hardware,” he claimed. “On network I/O we’re looking really good but we’ve got some extra network socket modes that we need to support. There are a couple of esoteric network tools that need particular types of socket support we can’t currently do, so we’re working with the Windows networking team to add those. We’re working with the storage and NTFS team to provide us some extra hooks so that we can make our disk storage and throughput more efficient.”

Those changes will take some time, he noted. “We will address [these areas] in the next version of Windows and then some more the version after that. The file system changes, in particular, are things that we have to take a great deal of care with, so we’re going to take our time with those, to make sure they’re done properly.”

Another area that’s frustrating for developers but also needs to be approached carefully to make sure the security model is correct is network file access. “People want to be able to mount NFS drives and ssh connections and SAMBA drives to be able to connect up to a Windows network share, for example. We don’t support arbitrary mounting right now. We kind of mount your local hard drive, but that’s ‘mount.’ We want to be able to support eventually the real mount capability within Linux itself, but we’ve got engineering and code to write to make that happen.”

WSL will also get better support for working with devices. “Developers want to build an Android build and deploy it to their phone, or they want to deploy to a Raspberry Pi module because they’re building an IoT device. Right now, we don’t support USB devices or serial [connections] but we want to be able to eventually add that as well.”

WSL doesn’t currently run on Windows Server, but that’s another possibility (again, it needs work).

Some requests would need more major changes, like being able to run persistent, background daemons and services, to make cron more useful. Currently, you can run any background services you want — but they only run as long as the bash console is open.

“Until you run Bash, no Linux process can run on your machine. As soon as you open Bash, you can choose to start any background services you want to have run. If you want to have MySQL or FSH or ssh or Postgres or Apache or whatever run, you can start them manually or autostart then with the .bashrc file,” Turner pointed out. “But as soon as you close the Bash console, we tear down any running Linux processes. So if you close the console window, you can no longer access your system via ssh, from your machine or any other.”

That’s something developers would find useful, Dustin Kirkland, Canonical’s technical lead for Microsoft Ubuntu development told the New Stack. “Users come to me from time to time and ask about persistent daemons. It’s one thing to have ssh working while the terminal is open, but it’s another thing if it’s always running.” After all, “if you’re running a Linux virtual machine with a network mapped, you can ssh into the VM as long as it’s running.

Ideally, says Turner, background services would integrate with Windows the way processes now do. “What we want to be able to do in future is figure out a decent way to have Linux processes running in the background as daemons or as services, and to be able to have those have auto-start on the machine if you want, in a manner similar to Windows services.”

Canonical would also find persistent cron useful for updating the Ubuntu image WSL uses. “On a traditional Ubuntu system, a cron job runs once a day, typically at night, that does the apt update and gets the list of packages available, and the updates that are security critical are automatically applied,” Kirkland pointed out.

That doesn’t happen with WSL, but Windows does periodically get information about available updates, Turner explained. “Right now, we have a background task that runs every five to eight days that periodically pings the apt cache and downloads the apt package index, so when you start a new session we can say what packages do you have installed and what the latest versions are and show you a message saying ‘hey, 43 of your packages are a bit behind, you might want to update.’ We don’t auto-update your distro. We will auto-update WSL itself, so we will patch the underlying implementations of the syscalls, we will patch our user mode tools, like the console itself and we’ll patch the install mechanism if necessary, but we don’t touch the internals of your Linux distro.”

“For Canonical to auto-update your system, they need the cron daemon running in the background. Once we’ve implemented that background mechanism, things like that can take place.”

Auto-updating isn’t uncontroversial. “Some customers are vehemently of the opinion that there should be no auto-update ever,” he noted. “But some, especially enterprise customers are vehement that all Linux instances must be updated to a certain patch level so that if a vulnerability is discovered in a distro, it is automatically patched and doesn’t become a problem. We want you to have both of those options, but we have to figure out the background process story to enable that.”

Kirkland also pointed out the way networking works in WSL. “Right now, the network space is completely flat, so the Windows desktop and the Linux shell are sharing the same IP address and the same set of ports. If you tried to use the same port on both of them, you’d end up with a port conflict. Microsoft has some protections in place for that but it’s an interesting consideration. Normally when you think about running one OS on top of another, each OS gets its own network space, it’s own IP address; that can be NAT’ed or bridge. But trying to collapse that TCP stack into a single network space is interesting. The team has made some good safe decisions from that perspective but it’s a complicated one to think about.”

The priorities for all these improvements depend both on community interest and what they add to WSL. “It all gets put on the stack and prioritized and we knock those things off in priority order based on how frequently we hear the request being made and how much impact we think the feature will have in the community,” Turner explained.

Some requests will need a lot more work. Support for Linux containers (as opposed to the Docker support for Windows containers that’s built into Windows 10 and Windows Server 2016) was one of the first requests Microsoft had for WSL. There are some high-level architectural similarities between WSL and what container based systems do that prompted those questions, but there are also some key differences.

“Implementing all the underlying kernel infrastructure that would be required to support containers and namespaces requires some work. It is on the backlog and at some point in the future, hopefully, we will be able to consider what it would take,” said Turner. “It’s something we do hear on a regular basis and we are interested in doing, but we have to figure how to find the time.”

Beyond Ubuntu

Officially, WSL only supports Ubuntu at the moment. Unofficially, WSL users have already been experimenting with running distros like SUSE; Microsoft has learned a lot from that work because it shows which syscalls need to be added or extended.

When work first started on WSL, Ubuntu was chosen because it was the most popular distro with the developers the team consulted. Kirkland puts that down to Ubuntu’s regular update policy for a wide range of Linux tools. “Developers using Ubuntu choose us because we provide the latest and greatest open source software on a very timely and predictable schedule. We do dozens of bug fixes and security updates every day in any one of the 25,000 open source projects which are built into 55,000 binaries that are now just one single app install away.”

Since WSL came out, though, developers have also asked for other distros like Alpine and ArchLinux and RedHat and SUSE, and more specialized distros too, Turner said. “We want this to be a distro-agnostic platform on which developers can be unblocked on doing what they need to do, so they can build the Linux portions of their code alongside their Windows code, and alongside UWP apps, and alongside everything else they need to be able to ship their mobile desktop and cloud solutions, regardless of what technology those solutions are targeting.”

“As far as WSL is concerned, we don’t even know what distro you’re running, it doesn’t know what apps are running on top of it; it just does the work that’s asked of it,” he points out. “Essentially, WSL is just a piece of kernel infrastructure that provides a layer that is compatible with the Linux kernel system call interface. When something calls it and says ‘open a file’ or ‘read from a file’ or ‘open a network socket’ or ‘allocate me some memory,’ it just allocates the memory and hands it back.”

Supporting other distros will need changes to the user mode tooling that lets you install and uninstall your distro, “because you don’t want to have to nuke an instance to replace it with another.” But first WSL needs to implement all the syscalls that the tools in all the different distros depend on.

“Each of those flavors of Linux has different idiosyncrasies; they have different installers, they have different file system layouts, different configuration systems for some tools… if you’re running a Red Hat, it might be using a different installer to Ubuntu and the installer for Red Hat might make use of a syscall that Ubuntu doesn’t, so it is one we’re missing.”

That means trying out distros to see what syscalls are missing — or what extra capabilities are needed in syscalls WSL already have.

“That will allow us to start supporting a broader set of tools and we will eventually get to the point where we feel comfortable that we can support a Red Hat or a SUSE or a CentOS or one of the other thousands of distros.”

Microsoft wants those distros to be formally supported on WSL, but not by having the distros do extra work for Windows. “What we intend to is provide enough infrastructure under the hood that those other distros can run on top without us having to do anything,” said Turner.

Kirkland welcomes that; not just because Linux is all about choice, but because it will “improve the overall quality and testing of the system calls WSL provides”.

At that point, WSL might be able to run not just more than one Linux distro — but even more than one Linux distro on the same PC. It would take a lot of work to make it happen, but it’s a common request and Turner can see the attraction for developers.

“Some people work on systems that span internal and cloud-based environments. You might have an internal system that manages information coming from customers, that’s in MySQL with a Redis cache; a lot of enterprises use RedHat and SUSE for that. But then when you have the cloud front end to that system it might be housed in AWS running on top of Ubuntu. If I have to work on things in my Ubuntu cloud interface and I also have to work on things in my RedHat or SUSE backend environment, then I need both environments to get my work done; I couldn’t do it all in Ubuntu.”

Today that would mean using at least one virtual machine, which means paying for a cloud service or having powerful enough hardware to run multiple VMs. Ironically, because WSL is a layer in Windows rather than a distro itself, it opens up new possibilities for creating an extremely flexible platform for developers that Linux itself can’t easily deliver.

Feature image by Stefan Kunze via Unsplash.