How Golang Evolves without Breaking Programs

The principal engineer for the Go programming language had a message. Speaking at the annual Go conference, Gophercon, Russ Cox said many people have asked him, “When should we expect the Go 2 specification that’s going to break all of our programs?”

“The answer is never,” Cox told the audience — driving home his point with three slides emphasizing that prioritizing backwards compatibility “was the most important design decision that we made in Go 1.”

“Go 2, in the sense of breaking with the past and throwing away all the old programs, is never going to happen.

“Go 2 in the sense of being the major revision of Go 1, that we started toward five years ago, has already happened. But there won’t be a Go 2 that breaks Go 1 programs,” Cox stressed to the audience.

The plan is instead “to double down on compatibility, which is far more valuable than anything we could get from a break with the past.”

“And we’ll keep calling this Go 1.x, not Go 2, to emphasize that continuity and backwards compatibility.”

The rest of Cox’s talk detailed how hard the Go team works to avoid breaking existing Go programs when updating the language. It explored specific examples and edge-case scenarios, showing some of the day-to-day logistics of maintaining a popular programming language. So while the talk was titled “Compatibility: How Go Programs Keep Working,” Cox delved deeply into the how and why of it all.

And towards the end, he discussed changes they’re considering that would mean even fewer broken programs in the future.

Stable and Boring

The remarks were made in early October at the Marriott Marquis Chicago as part of Cox’s keynote speech for GopherCon 2022 — so Cox’s reassurances came at the event’s first in-person gathering since 2019.

“Back in the early days of Go 1, Go was exciting and full of surprises,” Cox reminisced to the audience, adding “every week we cut a new release, and everyone got to roll the dice and see what we’d broken in their code.

“We released Go 1, and its compatibility promise, to stop that excitement, so that new releases of Go would be boring.

“Boring is stable,” he told the audience. Boring means being able to focus on your work, and not ours.” And then he launched into a talk on the “important, difficult work we do to keep Go boring.”

Cox began by remembering a warning from the end of the promise: that it’s impossible to guarantee that no future changes will break programs. One scenario: if a bug is fixed, it could theoretically break code that required its original buggy behavior. “But we try very hard to break as little as possible and to keep Go boring.”

And then Cox provided some instructive examples drawn from the history of Go. For example, in Go 1.1, they’d realized time values could be returned with a more-precise nanosecond measurement (rather than just a microsecond measurement) — but that this broke some tests which hadn’t expected that level of precision. The solution was to offer a way to discard that additional precision (by truncating time values down to the older, less-precise values). And then just documenting that workaround in the release notes.

Cox said much of the work on compatibility ends up getting spent on similar problems, of correct and reasonable changes to the language — that nonetheless can still break a user’s programs.

Another example: a change made in Go 1.6 had made the sort function run faster (by about 10%). But it sorted elements of equal length in a different order than it had before, so “Programs that expect a specific output are going to break.” Cox called it an example of why compatibility is so hard. “We don’t want to break your programs — but we also don’t want to be locked in to undocumented implementation details.”

Testing at Google

Cox emphasized one of the tools that the Go team uses to avoid breaking changes: heavy testing, and also beta-testing. Every new development version of Go runs a gauntlet of internal tests — and if it passes, Google starts using it internally. If it fails, they’ve got a head start on finding mitigations — or at least knowing to document the problem in release notes.

Cox shared an example where this early testing helped find a problem early. Go 1.8 changed the data compression package compress/flate to produce smaller outputs. This still broke a project inside Google that needed to be able to create exact reproductions of its archive builds. Google’s team ultimately had to fork both compress/flate and Go’s compress/gzip packages, “so that they could keep building the same bits that they did with earlier versions of Go.”

The Go team actually does the same thing with its compiler, Cox pointed out. “We use a copy of sort so that no matter what version of Go you use, the compiler produces the same outputs.”

Cox also told a story about how an early language decision came back to haunt him later. When he’d written a package to parse IP addresses, he’d used example IP addresses that had leading zeros in parts of the address. (Like 18.032.4.011, which in Go converted to 18.32.4.11). But later Cox learned that C libraries interpret leading 0’s as the indicator for an Octal (base 8) number — so 032 is interpreted as 26, and 011 is interpreted as 9. Cox explained the dilemma — and also the solution. “This is a serious mismatch between Go and the rest of the world. But changing the leading 0’s from one Go release to the next would also be a serious mismatch… a huge incompatibility.”

Instead in Go 1.17 the team changed how ParseIP works — but in a different way. When it encountered an IP address with a leading zero, it didn’t interpret them as an Octal number, but it also didn’t discard them. Instead, it just rejected the IP address altogether. “This way if Go and C both successfully parse an IP address, they agreed on what it means.”

While this change didn’t break anything within Google, “the Kubernetes team was concerned that saved configurations might’ve used these addresses, and now they wouldn’t parse anymore with the new Go.”

And to avoid the semantic change, Kubernetes forked Net.ParseIP. Cox believed those addresses should be removed from the configurations altogether, “because Go reads them differently than most other software.

“But that should happen on Kubernetes’ timeline, and not on ours.”

So what happens if a protocol changes, and in supporting that change, older Go programs start breaking? Cox provided the example of SHA-1 certificates for HTTPS. The protocol has long been deprecated as insecure, with the National Institute of Standards and Technology (NIST) formally recommending federal agencies stop using it for most use cases in 2015. Cert authorities stopped issuing them in 2015, browsers stopped accepting them in 2017, and so Go finally removed its own support for them in 2022. (With a way to turn them back on in Go 1.18 and 1.19 by using Go runtime’s debugging variable GODEBUG.)

The original plan was to remove even that in the next release, but in the end, the Go team decide to keep it in longer. The Kubernetes team had told them some private installations were still using private SHA-1 certificates, and “putting aside the security problems, it’s not Kubernetes’ place to force these enterprises to upgrade their certificate infrastructure on Kubernetes’ schedule or on Go’s.”

“After all, we want to break as few programs as possible.”

Looking Ahead

Compatibility is such a major concern that Cox ended his talk by proposing three new ideas for helping further reduce the possibility of program-breaking changes. For backward compatibility, if there’s a potentially problematic change, Cox is proposing an additional setting in the GODEBUG variable for disabling the change. (Along with a guarantee that that setting will last for at least two years — that is, through four Go releases.) Although some settings could last even longer — or even forever, Cox adds — noting they’re not planning to remove the overrides for HTTP/2.

But there’d also be a third feature: a counter in the runtime to track how often the program really behaved differently because of the disabled change, “so that users can monitor whether they’re still making use of the old settings or not.”

And they’d even add a way to override the program-breaking behavior from within the source code, “so that a binary can have its own defaults, independent of the environment.”

It would add one more element to the Go team’s long and ongoing effort of maintaining Go’s backward compatibility just as much as they can.

As Cox described it, “we’ll keep doing everything we can — to keep Go boring for all of you.”