Why the Storj Cloud Storage Service Switched to Golang
The Go programming language owes its popularity to a number of factors: speed, robustness and versatility, according to the Golang advocates. In this latest The New Stack Makers podcast, we explore how distributed storage provider Storj uses Go to support its development and operations in its quest to become the “Airbnb of storage.”
Guests for this episode are Storj Chief Technology Officer JT Olio and Storj Software Engineer Natalie Villasana. The New Stack founder and publisher Alex Williams and TNS news editor Darryl Taft hosted the conversation.
Why One Storage Provider Adopted Go as Its Programming Language
Storj’s switch to Golang made sense in several ways, especially for debugging, Villasana said.
Villasana described her trepidation of making the switch to Go, but quickly realized the language’s advantages for the programming requirements at hand. Aspects such as memory management and Pointers “make it seem like it’s going to be so much more complicated,” Villasana said. “But really, Go made things much easier in the long run.”
At a previous company Olio worked at, Go served as a faster programming language than Python, which was that company’s previous programming language of choice. Olio described how, at one point, team members transliterated every line of their 90,000 line Python codebase to Go “by hand.”
“We broke all of the engineers up into groups and allocated files to each team of engineers,” Olio said. “They pair programmed, transliterating, so at the end of this process we had two identical code bases including unit tests, including serialization formats — it was identical line-by-line Python and Go.”
Olio gained an understanding of Golang’s robustness and elegance during his Ph.D. research days. “It felt like before we were trying to build a person out of just muscles, and once we had the type system, it felt like there were bones to hang the muscles on there. It was like structure that the compiler was enforcing,” Olio said. “Our velocity for a bunch of things that we had expected would take months in terms of refactors were done in days. It was unbelievable.”
When Olio joined Storj as director of engineering, the company also had 160,000 lines of JavaScript. Olio felt JavaScript posed more difficulties in making the architectural changes needed support Storj’s environment and scaling goals (At that time, the company already was managing 150PB of advertised storage).
“I came in and was very interested in just sort of saying, ‘hey, let’s try out Go’… the team took to Go like fish to water, and everyone did wonderfully,” he said. “There’s more that we can talk about the Storj-specific decisions but my personal history with Go is that I was a skeptical adopter that became a massive fan.”
