T-Mobile’s ‘Eye Opening’ Shift to GitOps
GitLab sponsored this podcast.
It is certainly never easy to port existing software production pipelines and operations to a cloud native or, more typically, to a mix and match of cloud, bare metal and on-premises server environments. This assumption certainly held true for the situation cellular service provider T-Mobile was in during its digital journey. But while T-Mobile’s operations are on a gargantuan scale of magnitude compared to most organizations, T-Mobile succeeded by applying many of the same practices many organizations rely on, whether it is a 10-person shop or Google. This includes increasing reliance on GitOps for version control (VC) and repository functions but also as a way to combine all data access and functions into a single source to help manage the enormous complexity of multicloud and server environments, including, of course, Kubernetes deployments.
One of T-Mobile’s main goals was moving “a lot of those pieces out of the traditional location where they might be in a database of inside of a certain system of record, and a service registry and more into a repository. So, we’re able to recreate things on the go to manage our user base in a more dynamic fashion,” Philip Marc Schwartz, principle engineer, software, T-Mobile, said. “It’s been an eye-opening experience to be able to work through the problems that you see with those systems as you’re trying to get them working. And it’s very interesting to see those problems solved where you can really work on them at a finite level.”
In this episode of The New Stack Makers podcast, Schwartz and Priyanka Sharma, director of technical evangelism at GitLab, discuss the underlying challenges organizations face as they rely on GitOps and other processes when making their digital transformation in a cloud native-centric environments at GitLab Commit in Brooklyn, New York last week. This episode was hosted by Alex Williams, founder and editor in chief of The New Stack.
Among the more directly quantifiable results of adopting GitOps, previously, T-Mobile’s DevOps users would have to wait a couple of days between getting a service and having full directory access. Now, with the use of GitOps repositories, the lag time for onboarding has been reduced to a 30-minute wait so users can “immediately” start opening merge requests to request access to specific sections of code base, Schwartz said. “Our goal is to get that under five minutes to be able to have users quickly ramp up to get access to the system and to immediately open up merge requests that individual teams get to approve,” Schwartz said.
In its own move to full continuous integration/continuous delivery (CD) processes, GitLab actually relied on GitOps with legacy tools before making the jump to Kubernetes. GitLab made the move to CD “the old-fashioned way” without Kubernetes “because we wanted to get started and build the sort of the working muscle,” Sharma said. Specific to GitOps, “everybody works in GitLab’s interface,” Sharma said.
Needless to say, T-Mobile heavily relies on monitoring and observability, as well as time-series databases with its GitOps to manage the incredibly massive amount of data of its massive cell phone and communications network that can total 50 billion data points. This all creates a huge need for being able to log storage and log analysis data and “trying to have tools that can work through an entire data pipeline to really manage it for us so we can see it,” Schwartz said.
The “50 billion data point” metric caused Sharma’s “ears to perk up” as DataOps and changes in data engineering in a cloud native world were key themes during a panel discussion she was about to moderate after this podcast interview during GitLab Commit. “The whole reason we set up this panel was because, as every company is a software company, the more complexity you have, the more services you have, the more data points are generated and the more GitOps you do,” Sharma said. “How do you understand how to make sense of all that information?”