Digital Magic: Disney’s DevOps Transformation
There’s no doubt the 94-year-old The Walt Disney Company has embraced technology as the key factor in driving its growth in the entertainment industry. But when you are an enterprise of this age and size in the public eye, how can you scale without implosion?
During the DevOps Enterprise Summit in London this June, Disney Director of Systems Engineering Jason Cox explained his company culture and organizational structuring behind the digital magic, as well as Disney’s move to DevOps.
With a company of 200,000 employees and almost insurmountable technical debt, Cox has found orchestrating growth comes with challenges:
#1: Scale: “The digital expansion of business equals more work and firefighting,” Cox said. “We were so used to dealing with tens of servers and suddenly it was hundreds and thousands of servers,” which in turn created more work for the development teams.
#2: Speed: The engineering team realized they were unable to keep up with business needs. Cox explained this was because “What was ingrained was red tape — our business needs to get it out to our guests faster, getting that feedback loop,” and all the already in-place processes were getting in the way.
#3: Stability: Cox joked that the engineering team even name their servers and they’d all create individual personalities: Bashful would disappear from the network all the time, Sleepy had latency problems, and the whole lot was just Grumpy. They desperately needed to figure out how to get systems configuration automated and under control.
Institutionalized Shadow IT
The first step was to examine the corporate structure.
Each of the four different, immense divisions of Disney — Studios, Consumer Products and Interactive, Parks and Support, and Media Networks — had a separate, autonomous chief technology officer.
“Then each silo has multiple CTOs autonomous inside of it,” Cox said. “At Disney, we’ve institutionalized Shadow IT. Every business unit has CTO; and CTOs in every division.”
What Cox soon realized was that rapid growth meant a cross-company change. These different divisions had to find a way to talk to each other, both culturally and technically. So they embarked on a DevOps transformation.
DevOps — a hybrid of development and operations — typically refers to agile software development at an enterprise scale. The end goal of implementing a DevOps model is to improve communication and collaboration between these two business units and, often, across larger companies where technology is a driving factor.
Cox and his team were tasked with taking the principles of DevOps and applying them to the individual parts of the business. It wasn’t just a technical change, but rather a cultural one. Cox pointed out that while he and his team were approaching different divisions and departments saying “We’re here to help,” those exact words strike fear and replies of “Are you here to spy on us?”
“We aren’t there to evangelize, we’re there to listen,” he said, by breaking down silos and scaling DevOps. “Success is when we’re able to get the technologists working with each other simply to embrace new ideas.”
What were the results? With the 100 servers involved in the “Toy Story Mania” attraction, employees used to spend the eight hours of the park closed every night, manually updating each server. Now only one person can update the whole fleet in 30 minutes.
“Config management was magical to them,” Cox said. Configuration management or CM involves coordinating a systems engineering process in the name of consistency across performance and function, as well as requirements, design and operations across its lifecycle.
“You also have less system drift because you’re converging each system together, enforcing the configuration,” he said.
Colleagues began to understand more about their operations and how these fit in with the rest of the company, which allowed them to then be able to deliver continually.
In another example, there are 220 Disney stores across the U.S. alone. Each store has multiple point of sale devices. Through config management of the points of sale and converging them, each store, each employee, is able to verify that what’s out there is the right image, consistently.
But DevOps didn’t just streamline current operations, they could also help prepare the IT staff for future systems. With new features like the upcoming Star Wars park, engineers could now test the engines before they were live, with multiple builds a day in lab environments. When launching Shanghai Disneyland last year, Disney wasn’t going to have the supporting data center ready until launch day, but it was able to sell out entrances before opening day.
DevOps has also enabled the company to move to the cloud, including ABC cloud hosting and the Disney Movies Anywhere app. Using the public cloud and containers, they were even able to get more movies out while halving the cost. “And we found it had higher performance because we could integrate faster,” he said.
How to Lead a DevOps Transformation
Cox said that leading a change of this order of magnitude involved three crucial ingredients:
- Collaboration: break down silos, mutual objectives.
- Curiosity: keep experimenting.
- Courage: candor, challenge, no blaming or witch-hunting.
But these can come with its own leadership challenges, including:
- The politics of command and control.
- How new leadership can take a company in a new direction.
- The blame bias of who versus what.
He says that as long as he can keep them focused on emphasizing company-wide that the tech of the future is better than the tech of now and building a community, he believes tech will continue to drive Disney’s success.
“There’s no secret to creating digital magic. We keep moving forward, opening up new doors, doing more things because we’re curious,” he said.