How Dashboards are Changing Human Behavior in DevOps
CloudBees sponsored this story, as part of an ongoing series on “Cloud Native DevOps.”
What is a dashboard? Something readily visible to everyone that needs it that shows how something is functioning or progressing. When used right — whether on a car, aircraft or DevOps team — a dashboard is simple to read at a glance and a very powerful way to know you’re heading in the right direction.
In 2014, IBM’s Steve Poole took over leading a more than 100-person, somewhat distributed, European “Slow IT” team, as it moved toward the much faster cloud. They moved from a two-year shipping cycle to supporting SaaS products that shipped daily. He shared how dashboards helped close the communication gap among siloed teams with the Agile Tour London conference and in a follow-up interview with The New Stack.
Dashboards Act as Cross-IT Translators
Poole’s first challenge was to take the operations team that’s set up to run “traditional operations with traditional deadlines and time” and to convert them to a new 24/7 release cycle. They needed to understand the value of what was happening.
“We had to get people from the old way to a new way of thinking. Dashboarding became a major aid to making that happen.” — Steve Poole
“I learned that IT and dev don’t communicate. They just shout at each other in their own language,” Poole said, because they don’t share a common experience or perspective.
“IT teams tend to group themselves around the experts so you see a lot of mini-silos who are responsible for parts of it. And the team that I took over had lots of small teams.”
Poole then pointed out how even tech conferences are usually very silo-ed around certain languages, methodologies or roles.
He saw a need for a new “DevOps contract” between developers and operations that included self-service assets like Platform as a Service and Infrastructure as a Service, containers, and testing, and it must cover new availability requirements and an emphasis on speed to market and feedback loops.
This shared sense of purpose is illustrated and translated by dashboards.
Poole compares a digital transformation dashboard as similar to your car dashboard: very visual with:
- Just enough status
- Just enough insight
- Just enough warnings
- Clear emergency indicators.
You should be able to see something is wrong at a glance.
His area of IBM is simply littered with screens broadcasting information on dashboards everywhere, leveraging Raspberry Pis to create a network of monitors.
Within a year, everyone — every dev team, support team, manager and executive — had a dashboard, broadcasting more than four million events per day.
“You stop looking at your email and reports in your system, and you start looking at dashboards,” Poole said.
He continued that clear green and red notifications totally changed reaction speed.
“Prior to that, dev teams would do the work that they are planning and then the management team would come in and say ‘We have a bug, who’s going to fix that?’ Now, it’s a pager look for the whole team” — everyone on call with the same view.
They use Dashing, a framework created by Shopify, to create simple dashboard widgets.
This amalgamated larger team includes customer-facing, developer-facing and public-facing entities. This led to about five different groups of dashboards for:
- Each development team
- Each support team
- Management and Executives
- Unusual and quirky dashboards
All of the dashboards had one thing in common — to make sure the right people were the first to know if something went wrong.
Middle Management and Executive Dashboards
It can be difficult to measure the immediate success of DevOps transformations, since sometimes the process leads to a slow-down, missed budgets, or both. Giving something for both middle management and executive to measure helps keep them committed to the often disruptive change.
Middle management needs what Poole called “reactor dashboards,” that help flag what they actually have to deal with.
“We have a reasonable culture for having good insight into the dev process, bugs, [and the] agile process.”
Poole says, for middle and upper management, dashboards need to help them think about actions that can be taken. Middle management dashboards at IBM can include setting a bar on the number of bugs in the backlog or measuring incoming rates.
While middle management dashboards is all about seeing where intervention is necessary, executives are desperate for more accurate data to guide which levers to pull to redirect the organization.
For example, one of the things dev teams were told to do is make better use of cloud capacity, moving from on-premise. Poole says to do that, you have to understand your current on-prem status. With dashboards, the large team went from numbers to a progress report, a line that goes up when cloud is gaining and local is going down, with a projection for the future. Of course, comparing cloud to on-prem is like comparing apples to pineapples. In order to do this, they had to classify servers by the type of workloads and to classify on-prem workload in term of cloud characteristics, like CPUs, memory, and multi-tenancy to bare metal.
With this, managers were able to justify budget increases more effectively, by showing how their existing capacity is being used, and they could better predict the overall cost of the move to the cloud.
Everything had to be handcrafted for the executive dashboards, but then, since it’s DevOps, the information retrieval was automated.
Poole said, “When you come up with a common vision, you chuck all the titles away and ‘Let’s just sit down and see what you want to do and what you want to get out of it.’ You can educate the exec in the realities of the challenges and he can come back and say ‘I accept that’ or ‘I don’t.’ The creation of the dashboard [makes it] a two-way activity.”
Like all dashboards and big data, a big challenge to DevOps automation is cleaning up inaccurate, unrealistic, and stale data.
It took the team about a month of conversations to understand what data that was coming out of thousands of machines, understanding purposes, use cases, and the size of said data. Only then could the data be cleaned, bringing usefulness to the dashboards.
A bonus of just the dashboard on the move on-prem to cloud was that IT support also gained an improved understanding of what their clients are using machines for.
Support, Developer and Operations Dashboards
“My IT team was reactive and I really hate that, and I want my teams to be going out to customers and be the first to say: ‘You’ve got a problem.’” — Steve Poole
If you really want to get devs, ops, and support on a united front, you need to offer very transparent feedback and let them visibly see they are on a path together toward shipping better code.
In one dashboard, Poole’s team charted out how many lines of code they were shipping versus the level of complication in said code. If there was a certain combination of quantity and complexity, it should be paused from being shipped.
Poole said, “They were trying to figure out how to get to the point where they had a delivery that was reasonable at all times — it wasn’t super large or small or complicated or simple. To dashboard the quality of your contributions. It’s all about insights and you looking for patterns, easier with a graph, more actionable.”
They followed this mindset with all the IT dashboards, like improving ticketing dashboards, measuring things like “How long to close 80 percent of the tickets?” And they used dashboards to also highlight the different processes each team uses.
And they weren’t just sharing this info within teams. They were sharing teams’ initial response times and time-to-resolution figures with their end users, whether the devs being served by the ops or sometimes even externally. They even have some dashboard variations for key SaaS customers.
“To make that real, you have to have a conversation with your end user, what it means to be down. What does red mean? Depending on what service that you’re looking at it can be different.” Poole continued, “What we taught the IT teams is to understand what it really meant for a service to be available, so when it was unavailable it really meant unavailable.”
Dashboards Don’t Change People, but They Can Reflect Real Change
For Poole, the biggest challenge overcame was “You’ve got to get both sides of the dashboard — those that are using the data and those that are consuming the data — to agree.”
He described it as a constant negotiation, with people on both sides having to learn how the other side operates.
“You’re washing your dirty linen: ‘You do that? Why do you do that?’ Sometimes it’s a fair question and sometimes it’s just not understanding,” he explained.
But once this dashboard infrastructure was up and running, the value was clear.
He clarified that none of these dashboards are tied to business goals or budgets, but rather service availability, developer productivity and development statuses.
“We don’t really need to put it down into dollars, it’s all about progress and status,” Poole said.
Can dashboards really change culture? Of course not, but they act as a good mirror to see if the culture has embraced transparency, autonomy and automation.
“The important thing about this is it changed how people behaved. You go from a team to where the body language is very closed, they don’t want to talk, and they don’t want to share because they are afraid they’re being told off,” Poole said.
Once they understood their customers loved what they were doing, he says, their behavior changed. “We went from very negative teams to understanding they were valued,” Poole said.
Screenshots from Steve Poole’s talk. Photo courtesy of Agile Tour London.