What news from AWS re:Invent last week will have the most impact on you?
Amazon Q, an AI chatbot for explaining how AWS works.
Super-fast S3 Express storage.
New Graviton 4 processor instances.
Emily Freeman leaving AWS.
I don't use AWS, so none of this will affect me.
DevOps / Tech Careers / Tech Culture

Kelsey Hightower on How to Become a Better Engineering Team

Jun 22nd, 2022 7:00am by
Featued image for: Kelsey Hightower on How to Become a Better Engineering Team

Technology work has moved far past the idea of the code monkey or button pusher. If we are truly knowledge workers, then tech work should be as much about communication as code. And it needs to evolve past one-off trainings and university degrees and move toward continuously honing a craft that’s constantly evolving pursuant to building high-performing teams. 

This was the argument Google Cloud’s Kelsey Hightower was making at DevOpsDays Ukraine: DevOps in Crisis, in his fireside chat with DataDog’s Daniel Maher. He also argued that the secret to scaling isn’t DevOps or site reliability engineering or even the tooling, but better communication and alignment. This starts with focusing on the team and then the individuals within the team or teams.

Alignment Around Shared Understanding

Maher kicked off the virtual conversation by asking Hightower to pinpoint the right time for a growing company to implement DevOps or SRE. When you’re a small team, perhaps at a startup, everyone does everything and job roles blur together, but, Hightower explained, everyone has a shared understanding of everyone else’s role — or roles. 

“I think the bigger you get, the larger the company, the more time you’re at it, that tends to get deluded over time, where people just focus only on their job, not necessarily the job to be done for the company or for the customers,” he said. 

As a team scales, Hightower said, you need to be more explicit not about formal job descriptions, but about what people need to work together, especially when they’re coming from different places, backgrounds and longevity within the company. “So they need more guidance and frameworks about what to do and how to collaborate.”

Specifically, he said, site reliability engineering is well suited for this as it’s a discipline that makes engineering plans via actual metrics and science to prioritize objectives and goals. It’s just important to amplify those lessons learned across the organization.

Team members need to answer: “What skills do you bring to the table, and let’s not redefine ourselves based on HR job descriptions. We should be able to go beyond that and be willing to collaborate even when it’s hard or even when that’s not necessarily our job,” Hightower said. 

Whether it’s a sports team, a family or a group of colleagues, he continued, the hardest thing to do in any human system is to consistently clarify “who owns what, can they deliver, and how do we support them in doing that even though we all have our own lanes to swim in?” DevOps is, of course, all about breaking down those swim lanes, which demands even more clarity over work-as-done.

There are several definitions, dogmas, frameworks and personal interpretations of what DevOps and SRE are, but Hightower said that it’s much simpler — and thus more challenging — than that. Teams are able to scale successfully when they can:

  • Hold themselves accountable — what have you committed to in what time frame? 
  • Address codependency — one person isn’t the only person able to do a particular thing.
  • Divide up the work — to make collaboration efficient.
  • Support each other — even when folks fall short.

If they cannot achieve these, it doesn’t really matter if a team is following Waterfall or DevOps or SRE, it’s just not working.

Promise Theory and Codifying Team-to-Team Relationships

Agile, DevOps, Scrum, they have all gotten good at measuring team productivity, but what about the interconnectedness and dependencies among teams?

“We don’t have a lot of experience with management so far — a lot of our industry is copy and paste: ‘This company uses objectives and key results [OKRs], so we’re going to use OKRs. I heard about DevOps, so we’re going to do DevOps.’ And people have no context of what this is.” Hightower also explained that team building is a unique — and still rare — skill. “You have to constantly know the capacities of your team members at an individual level and then be constantly optimizing for those individuals.” 

But when you to have to optimize at the team-to-team level, it gets very complicated because, as he said, choices in tech aren’t necessarily deliberate. Oracle may have been chosen a decade ago by one person, or Java was the programming language that the company founder knew best. The longer ago these decisions were made, the harder it is to change. But, Hightower contended, that doesn’t necessarily mean these decisions are made explicit to everyone. This is why it’s necessary to not only document technical decisions but the reasoning and context that went into them.

“When we start to interface with each other, it’s equally ambiguous about what we are doing,” he said. “Some people say silos are bad, but are they really?” Hightower instead said it may be better to have a silo but with a clear API contract and SLA to pass data securely from it. “I just don’t think everyone understands the nuance that comes into working with individuals and defining the contract between the groups.”

This is where documentation and practices like Wardley mapping are very valuable in creating a common language.

Designing Distributed Empathy

Maher brought up how there’s a lot of talk about customer empathy as a necessary ingredient for development. But how do you identify and cultivate that on teams? Hightower said the opportunity for empathy begins in the job description and then the tech interview process. 

“You come in and you understand this narrow part of the equation, and you kind of just do your thing. The truth is you can get very far in our industry without ever using the product or meeting any of your customers ever,” Hightower said. You may even get promoted by keeping your head down and writing good code, but you are unable to engage in the whole organizational structure at a higher, more strategic level.

This is how the chasm between tech and business persists. If you don’t dogfood the product and have a grasp of the market conditions surrounding it, you can’t answer “why am I here?” Tech workers need to understand how they fit into the complete picture and increasingly demand purpose in their work. 

“A lot of times as technologists — and I’ve been guilty of this before in my career — we assume engineering is the most important function of any business. We believe it’s the hardest task. We believe it’s the thing that requires the smartest people. And so sometimes we get into this world of ‘as long as technology has this bubble, we will power the business. We will transform the business.’” But you won’t, Hightower said. 

Technology supports business goals, so engineering teams need to first understand the customers and business they are supporting before understanding the purpose and impact of that technology, Hightower said. 

How do you build that empathy? Hightower recommended that you be a customer for a day and ask yourself: Why would I pay for this?

This mindset, he promised, will influence the way you write tests and documentation and present the product via user experience. It’ll especially impact how you will aid your support team that interacts with customers daily.

Engineers Do Ride-Alongs, Too

Of course, as Maher said, while some organizations would be happy for engineers to join sales calls, others want them to keep their heads down to the code. However, Hightower said that’s a risk.

He gave the example of a tractor manufacturer. An urban engineer may not have any rural needs, but what happens if they are called on to contribute to a predictive maintenance roll-out? Now, this involves sensor placement and data collection, followed by localized and centralized data modeling. Hightower said you can have all the DevOps you want, but you still won’t be able to tune your models without answering:

  • Who is the customer? 
  • How do they want to get information about their tractor needing to be fixed?
  • What happens if you fix the tractor prematurely during harvest season?
  • What happens if you optimize for off-season maintenance?
  • What happens if your models are “too tuned” for temperature? Not enough?

It’s imperative for the engineer or data scientist to go out to the manufacturing or testing site, or even go on a ride-along with a farmer, to understand the difference between a working tractor, a malfunctioning one and one that seems a bit off. 

He continued that you should ask a current user:

  • Before you had this capability, how did you know it needed to be fixed?
  • How long is an acceptable cycle for it to be fixed within?
  • What do you like or dislike about this capability?

You’ll have great context and empathy for your customer’s experience. Bonus if you reach out to a lost deal or former customer to understand why they were unhappy with the product.

“That’s a cheat code if you want to celebrate your ability to build stuff for actual customers,” he said. 

This is just part of the continuous learning that tech work demands. Hightower reminded the DevOpsDays Ukraine audience that tech work is more and more a creative craft or trade, able to be honed. And, no matter if it’s a new language or a new way to connect with customers, seeking out new ways to learn will not only make you a better engineer and engineering teammate, it’ll make you more confident on your next project or team.

“A lot of success is just people attempting,” he said. Once you can learn how to do something in one language, you have more confidence to try it in another. “That is the best feedback loop because it’s internal, you feel it… I can’t believe I can do this!” 

And then it’s up to engineering leadership to make sure teams and individuals have the time, space and ability to pursue those moments.

The Illegal Invasion of Ukraine Continues

DevOpsDays Ukraine was a virtual event featuring 17 sessions from worldwide DevOps leadership volunteering their time to raise funds for charities on the ground in this war zone. This DevOps in Crisis event has already raised more than $100,000 for the following charities:

  • Razom — to support short- and long-term projects that foster democracy and prosperity in Ukraine.
  • Kolo Foundation — to organize humanitarian aid and coordinate civilian evacuations.
  • Insight — to support LGBTQI communities in Ukraine.
  • Happy Paw — to provide assistance for homeless cats and dogs.
  • Monstrov Corporation Foundation — to provide humanitarian and medical aid to those in crisis.
  • Voices of Children — to provide psychological support for child victims of war.

You can support these essential causes directly or by buying tickets to DevOpsDays Ukraine.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Hightower.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.