Culture / What Is DevOps?

Tales from a Red Hat Quality Engineering Manager

28 Nov 2021 6:00am, by
Bassem Dghaidi interview Red Hat quality engineering manager Joe Rahme - 2021 screenshot from glich podcast

There’s a lot to be learned from the people around us and the ways they’ve approached challenges in both their roles and in their organizations. One person who’s experienced this firsthand is Bassem Dghaidi, a senior solution architect working at GitHub.

This August Bassem launched a YouTube show named “glich” promising “software engineering lessons from the real world,” and he’s constantly expanding his knowledge base by interviewing others in the technology community. “I hope my content will have a positive impact on your journeys,” Dghaidi says in a first video welcoming his audience. Since then topics have included everything from AI and GitHub Copilot to the importance of product managers and even to a comparison of Docker and podman.

Recently Dhaidi interviewed Joe Rahme, who for more than two years has been a Milan-based quality engineering manager at Red Hat, leading a team of 10 quality engineers validating new releases of Red Hat’s cloud-computing platform OpenStack.

Rahme offered a surprisingly insightful look into the role of a quality engineering manager — from the unique challenges of overseeing a team of developers and creating a supportive culture to the larger software-related challenges of testing a sprawling cloud-computing platform.

Dghaidi titled the video “What does an engineering manager at Red Hat do?

And before the conversation ended, there were also a number of other useful lessons…

Establishing Technical Authority

The first lesson? To do the job of a quality engineering manager, a certain amount of technical knowledge helps.

Rahme points out that the job involves working closely with developers, “and the better you understand their job, their tools, their process, their way of organizing things — the more you are able to collaborate with them.”

Rahme’s own background includes earlier stints doing integration, packaging/source control merging, system administration, and quality engineering — all of which he describes as working around developers, if not actually working as a developer. (And he also spent one short interval as a web developer.) So his management style is ultimately informed by a familiarity with the product and processes.

“I couldn’t do this if I didn’t have this technical background” — Joe Rahme

And it’s a point Rahme emphasizes. “I’ll say just that for me, the idea that you don’t have the technical knowledge and you still want to manage it — sounds scary…” Rahme explains later that “the obvious bullshit, should not pass through you.”

And that establishing some technical authority “sets the tone immediately. People start respecting you.”

Once you’ve got the respect of the team, the role of the manager becomes more apparent. Rahme acknowledges that he can only speak about his own experiences in his own corner of Red Hat, but it’s been an instructive experience.

At one point Rahme describes the style that emerged as “We have engineers; let them engineer. I’m not going to out-engineer the engineers. That’s not my role. Even to a certain extent — my role is not to give the solution to a technical problem. My role is not even to give a solution to how are the processes and how are we organizing ourselves. But where I have a lot of impact is how do we shape the culture into something that makes sense.”

Just for an example, Rahme points out that if there’s people who are afraid of saying I don’t know or I don’t understand, “As a manager, this is something you can solve that nobody else can. And the fact that you can actually solve these things can have a lot of positive impact on the long-term.” Later he adds an additional area of impact: making sure that everyone’s opinion is being considered. And he also emphasizes something that’s often overlooked: that culture “is very contagious.”

Later Rahme succinctly sums it all up. “I think that a good manager is not in charge of the processes or the solutions — they’re in charge of the culture.”

And he also has some advice for all-remote offices: have your whole team meet in person at least once or twice a year. “The impact is insane! After my first time doing this with me team, I came home and I was like, ‘Now I know the guys!'”

Learning to Make a Difference

Red Hat quality engineering manager Joe Rahme - 2021 screenshot from glich podcast

So where do good managers come from? Red Hat gives all new managers a nine-month training course — which Rahme believes is also valuable in another way. “Putting you in contact with other managers who are discovering things — a lot of them are ex-engineers, long-time engineers, a lot of them are long-time managers from other companies. Being in contact with them and sharing stories and sharing strategies started making me see that there are patterns, some ways where — ‘Okay, I see how in this case a good manager made a difference….’ And I see how management is something that helps an engineer.”

That’s his advice for other management training programs. Besides running through sample scenarios, “share actual experiences… This is the sort of training that will make you be a good manager, because it will force you to think, how am I adding value? How am I making a difference here…?”

And he has a carefully-chosen phrase to describe the management philosophy of organizations that aren’t giving their managers training: wishful thinking.

But he emphasizes later that with training, all things are possible. “I really believe that, even if you’re not born with it, you can learn it. ”

So what do you do as a Red Hat quality engineering manager? Soon Rahme is sharing his own actual work experiences. For example, how does Red Hat go about testing something that’s as large as the OpenStack cloud computing platform?

Rahme points out a lot of the testing and quality assurance is “inherited” from an upstream community of developers and testers: “Now, I say ‘inherit’ as if they’re completely separate people, but Red Hat is part of the upstream community. Therefore a lot of the upstream testing which is supposedly different than what we’re doing is done by Red Hatters. So we test the upstream first, and then once we package our downstream builds, we test them separately..”

“Downstream, a lot of the emphasis is put on testing the integration between different components… [W]e have access to bigger parts — more nodes, more physical hardware — and then we start testing all this integration with the different projects. We pin down specific versions,” he said.

The Road Not Taken

In a surprisingly personal moment, Dghaidi asked if there were things Rahme missed after moving into management. “The obvious one is, of course, spending time in your code editor,” Rahme begins. “I still do — it’s not that I don’t — but I don’t do it as much.”

But later he admits candidly that the thing he misses most is participating in those heated reviews — “when you had something that was contentious, and a lot of people had their own opinion. Being, like, on the battlefield, just taking everybody’s opinion. Trying to find compromises, trying to find who’s making sense, who’s not making sense.

“Now, more, I take a step back, and I’m more a facilitator. Even though I have my opinion, I don’t chime in as strongly, as vigorously as I used to,” he said.

And that seems to get at something deeper. “But there is also, in general” — and here Rahme paused reflectively, before saying “I miss being able to talk to the team without being ‘The Manager.’ Like maybe participating in rumors, or maybe giving my opinion — or maybe giving someone encouragement, without it having the subtext of ‘This is your manager talking to you…'”

He’s managing people that he used to work beside, for years and years. “So there’s this aspect that kind of I miss — being able to speak more freely and goof off.”

It’s a theme Rahme returns to later in the program, when he admits his biggest challenge, throughout his entire management career, has been holding real responsibility over other people’s salaries, raises, bonuses.

“When you’re an engineer, most of what you’re saying is — it’s fun. ‘I have ideas of what we should do in Jenkins. I have ideas on how we should test this Python thing.’ It only affects you from Monday to Friday, nine to five, and then you turn off your laptop, you go on your merry way, and you do whatever you want to do… But now if I’m in charge of your raise and your bonus, I’m affecting more than this. I’m affecting your weekends, I’m affecting your vacation, I’m affecting your Christmas shopping, I’m affecting things that were new and different to me…

“I cannot take this decision as lightly as I used to… If I mess this up, I’m really screwing someone up…”

It’s a heavy responsibility, and for people considering it as a career, Rahme ultimately cautioned that “It’s not a very glamorous role. A lot of it is supporting people who are doing fun things. But when you see the team working, having good cohesion — it’s a very fun thing to say, ‘Hey, we built this together…’

“And promoting people is one of the best feelings in the world…”

The New Stack is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.