Esther Derby on ‘Change by Attraction’ and Retrospectives at Scale
When we asked organizational change consultant Esther Derby to describe herself, she replied that she’s “someone who thinks deeply about leadership, management and change — making workplaces more humane.” She didn’t use the word agile or the reflective ritual of retrospectives that she’s most associated with. It could be because agile, at 20, is so pervasive in the software development world that it doesn’t bear mention in titles anymore. Or it could be because any knowledge organization nowadays is responding to change, irrespective if they embrace an agile mindset or not.
Later this week, Derby will be speaking at the Ninth Annual Agile Tour London, live and online Oct. 21-22.
In this interview, we learn how Derby turned a love of flowcharts as a young child into a career as a best-selling author advising organizations — from startups to Fortune 500s — on change leadership. We learn from her more than four decades in the software world, including how to fix and scale retros and some techniques to influence people and attract them to a change — instead of forcing change upon them.
The New Stack: Let’s start with your tech and agile journey.
Esther Derby: I am one of those used-to-be technical, not-so-much-now people. I started as a programmer, but I haven’t actually written anything but HTML since the late 1980s, early 90s.
I think it actually started when I was nine years old. And my father, who was a bureaucrat, brought home one of those aqua-colored IBM flowcharting templates because they were automating some of the processes in the office where he worked. He was the director of one of those offices for the state. So I started flowcharting things.
What does a nine-year-old do with a flowchart?
Whatever you’re doing, you know, how to clean your room, how to rig up a little string and pulley system so you can turn a light switch from your bed — all those things that nine-year-olds do.
I actually have an undergraduate degree in art history, and you can’t make a living with a degree in art history. So I thought about my flowcharting template and how much fun that was, and I went into programming. My first professional job was as a programmer.
How’d you move from programming to management?
I did the tried and true thing. Because I was a good programmer, they made me a manager. Makes no sense. So I had to learn a completely different set of skills, so I could be a competent manager. Being able to find problems in the code is not the same as being able to deal effectively with humans.
Did you find you were a good manager?
I think I became good at it. I was very fortunate in that I ran into Jerry Weinberg about that time. He was also, at one point, extremely technical — he worked on the Mercury Project — and then became very interested in people. He influenced how I thought about management, how I did management, how I thought about organizations, and eventually I got a master’s in organizational leadership.
A number of things along the way helped me make the shift into working effectively with people. I will say that the training I received from the company where I worked was not part of that helpful learning — filling out forms in quadruplicate, and who got which copy, how to fill out a personnel requisition form. But nothing about how to think about what kind of people you need on your team, what kind of skills you need on your team, and how to how to grow a team. So I learned most of that on my own.
How did you self-teach to be a better manager? Were there a lot of management books in this pre-agile time?
Well, if you ask a lot of people who were programming in the 60s, 70s, 80s, the way we used to — maybe not so much the 80s because a lot of heavyweight process had taken over then — but in the 60s and 70s, people were working in many ways that resemble agile. We didn’t have the same development environments, we didn’t have the same testing tools, so there were some things we couldn’t do, but we did tend to work in small increments and get something running — if for no other reason than it’s much easier to debug a very small program than a very large program, particularly if you’re waiting for computer time. Sometimes you want to get one thing running, rather than have to debug a huge program.
You started your own consulting business in 1997. How has it changed?
I have reinvented my business a number of times, as the world has changed, and my interests have changed. And, I became active in the agile community fairly early on — I wasn’t in the first circle, but I was in one of the early circles. It was a good fit for me because of the explicit valuing of people. And because it in many ways resembled ways I had worked. Even when I was in a shop that was nominally waterfall, I still tended to manage projects in a way that we get a little piece working, get it in front of people, talking to our customers every day. So it was refreshing to see that after the 80s and 90s were being dominated by heavyweight methodologies.
What sparked you to then write books about agile software development?
I told you about my experience becoming a manager, which was “You’re a programmer on Wednesday and you’re a manager on Thursday.” My friend and colleague, Johanna Rothman had a very similar experience, and we had seen many, many people go through that.
It’s hard on everybody — it’s hard on people who report to the manager, it’s hard on the manager. So we decided to write “Behind Closed Doors: Secrets of Great Management” as a resource for managers, so they could see how it happens, not just how to fill out the forms. They could see some of the conversations, they could see some of the thought process of looking at the system and working on the system.
Retros, alongside daily stand-ups, seem to be the most popular agile rituals. You’ve literally written the book on retrospectives. What mistakes are teams still making? What’s still missing?
One is a sort of systemic mistake, where teams are told, “You must do a retrospective,” “It’s an agile meeting, you must do it,” but they are not given any time or space to learn anything to try anything to fix anything. So their iteration backlog is chock full of feature work, and there’s no time for learning work, investigative work, research, actually setting up something new or lowering the time to learn how to do something new. That’s a systemic issue with the way the company is looking at utilization. And if you get 80% utilization, you have cut off learning entirely, there’s no time for people to learn. It’s very shortsighted to always jam an iteration [also called a sprint] with feature work, so you have no time to learn and no time for adaptive capacity, should something come up.
The other thing I see commonly — and I think it’s taught in a lot of scrum masterclasses — is that a retrospective consists of three columns, and you fill in the three columns, and then you vote and that’s what you decide you’re going to improve on. You have to have some facilitation skills to get beneath the surface and talk about:
- Why do we think this worked?
- Why do we think this went wrong?
- What’s going on in the system?
- Why do some people think it was good and some people think it was bad?
You can have a good retrospective with that [dot voting], but, most of the time, they’re just kind of random, scattershot. People don’t have a shared pool of data so one person says, “That went well” and another person says “That went badly.” They’re seeing something different, but they don’t discuss that or, even if you are in the same room, they don’t necessarily see the same things.
Now that many people are working remotely, they’re not having the same experience. They don’t have shared data. They don’t stop, look at the data and try to figure out what it means and identify several options, and then choose an option — they just make a list. So it’s kind of like they’re starting in the middle. They’re skipping that insights part. They have some opinions, and then they decide what to do.
It’s not conducive to the process that people go through when they think and learn together, which is what the framework in the retrospectives book is based on. It’s really based on a model that when groups think, learn and decide together, they have to start with a focus, then they diverge on increasing data, and then they are able to analyze and converge on something.
We have seen a lot of essentially public — because nothing’s private at scale — town halls and AMAs [ask me anything] with well-known CEOs, but they aren’t necessarily discussing or challenging any assumptions. How well do retros scale up? Should retrospectives be at scale?
I think it depends on how well you do it. I have been in companies where each team does the retrospective, and then representatives of those teams get together and talk about what they were looking at and look at patterns that can be effective.
Very often teams don’t have the breadth of view, not because they don’t understand it, but they [only] see what’s going on in their team. They may not see, though, that the same problem they’re having is replicated across several teams, [which] tells you it’s a systemic issue.
One of the keys to making them [retrospectives] much more effective at scale is having a way to surface those systemic issues to the people who actually are in a position and have the authority and the influence to do something with them.
How frequent should retros at different levels or scales be?
I want to expand the question to include all sorts of different kinds of retrospectives. Because there are teams that do an itsy-bitsy retrospective after a pomodoro. If they’re pairing [pair programming], they’ll do a tiny retrospective then, or a retrospective after a mobbing [mob programming] session or an ensemble session. There’s a number of companies that have been been able to do some pretty amazing things with doing these really tiny retrospectives, and then rolling it up and doing a larger team-based retrospective to step back and get a broader view.
At scale, I think it makes more sense to have a slightly broader time period because what you’re looking for are patterns and meaningful events that repeat over time — and you can’t see those if you’re looking every day. You’re more likely to see them if you have a longer temporal view — not a year, not six months — enough time so that you can see patterns and be able to discern delayed feedback loops, but not so long that your feedback loops are so long that you’ve missed the opportunity to do anything about it.
So as quickly as you can act on feedback, and as quickly as you can see patterns.
The focus of your most recent book — “7 Rules for Positive, Productive Change: Micro Shifts, Macros Results” — is on leadership through organizational change. Aren’t all organizations complex adaptive systems and thus continuously changing?
Certainly, a lot of them are, although I think there is a legacy of organizations being built to be stable and unchangeable. There are certainly organizations that are designed that way, and that, for a long time, was considered very desirable. You didn’t want things to change quickly. But most organizations, particularly in knowledge work, are complex adaptive systems, [which is] a better model than a machine.
It’s super important for all managers to have some understanding about how change works. I think about changes in a much more holistic, much more organic, much more complexity-informed way.
Let’s talk about your seven rules for change by attraction and influence.
A lot of changes are imposed on people, right? It’s like, “Oh, here’s your new process, you will follow these checklists, and you will do this.” That relies on positional power and sometimes coercive power. The best you can hope for when you’re using that sort of power is compliance.
[There are] ways of attracting someone to a change, rather than saying, “Here’s your new process, we’re going to do it.” [Instead] the whole point is to get people’s fingerprints on it. Because then they feel like it’s theirs, they have shaped it to their unique context. And it’s no longer an imposition.
But there are other sorts of power that rely more on attraction. People who are respected can float an idea and people will take it up — “If she thinks this is worthwhile it might be worth doing.”
Modeling something is also an attractor — just starting to do something and seeing who gets interested in it can be an attractor. For a larger change, who wants to have a pilot program, who wants to be the first to try this thing?
A friend of mine was really interested in having his team try Kanban. This was several years ago, and, rather than try to talk them into it, he started tracking his own work on a Kanban board, with Post-Its in his cubicle. In a matter of days, someone said, “Oh, what’s that?” So he explained what it was — didn’t try to sell them — and just said, “You know, it’s helping me in these ways.” And the guy said, “That’s kind of interesting. Maybe I’ll try it.” So he started doing the personal Kanban board. And pretty soon there were three people doing it. And at that point, it was really easy to make the step to, “Let’s try it for the team.” That wasn’t done by trying to talk people into it or trying to push an idea. It was like, “I’m doing something” and somebody became attracted to it and got interested in it, then he had an ally, and then another person saw it. And so it happened very organically without trying to convince people.
This is similar to when an innovation hub, cohort or sandbox within a company tries something out and becomes a model for the rest of the organization.
And it doesn’t have to be your innovation center. It can be just anybody. Although that is another attraction principle that I’ve learned from Jerry Weinberg — the scarcity principle. Sometimes if you want to try a new method or a new process, you just say, “We’re going to be trying this with just one group of people who apply to it.” And then that becomes special.
And the nice thing about working that way is, if they can get it to work, then you have effectively taken “Oh, that won’t work here” off the table. You’ve learned a bunch of stuff about what you need to do to get it to work. And if they can’t get it to work, even those people are highly motivated.
The most recent episode of your podcast “Change By Attraction” featured scrum master Matthew Carlson. You spoke about how, after 20 years, there are no early adopters of agile anymore. Tell us about that.
We were talking about what is different about the motivations of people or companies who want to embrace agile methods now versus 20 years ago when it was new. If you’ve ever read about diffusion of ideas [also called diffusion of innovation], you know the people who are early on want to try something new and they’re exploring and they have a lot of enthusiasm. But the motivations of people who are looking at something ten or 15 years after it’s been in the market, their motivations are very different… but also legitimate: “We want to do what the successful companies are doing.”
The problem with that sort of benchmarking way of adopting — “because some other company is doing it we should do it” — is that they tend to copy the most visible and often least critical parts of the process. They copy in the external form without the thought process that went into it.
Tune in to the Ninth Annual Agile Tour London, Esther Derby speak on silos and specializations, she will join the keynote stage along with Simon Derby and Gene Kim at the live and online Oct. 21-22. The author of this piece is a co-organizer of this event.
