On the Product Management Treasure Hunt, Experiments are Your Metal Detector

Somewhere in a vast expanse of canceled launches, forgotten features, failed products and opportunity cost there are incredible gems to be found: products that users treasure, ones that massive, world-changing business is built on. And the trillion dollar question is: how do we make more of those, and less of everything else?

Our objective as product managers is to find these products and features that truly matter for our users and the business. To guide our efforts in finding those products, we need a process that can steer us in the right direction, tell us how far to dig, and evaluate what we’ve found at the bottom. We need tighter integration with software engineering teams, and to do so, we must adapt an experimentation-first mindset.

If PMs are treasure hunters, then experiments are our metal detectors. These techniques are the most powerful tools we have for reliably avoiding waste and discovering opportunities by guiding product teams through each stage of the product development “treasure hunt.”

Experimentation is our metal detector. Product managers have many tools for steering product development. We can interview users, run A/B tests, analyze historical data, perform gradual rollouts, and run clever studies. Each has its place, but what they all have in common is the broad theme of experimentation, the idea of exposing users to new features so they understand how they’ll use it and what value it provides. Experimentation means eliminating guesswork by testing things out in the wild as early as possible.

In my experience, you can always cut scope further, but only if you have a deep understanding of your users.

Experiments tell us where to dig. The best place to start a new project is always generative research, building up deep empathy for your customer and the world they inhabit. The next stage is understanding that user’s problems, digging not just for interest but proven intent to solve. The final step is to explore solutions, defining through a process of design and engineering how your product can uniquely solve those problems — essentially the “X” that marks the spot.

Experiments help us steer. Now, we know our customers are interested in sports content. But do they want it in video form or written articles? Do they prefer editorial or news? How do ads impact the experience? Nailing a product launch is about tweaking a hundred different variables to zero in on the perfect balance. Getting any of these wrong can defeat the whole project, like digging a hole just inches away from a treasure chest and finding nothing but dirt. A/B/n and multivariate testing help us build many different variations and find the ideal combination, without the risky guesswork.

Experiments tell us how deep to dig. Product management isn’t just about prioritizing features to work on. It’s just as much about deciding scope and recognizing when the perfect is the enemy of the good. Which functionality needs to be easy, and which needs to be merely possible? In my experience, you can always cut scope further, but only if you have a deep understanding of your users. Techniques like staged rollouts and iterative A/B testing let you quantify the value of a minimal feature set without having to build the whole thing. These experiments can tell you when to quit early, buying time to chase other opportunities rather than getting caught in an ever-expanding hole.

Experiments find landmines. One of the biggest risks in product development is chasing an idea that seems appealing but actually makes your product much worse. For example, we might build a social feature in the hopes of driving engagement, then find that it completely alienates our users. Or we might redesign our navigation, then hopelessly confuse people in the process. We might even nail the design, but make a simple engineering mistake that degrades performance or causes a crash. Experimentation techniques like feature flagging and gradual rollouts protect against this downside risk, ensuring that we stop digging before we hit trouble and have the chance to correct.

Experiments quantify impact. The final gift of experimentation is its power to tie the work we do to business impact. Through A/B testing and careful analysis, we can measure how each product launch directly contributes to a company’s ultimate goals, whether that’s revenue or readership or activity. Experiments run by Microsoft’s Bing team, for example, has increased revenue per search by 10 percent to 25 percent each year and raised its share of U.S. searches conducted on personal computers to 23 percent (up from 8 percent in 2009). And this means we not only make an impact but also gain the data we need to justify bigger and more ambitious projects. If you want a bigger pirate crew and a nicer boat, you have to prove to the business that you can find real gold.

With an experimental mindset, product managers and software engineers can collaborate more effectively and see the best results. It’s a journey where you don’t need to have all the answers, you just need to run the right experiments.

Feature image via Pixabay.