Presented by Optimizely
Experimentation is an incredibly powerful technique — but there’s some common techniques developers make when they’re testing. Learn how to make the most of your A/B testing, choose the right metrics, avoid the biggest testing pitfalls and more when you join this VB Live event!
Every year, Netflix runs hundreds of experiments; Excitable Airbnb runs over five hundred a month, and Jeff Bezos says that Amazon’s success comes from the number of tests his teams runs not just yearly or monthly, but daily. Having a culture of experimentation is a competitive differentiator.
Experimentation is great for companies that want to design more products, better products, and products that get launched more quickly and more successfully. It’s supposed to be a formula for no-fail success. Unfortunately, it’s not a no-fail technique.
For every A/B test that unveils a winner, there’s a well-meaning mistake, an all-too-common error, a typical misconception. Suddenly product teams look up and realize they’ve been heading down the completely wrong testing path, and the only way back is to scrap days or even months of work. That’s a huge amount of wasted time, resources, and money.
Experimenting with the wrong metrics is the most common mistake that product teams end up making — and when it happens, it’s like using a very powerful gun to shoot yourself right in the foot. It’s easy when you have a very specific goal in mind — increase conversions to the landing page, for instance. That’s a cut-and-dried case.
But when your goal is more nuanced, like increasing user engagement, or improving user experience or boosting long-term retention, choosing the right metrics is far trickier. Plus the metric you choose will have a profound impact on your product, for good or bad.
The tests you run also have to have enough power to move your metrics, or you’ll find that you’re only clocking the most significant behaviors, or even just the outliers. When you realize you haven’t captured the data you need, that’s a wasted experiment. When it takes you some time to realize that the data you’re iterating on represents only the most drastic effects on your experiment, you’re in the weeds again.
Misinterpreting an experiment is many magnitudes worse than simply not running the experiment. Like all these errors or pitfalls, one small carrying error here results in the space shuttle flying into the sun.
But you can’t let that stop you, because maybe the biggest mistake product teams make is settling for the minimum number of tests you think you need in order to optimize and improve. A handful of A/B tests just to get you started is great — but why do you stop there? You shouldn’t.
Unfortunately, going from none experiments to all the experiments isn’t an easy road. Your culture of experimentation can fizzle out when you try to scale, because looping your team into the importance of experimentation is one thing; evangelizing across the company is another. Without higher level support, your own department can be looking at an uphill battle.
These are just a few of the challenges you’ll face when you leap into the world of experimentation, or the challenges you might already be facing. It’s time to find some solutions. Register now for this VB Live event, where we’ll cover some of the biggest pitfalls and challenges product teams encounter, and how to tackle each one, plus a look at real world case studies and important takeaways from giants like Amazon, Airbnb, and Microsoft.
Don’t miss out!
You’ll learn about:
- The most common mistakes product teams make when running experiments
- Which metrics correlate best with your business’s success
- Strategies to scale experimentation across multiple teams and squads
- How the world’s top technology companies are able to experiment on all product decisions
Speakers:
- Jon Noronha, Director of Product Management at Optimizely
