Product Experimentation talks

Product experimentation is the practice of testing changes to a product with real users and measuring the effect before deciding what to keep. The best-known method is the A/B test, where different groups of users see different versions and their behaviour is compared, but the field also includes broader uses of data to guide design and growth.

For product and design teams, experimentation replaces opinion with evidence. It shows whether a new design actually improves sign-ups, purchases or retention, and it can reveal that a confident assumption was wrong. Used carelessly, though, metrics can push a product towards short-term gains at the expense of the user experience, which is why the relationship between data and design is a recurring question.

The talks here cover four sub-topics. A/B testing is the focus of Karl Gills’s How to create better A/B tests based on User Research and Craig Sullivan’s The Myths, Lies and Illusions of Cross Device Testing. Conversion optimization is covered by Bryan Eisenberg’s 21 Secrets for Top Converting Websites and Hidden Secrets of the Amazon, and Tim Riesterer on buying intent. Designing with data includes Jared Spool’s Is Design Metrically Opposed?, Rochelle King on the complex relationship between data and design in UX, and Jim Sterne on augmenting UX with data. Growth hacking features Gustaf Alströmer on getting users and growing, and on growth and referral at Airbnb.

Other talks connect experimentation to wider product practice: Anders Toxboe’s Experiment to innovate, Nir Eyal on habit-forming products, Maurits Kaptein on persuasive profiling, Luke Wroblewski’s One Design to Rule Them All and Des Traynor on product strategy in a growing company. Jason Fried and Julie Zhuo also appear, with talks from The Next Web Conference. The recordings span 2011 to 2019.

For how experimentation works at scale, Lukas Vermeer explains how Booking.com democratised online controlled experiments, in a talk from Mind the Product in 2018.