Democratising Online Controlled Experiments at Booking.com

Lukas Vermeer of Booking.com explains the culture of experimentation at the company in this keynote from Mind the Product Engage in Hamburg, given in April 2018. Vermeer was one of the people who set up not only the tooling but also the culture behind online controlled experiments, or A/B tests, at Booking.com.

The title refers to an academic paper Vermeer co-wrote with colleagues on democratising experimentation at Booking.com, but the talk itself focuses on the cultural side. He describes how a centralised and standardised experimentation system is a necessary basis for independent, decentralised decision-making by product teams.

He also stresses the importance of formulating hypotheses and testing them, and recommends ranking your assumptions so that you can identify and test the riskiest ones first. The talk is useful for anyone who wants experimentation to be something every team does, rather than the job of a few specialists.