"Strongly pro-experimentation. Believes in 'test everything' because human intuition about what works is demonstrably unreliable -- 66-92% of ideas fail when tested."
"A/B testing is essential but most ideas fail - be ready for 80% failure rate"
Evidence from the Archive
Airbnb, Microsoft, Amazon
Airbnb search relevance: 250 experiments, only 8% improved target metric, but collectively drove 6% revenue improvement
Bing ad text change: moving second line to first line generated $100M/year in revenue -- languished in backlog for months because nobody's intuition flagged it
Widely regarded as the world's leading expert on A/B testing, having built experimentation platforms at Microsoft (25,000 experiments/year), Amazon, and Airbnb; author of the definitive textbook on online controlled experiments. Their core argument: Strongly pro-experimentation. Believes in 'test everything' because human intuition about what works is demonstrably unreliable -- 66-92% of ideas fail when tested.
The evidence is specific: Bing ad text change: moving second line to first line generated $100M/year in revenue -- languished in backlog for months because nobody's intuition flagged it. Furthermore, airbnb search relevance: 250 experiments, only 8% improved target metric, but collectively drove 6% revenue improvement. Airbnb 'open listings in new tab': one of search's biggest wins, heavily debated and resisted by designers before data proved them right.
In Ronny Kohavi's own words: "I'm very clear that I'm a big fan of test everything, which is any code change that you make, any feature that you introduce has to be in some experiment. Because again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising, unexpected impact." (Responding to whether you can be too experiment-driven.)
Airbnb, Microsoft, Amazon
Airbnb search relevance: 250 experiments, only 8% improved target metric, but collectively drove 6% revenue improvement
Bing ad text change: moving second line to first line generated $100M/year in revenue, languished in backlog because nobody's intuition flagged it
Widely regarded as the world's leading expert on A/B testing, having built experimentation platforms at Microsoft (running 20,000-25,000 experiments/year), Amazon, and Airbnb; author of the definitive textbook Trustworthy Online Controlled Experiments. Their core argument: A/B testing is essential and you cannot do too much of it -- but you must accept a 66-92% failure rate and maintain a portfolio that balances incremental gains with high-risk, high-reward bets.
The evidence is specific: Bing ad text change: moving second line to first line generated $100M/year in revenue, languished in backlog because nobody's intuition flagged it. Furthermore, airbnb search relevance: 250 experiments, only 8% improved target metric, but collectively drove 6% revenue improvement. Airbnb 'open listings in new tab': one of search's biggest wins, heavily debated and resisted by designers before data proved them right.
In Ronny Kohavi's own words: "I'm very clear that I'm a big fan of test everything, which is any code change that you make, any feature that you introduce has to be in some experiment. Because again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising, unexpected impact." (Opening statement on why every change should be tested.)