Metrics · Featured Debate

9 guests 19 episodes 4,050 words

The $100 Million Decision Nobody Believed In

Should product decisions be driven by data or intuition?

A small code change sat in Bing's backlog for months. It moved the second line of ad text to the first line. Nobody rated it as a priority. An engineer who kept seeing it in the backlog said, "My God, we're spending too much time discussing it. I could just implement it." He spent a couple of days on it and launched the experiment. An alarm fired: something was wrong with the revenue metric. But there was no bug. That trivial change increased revenue by about 12% -- worth $100 million annually. It was the biggest revenue impact in Bing's entire history.

That story should haunt every product leader who has ever said, "I just know this is the right call." It should also haunt every product leader who has ever said, "Let the data decide." Neither instinct nor data would have surfaced that change on its own. Instinct failed -- nobody believed in it. Data could not help until someone actually ran the test.

This is the real tension at the heart of product decision-making. Not "data vs. intuition," but something far more slippery: knowing when to trust your gut, when to trust the numbers, and when both are lying to you.

Nine conversations from the Lenny's Podcast archive reveal that the best product leaders in the world disagree sharply on this question. But when you lay their positions side by side, a surprising consensus emerges -- and it has nothing to do with which approach is "better."

The 4 Positions

1 / 4

Trust the Gut" -- Ship on Conviction

Jason Fried · Karri Saarinen · Peter Deng

Jason Fried has run 37signals (Basecamp, HEY) profitably for over 20 years with roughly 70 employees, competing against companies with 1,500 to 3,000. He has no investors, no board, and no one who can fire him -- a privilege he openly acknowledges. That structural freedom has allowed him to test a radical proposition: what happens when you run a software company on pure intuition?

"I'll just admit that that's the only way I've ever known how to work. I don't know how to make decisions by numbers or I don't find any joy in it, frankly. I've always been intuition driven and gut driven. And frankly, to be honest, I think everyone actually is."
Jason Fried
▶ 00:44:15

Fried's argument is not anti-intellectual. It is an epistemological claim: even "data-driven" decisions are judgment calls in disguise. Someone chose what to measure, how to interpret the results, and what to do about them. That chain of choices is itself driven by intuition. He points to hiring as proof -- when companies seek executives, they look for "judgment and experience," not the ability to read spreadsheets.

"I think everything's a judgment call. And so yeah, data can play a part. All sorts of things can play a part, but unless you're letting a machine make the decision, that's purely rational. If you're asking a human to make it, it's a judgment call."
Jason Fried
▶ 00:44:15

37signals operates without financial goals, OKRs, KPIs, or revenue targets. The only check is binary: did we make more than we spent? The dominant questions at work are about feeling: "How does this screen feel? How does this flow feel? How does this feature feel?" Fried is currently writing a book celebrating gut and intuition in business, arguing most business literature gets this backward.

Karri Saarinen, co-founder and CEO of Linear, pushes this further. A designer who built what may be the most beloved issue tracking tool in the world, Saarinen has proven that taste-driven development can produce extraordinary results: profitable for 2+ years, net negative lifetime burn rate, $35K total ever spent on marketing, and only two employee departures in the company's history.

"We don't do A/B tests. We validate ideas and assumptions that are driven by taste and opinions, rather than the other way around where tests drive decisions."
Karri Saarinen
via Lenny's Newsletter

Linear does not measure activation. Saarinen's reasoning: in B2B tools, the real "activation" can take months or years as teams gradually adopt the product. Optimizing a short-term activation metric would lead them to build the wrong product. Instead, Linear hires for product sense, taste, and judgment across all roles -- engineering and design included -- and uses paid work trials where candidates work alongside the team for one to five days on a real project. The insight is that collective taste, systematically hired for, can substitute for metrics.

Their development process reveals what taste-driven actually looks like in practice. Teams ship features to production within the first week -- visible only to themselves and opt-in beta customers. They iterate based on direct feedback, not dashboards. An engineer independently built dynamic safe zones for right-click submenus, mimicking the Mac OS behavior -- the kind of craft improvement that emerges from ownership, not from a metric telling you your submenu miss-click rate is too high.

"My belief is that, like any domain or industry, the more it matters, the more the design matters. As you build the 100,000 different email clients, any email client now has to be pretty good to be even considered an email client."
Karri Saarinen
▶ 00:00:00

Peter Deng, who led product at Instagram and Uber and is now a general partner at Felicis, reinforces this with a striking dual perspective. At Uber, he learned the painful lesson that "the price and the ETA at Uber was the product" -- the pixels did not matter much. But at Instagram, the opposite was true:

"The idea was super simple. It was showing photos and visual sharing. But the craft that Mike and Kevin had in putting in the hard work to get the product just right, that's what made it really take off. It wasn't anything that any other company couldn't have done, but it was that product taste that Kevin and Mike had."
Peter Deng
▶ 00:25:10

Deng adds a forward-looking claim: even companies with huge technological breakthroughs will eventually face the moment when product experience starts mattering. "How long does that technological advantage last before humans wisen up?" -- a question with direct relevance to every company building on LLMs today.

When This Applies

You are pre-product-market-fit and still searching for what to build
Craft and design quality are the primary differentiators (Linear in project management, Instagram in photo sharing, Figma in design tools)
Your product has long adoption cycles where short-term metrics are meaningless (B2B developer tools, enterprise software)
You are a small, founder-led team that can iterate rapidly and absorb mistakes (37signals with 70 employees vs. 1,500+ at competitors)
Technological advantages are commoditizing and taste becomes the differentiator (current AI/LLM landscape)

Test Everything" -- Data as the Honest Broker

Ronny Kohavi · Naomi Gleit

Ronny Kohavi is widely regarded as the world's leading expert on A/B testing. He built experimentation platforms at Microsoft (which ran 20,000-25,000 experiments per year by 2019), Amazon, and Airbnb, and authored the definitive textbook on online controlled experiments. His position is grounded in a devastating statistic:

"Overall at Microsoft, about 66%, two thirds of ideas fail. At Bing, which is a much more optimized domain after we've been optimizing it for a while, the failure rate was around 85%. And then at Airbnb, this 92% number is the highest failure rate that I've observed."
Ronny Kohavi
▶ 00:13:31

At Airbnb search relevance, his team ran 250 experiments. Only 8% improved the target metric -- but collectively, those wins drove a 6% improvement to revenue. Nine out of ten ideas -- researched, debated, designed, built, and shipped by smart, experienced people -- did not work.

"Very humbling. I know that every group that starts to run experiments, they always start off by thinking that somehow, they're different. And their success rate's going to be much, much higher, and they're all humbled."
Ronny Kohavi
▶ 00:15:17

Kohavi directly addresses the concern that experimentation leads only to micro-optimizations. He frames it as portfolio management: you need incremental experiments that compound over time (Bing's relevance team targeted 2% annual improvement through many small gains), but you also need high-risk, high-reward bets. "I don't think it's possible to experiment too much," he says. "I think it is possible to focus on incremental changes." The key is maintaining a portfolio of both.

Even the famous Airbnb experiment of opening listings in a new tab -- one of search's biggest wins -- was heavily debated and resisted by designers. Kohavi had run the same experiment at Microsoft years earlier and knew it worked, but institutional memory had been lost. The data proved the designers wrong, again.

Naomi Gleit, Meta's head of product and its longest-serving executive after Zuckerberg (employee #29), represents the practical application of data-driven thinking at historic scale. She was on the founding growth team that invented the modern playbook for product-led growth.

"What the growth team really pioneered was being data-driven and product-driven, especially in an area that was historically more of a business function. The insight that we had is actually the product is the biggest lever to drive growth."
Naomi Gleit
▶ 00:28:48

In January 2009, the growth team made a radical call: they stopped all roadmap work to focus exclusively on data instrumentation. Every step of registration, every step of onboarding, every point of drop-off -- all measured. Alex Schultz's mantra was on a t-shirt: "I guess when you can know."

But Gleit offers a revealing concession. When Lenny pressed her on the legendary "7 friends in 10 days" activation metric, she admitted she had to go back and ask former colleagues -- and they thought it was "10 in 14." The specific number may never have mattered:

"I think the majority of the value is in the latter, is just having extreme clarity around the goal and that allowed everybody to work towards optimizing the same goal."
Naomi Gleit
▶ 00:34:19

If the value of the activation metric was primarily about organizational alignment rather than measurement precision, then the data-driven and intuition-driven camps may be closer than they appear. You need the discipline of a shared goal. The exact number is secondary.

When This Applies

You have a large user base where you can run statistically significant experiments (Meta with billions of users, Microsoft with 100+ treatments per day)
You are optimizing a mature product in a well-understood domain (Bing search relevance, Airbnb search ranking)
The cost of shipping a bad change is high (payments, health, safety-critical features)
You need organizational alignment -- sometimes the specific metric matters less than having a shared goal (Facebook's "7 friends in 10 days")

Intuition Sets the Direction, Evidence Checks the Work

Dylan Field · Itamar Gilad

Dylan Field, CEO and co-founder of Figma, offers what may be the most elegant synthesis. He built Figma from a startup into the dominant design tool, displacing Adobe's own products -- demonstrating that taste-informed decisions can win even against massive incumbents.

"I think intuition is like a hypothesis generator and you're constantly generating these hypotheses and others are generating hypotheses as well. And you then take these hypotheses and you put them forward and you debate them and you try to find data to support them or negate them. And then you winnow it down into what is our working hypothesis? And from that you move forward."
Dylan Field
▶ 00:11:19

This is a genuinely useful mental model. Intuition is not a replacement for data -- it is the engine that generates the questions data can answer. Without intuition, you have nothing to test. Without data, you have no way to know if your intuition is right. The quality of your hypothesis generator depends on the breadth and quality of your information intake: Field reads every tweet mentioning Figma, monitors support channels, and constantly asks questions to find root problems.

Itamar Gilad lived both extremes at Google. A product coach and author of Evidence-Guided, he worked inside Google during both its greatest product failure and one of its greatest product successes -- giving him firsthand experience of what happens when the same company uses opposite approaches.

Google+ was opinion-based development at its most expensive. Google committed over 1,000 engineers to a division the size of Android and Docs, housed in its own buildings. The leadership had conviction. Gilad himself was "the guy pushing for the ideas." After years of work, Google+ was shut down and all Gmail integration was rolled back. Meanwhile, Google missed WhatsApp and Snapchat.

"We come up with an idea, we believe in it, all the indications show it's good. Maybe the early tests show it's good, then we just go all in and we try to implement it. And I made this very mistake many times as the product manager."
Itamar Gilad
▶ 00:07:13

Gmail's tabbed inbox was the reverse. It started as a small idea nobody believed in. Gilad wanted to push it through, but his colleagues challenged him: "We actually tried this. We have a bunch of ideas to help people organize their inbox, they're not using it. Why is your idea good?" That pushback forced progressive evidence-gathering: testing on their own inboxes, recruiting other Googlers, running usability studies, even building a Wizard of Oz test where users saw what appeared to be a working tabbed inbox but emails were manually sorted behind the scenes.

"Every successful product company out there that you look at Amazon, Airbnb, anyone you will check, at least in their best periods they found a way to balance human judgment with evidence. They didn't try to obliterate human judgment and opinion just to supercharge them with evidence."
Itamar Gilad
▶ 00:12:45

The failure mode is not having opinions. It is having too much confidence in your opinions before you have tested them.

When This Applies

You have conviction but the stakes are significant (Gmail's tabbed inbox: start with Wizard of Oz tests, scale investment with confidence)
You are making a large resource commitment (the bigger the bet, the more evidence you need -- Google+ committed 1,000 engineers without it)
Your product has both a "core experience" (taste-driven) and a "growth engine" (data-driven), like Shopify's deliberate dual structure

Your Metrics Are Probably Lying

Archie Abrams · Maggie Crowley

This is the most unsettling position because it undermines the data camp from within. Archie Abrams, VP of Product and Head of Growth at Shopify, leads a 600-person org at a company that handles roughly 10% of US e-commerce ($235 billion GMV in 2023). As head of growth, he has every incentive to champion short-term metrics -- which makes his warning about their unreliability especially credible.

Shopify deliberately sets up different parts of the organization with different decision-making approaches. Core product builds toward Tobi Lutke's 100-year vision with no KPIs -- they are essentially banned. Growth runs experiments but with uniquely long time horizons: holdout groups maintained for one to three years, with their experimentation tools automatically emailing all experimenters with updated results at 3, 6, 9, and 12 months. You cannot hide from what your experiment actually produced.

"In quite a few cases, you get a lift on a metric up front, a more short-term metric. Number of people who become a paying shopper, number of people who make their first sale in Shopify. And then you look a year later, and there's actually no incremental lift on GMV from that cohort."
Archie Abrams
▶ 00:14:59

How often does this happen? "It's in the 30 to 40% range." Roughly a third of experiments showing early lift have no long-term effect. The short-term wins were pull-forward effects -- revenue moved from next month to this month and called growth.

"I would encourage everyone, if you can, look at some of the experiments that you thought were your biggest winners. Look at the downstream metrics for a year, two years on that experiment. And I'll bet you'd be surprised how many times the metric is different than what you thought it would be after a year."
Archie Abrams
▶ 00:14:17

But the insight is not simply "metrics are broken." In some cases, Shopify discovered pockets of merchants who were extremely valuable entrepreneurs that short-term metrics missed entirely. When they reduced monetary friction for new merchants, conventional wisdom predicted lower-quality sign-ups. Instead, they causally changed some merchants' ability to succeed by giving them more time to validate their business ideas. Short-term metrics would have killed this experiment. Long-term holdouts revealed its real value.

Maggie Crowley, VP of Product at Toast, attacks from a different angle. Having hired and managed PMs across startups at every stage (TripAdvisor, BevSpot, Drift, Charlie Health, Toast), she sees "data-driven" as a red flag -- not because data is bad, but because the label becomes a crutch that replaces genuine product thinking.

"It's one tiny slice. You do a strategy, but it's 5% of the work that you do. Yes, it's important because you want to get your strategy and you need to pick the right products, but at the end of the day, the person who has a good strategy will not be as successful as the PM who ships more stuff, gets more reps and has the ability to actually create impact."
Maggie Crowley
▶ 00:09:50

Crowley places analytical ability as a "basic of the role" -- table stakes, not a differentiator. What separates great PMs from good ones is reps: shipping, failing, learning, and repeating. The PM who hides behind data -- who always needs one more analysis before making a decision -- may be avoiding the reps they need to build the intuition that would make them truly effective.

When This Applies

Your biggest experiment "wins" have not translated to business outcomes -- Shopify found 30-40% of early lifts wash out
You have never run a long-term holdout on your most important experiments
Short-term metric lifts are the primary way your team justifies its work
You are using "data-driven" as a label to avoid the hard, unglamorous work of building genuine product sense (Maggie Crowley's warning)

Evidence from the Archive

Airbnb, Microsoft, Amazon

Airbnb search relevance: 250 experiments, only 8% improved target metric, but collectively drove 6% revenue improvement

Bing ad text change: moving second line to first line generated $100M/year in revenue -- languished in backlog for months because nobody's intuition flagged it

Ronny Kohavi · VP/Technical Fellow (Experimentation) ▶ 00:21:07

Shopify

30-40% of experiments showing short-term lift have no GMV lift a year later

Shopify's three product groups: core (100-year vision, no KPIs), merchant services (medium-term), growth (experiments with long-term holdouts)

Archie Abrams · VP Product, Head of Growth ▶ 00:14:59

37signals (Basecamp/HEY)

No OKRs, KPIs, revenue targets, or growth targets -- only 'did we make more than we spent?'

37signals runs Basecamp and HEY with ~70 employees vs competitors with 1,500-3,000 employees, with similar customer counts and higher profitability

Jason Fried · Co-founder and CEO ▶ 00:44:15

Linear

Profitable for 2+ years, $35K total marketing spend, net negative burn rate, only 2 departures ever

Linear has no A/B tests, one PM (Head of Product), and no metrics-based goals

Karri Saarinen · Co-founder and CEO ▶ Watch

Facebook growth team stopped all roadmap work in January 2009 to focus exclusively on data instrumentation

7 friends in 10 days (or 10 in 14) -- even the original team is not sure which was first, suggesting the specific number mattered less than the shared goal

Naomi Gleit · Head of Product (employee #29) ▶ 00:28:48

Gmail, YouTube, Microsoft

Google+ consumed 1,000+ engineers over multiple years, built a division the size of Android, and was shut down in...

Google+ consumed 1,000+ engineers over multiple years, built a division the size of Android, and was shut down in 2019 -- opinion-based development at its most expensive

Itamar Gilad · Product coach, author of Evidence-Guided ▶ 00:12:45

The Synthesis

When you lay these nine perspectives on top of each other, the "data vs. intuition" framing collapses. The right decision-making approach is not a philosophical choice. It is a function of three variables: the cost of being wrong, the speed at which you can correct mistakes, and the time horizon over which value is created.

Three Decision Variables

What actually determines whether to use data or intuition?

Data Misleading Risk

Can being data-driven actually lead you astray?

Hypothesis Generator

How do intuition and data actually work together?

Intuition as Learnable Skill

Is product intuition a gift or a developed skill?

The right approach is a function of three variables: the cost of being wrong, the speed at which you can correct mistakes, and the time horizon over which value is created. Pure intuition works when cost of being wrong is low and iteration speed is high. Rigorous testing is essential when a bad change hits hundreds of millions of users.

If 30-40% of short-term experiment wins wash out over a year, the standard two-to-four-week experiment window is actively misleading. Being 'data-driven' over the wrong time horizon means being data-misled. A change that looks like a loss at two weeks might be a massive win at six months.

Intuition generates the questions, data answers them. Stop generating questions (pure data optimization) and you converge on local maxima. Stop answering questions (pure intuition) and you get Google+ -- massive resource commitments to untested beliefs. The pathology comes when you skip one side.

Product intuition is defined as empathy plus creativity, developed through observing users, deconstructing products, and curiosity about technology. This reframes the debate: the question is not whether to use data or intuition, but whether you have invested enough in developing the quality of both your measurements and your judgment.

Which Approach Fits You?

Answer 3 questions about your situation. We'll match you to the right approach.

Question 1

What is the cost of being wrong with this decision?

Question 2

How much data do you have to work with?

Question 3

What is your product's time horizon for delivering value?

Notable Absences

The Bottom Line

The best product leaders are not "data-driven" or "intuition-driven." They are *well-calibrated*. They know when they are in a domain where their intuition is reliable -- high reps, fast feedback, low cost of failure -- and when they are in a domain where it is not. They hold convictions loosely enough to let evidence reshape them, but tightly enough to avoid the paralysis of testing everything.

If you take one thing from this analysis, let it be Archie Abrams's challenge: go look at your biggest experiment wins from a year ago. Check the downstream metrics. If the lift held, your process is working. If it did not, you may be optimizing for an illusion -- and a well-calibrated leader with strong product taste would have served you better than a thousand A/B tests.

Sources

Ronny Kohavi — "The ultimate guide to A/B testing" — Lenny's Podcast, July 27, 2023
Itamar Gilad — "Becoming evidence-guided" — Lenny's Podcast, September 21, 2023
Karri Saarinen — "Inside Linear: Building with taste, craft, and focus" — Lenny's Podcast, October 8, 2023
Maggie Crowley — "Mastering product strategy and growing as a PM" — Lenny's Podcast, November 5, 2023
Jason Fried — "Jason Fried challenges your thinking on fundraising, goals, growth, and more" — Lenny's Podcast, December 17, 2023
Dylan Field — "Figma's CEO: Why AI makes design, craft, and quality the new moat for startups" — Lenny's Podcast, June 30, 2024
Naomi Gleit — "Meta's Head of Product on working with Mark Zuckerberg, early growth tactics, and more" — Lenny's Podcast, October 27, 2024
Archie Abrams — "Breaking the rules of growth: Why Shopify bans KPIs, optimizes for churn, and builds toward a 100-year vision" — Lenny's Podcast, November 7, 2024
Peter Deng — "From ChatGPT to Instagram to Uber: The quiet architect behind the world's most popular products" — Lenny's Podcast, June 22, 2025
Karri Saarinen — "How Linear builds product" — Lenny's Newsletter, September 26, 2023
Jules Walter — "How to develop product sense" — Lenny's Newsletter, March 15, 2022