4 guests 6 episodes 2,432 words

The User Research Trap: Why Most Teams Do Research to Feel Productive, Not to Learn

How much should user research influence product decisions?

Jeff Weinstein at Stripe leaves meetings to text customers back. Kevin Yien argues most user research is performative. Naomi Gleit stopped everything at Facebook for data instrumentation before any research could begin. Scott Belsky says the biggest mistake is falling in love with a solution before understanding the problem.

There is a dirty secret in product development: most user research exists to make teams feel rigorous, not to actually change decisions. The research happens, the insights are presented, and then the team builds what they were going to build anyway. The teams that actually learn from users look nothing like the teams that conduct formal research programs.

How much should product teams invest in user research? What forms of research actually change decisions, and what forms are performative -- creating the illusion of customer-centricity without the substance?

The 4 Positions

1 / 4

Automate and Personalize -- Make Research Continuous, Not Episodic

Kevin Yien (Product lead · Stripe · Square · Mutiny)

Advocates: Kevin Yien (Product lead, Stripe, Square, Mutiny)

Having led product at Stripe (merchant experiences), built the restaurant business at Square, and headed product and design at Mutiny, Kevin Yien has operated across B2B and B2C contexts where customer research directly drives product decisions at massive scale. His position begins with a provocative redefinition of product sense.

"We all talk about product sense. To me, it's just a fancy way of saying you can make good decisions with insufficient data. PMs need as many reps as possible in making decisions, documenting them, and seeing how they played out."
Kevin Yien
▶ 00:00:13

This framing is important because it positions research not as a discrete activity but as a means of building decision-making muscle. PMs need direct exposure to raw material -- not research reports processed by someone else, not insights relayed by a salesperson, not market research filtered through an analyst.

"No PM should settle for looking through bent glass in my opinion because whether it's a research report, whether it's something a salesperson is telling you, whether it's market research, it's been processed by someone, and PMs need direct exposure to raw material. End of story."
Kevin Yien
▶ 00:54:39

Rather than conducting formal research sprints before big decisions, Yien advocates building automated customer research pipelines that deliver continuous signal. The system should be designed so that customer insights flow to product teams as a default, not as a special effort. His decision log practice -- documenting rationale and checking outcomes -- builds product sense through deliberate reps over time.

"If you are in a situation where the product manager is literally not allowed to talk to a customer, there is something structurally wrong."
Kevin Yien
▶ 00:55:08

At Stripe, Yien advocates PMs maintain direct text and message relationships with key customers. At Square, building the restaurant business required deep direct customer understanding of a specific vertical -- not a formal research program but sustained immersion. He explicitly challenges the notion that UXR teams alone should own customer understanding -- while their work is valuable, PMs who delegate all customer contact to researchers are abdicating a core responsibility.

When This Applies

for genuinely new problems -- market entry, new product categories, or user segments you do not yet understand. Use it to explore, not to validate decisions you have already made.

Obsessive Direct Contact Over Formal Research

Jeff Weinstein (Product lead · Stripe)

Advocates: Jeff Weinstein (Product lead, Stripe)

As the product lead who scaled Stripe payments to hundreds of billions in volume and grew Stripe Atlas to incorporate one in six new Delaware corporations, Jeff Weinstein has demonstrated that obsessive customer closeness drives both craft and business results at extraordinary scale. His approach is grounded in a formative failure.

At his previous startup, the service went down for 20 minutes and customers were not furious -- they were not emailing, texting, or showing up at the office. At the time, Weinstein celebrated the quick recovery. Only a year later did he realize that the lack of outrage was the signal: they did not have product-market fit. He wasted years because he was not listening closely enough to what customer behavior was actually telling him.

"Craft is a dessert that you get after the meal of does your thing solve a real problem in the world and are people clamoring, needing it badly?"
Jeff Weinstein
▶ 00:18:55

His tactical approach is remarkably specific. He shares his email constantly on Twitter, gets on calls with customers directly, and maintains text message relationships with five to ten key customers.

"If someone goes out of their way to talk about some problem, that's an unbelievable gift. I will leave a meeting to just get one message back to them. If you're text message friendly with five or 10 of those, you are going to have so much direct signal that is infectious."
Jeff Weinstein
▶ 00:18:55

His study groups at Stripe represent a structured alternative to traditional research.

"We show up four to eight people total pretend to be some company with some outcome problem. Rule one is you do not work at Stripe and rule two is we're not here to solve any problems. This is just about practicing empathy for the customer."
Jeff Weinstein
▶ 00:00:39

The two rules reveal the methodology's wisdom: drop your assumptions and resist the urge to jump to solutions. This is pure empathy practice -- understanding the customer's world without the distortion of your product lens. These study groups do not produce research reports or slide decks. They produce something more valuable: internalized customer empathy that influences hundreds of small decisions over the following months.

"People don't really get out of bed for their second problem. They get out of bed for their first problem. And you have to carefully listen and not pitch your customer."
Jeff Weinstein
▶ 00:20:34

Weinstein's approach to selecting which customers to listen to is also nuanced. At Stripe's scale of millions of businesses, you pattern-match for the most ambitious, most technical, fastest-growing, most detailed customers. You do not need 10,000 of those relationships -- five to ten direct connections provide enough signal.

When This Applies

always, regardless of company stage. Every PM should have 5-10 customers they can text. This is the foundation -- Stripe, Square, and Mutiny all demonstrate that this direct channel surfaces problems that formal studies miss.

Data Instrumentation First, Then Research

Naomi Gleit (Head of Product · Meta)

Advocates: Naomi Gleit (Head of Product, Meta)

As Meta's longest-serving executive after Zuckerberg (employee #29, 19+ years), Naomi Gleit helped build the legendary Facebook growth team that pioneered data-driven product growth and invented frameworks like the activation metric (7 friends in 10 days) now used across the industry. In January 2009, the growth team made a radical decision.

"In 2009 in January, we basically stopped doing anything on our roadmap except data instrumentation. And that's when we instrumented every step of the registration flow, instrumented every step of the new user onboarding experience."
Naomi Gleit
▶ 00:29:43

They stopped all product work for data instrumentation. Only then could they identify opportunities and execute by building products. The philosophy is captured by Alex Schultz's shirt slogan.

"Alex often wears a shirt that says, 'I guess when you can know.' We just didn't have the data that we needed to make informed decisions to know really what were the biggest levers to drive growth."
Naomi Gleit
▶ 00:29:43

Quantitative data should precede qualitative research so you know exactly where to dig deeper. The registration flow drop-offs told them where to focus; qualitative research could then explain why. The growth accounting framework revealed that churn and resurrection lines were much larger than the new user acquisition line, implying that retention was the biggest lever -- an insight invisible without instrumentation.

"Having this data-driven product-driven approach to what I think historically was more of a business responsibility at a company was sort of the special sauce of the growth team."
Naomi Gleit
▶ 00:30:50

The genius of the growth team was not just being data-driven but being product-driven about traditionally business-owned functions. Before the growth team, growth was expected to come from marketing or communications. The insight was that the product itself is the biggest lever for growth. Community translation replaced professional translators -- a product-driven approach to internationalization. Social impact became a product team, not a CSR wing, eventually raising billions from the community. Gleit's drive for canonical documents -- one single source of truth for every project -- reflects the same philosophy applied to research: rather than hundreds of scattered insights, have one clearly instrumented view of what is happening and why.

When This Applies

when you have enough users to see patterns (typically 1,000+) and when you are optimizing an existing product. Facebook's growth team proved that knowing WHERE the problems are before investigating WHY eliminates wasted research effort.

Empathy and Curiosity Over Formal Methods

Scott Belsky (CPO/CSO · Adobe and Behance)

Advocates: Scott Belsky (CPO/CSO, Adobe and Behance)

As the founder of Behance and former CPO/CSO of Adobe overseeing design, product, and engineering, Scott Belsky has applied empathy-driven product development at both startup and Fortune 500 scale, including reimagining first-mile experiences for products like Photoshop used by millions.

"First of all, I think the biggest mistakes that teams make is they become very passionate about a solution to a problem they're trying to solve before they even fully understand the problem."
Scott Belsky
▶ 00:08:56

Belsky's framework revolves around what he calls the "first mile" -- the first thirty seconds of a user's experience with any product. In those moments, users are governed by psychology that most research methods fail to capture.

"In the first 30 seconds of using a new product, you are lazy, vain, and selfish. You want to get it done super quickly. You want to look good to your colleagues or to your friends. You want to feel successful very quickly."
Scott Belsky
▶ 00:09:51

Most teams spend the final mile of building considering the first mile of the customer's experience, which is backwards. If you can get more customers through that top of funnel, you are a world-class product team.

The research method Belsky advocates is shoulder-to-shoulder customer observation -- not just watching customers use your product, but watching them go about their entire day.

"The most humbling moments for me as a product leader have always been shoulder to shoulder to customers. Watching them actually go about their day, not just use my product but go about their day because what you end up getting is context for a lot of data that you're missing."
Scott Belsky
▶ 00:13:41

In the enterprise, that means understanding the other meetings, other products, and pings they deal with. As a consumer product, it means understanding the context of kids, Netflix, and everything else competing for attention. This contextual understanding is what formal research sessions, which isolate the product from its environment, often miss. Belsky also challenges the assumption that product-led growth comes from making products great. People do not talk about products that do exactly what they expected -- they talk about products that do what they did not expect. Even with a product like Photoshop, which has existed for decades, the first mile still requires constant reimagination because each new cohort of customers is different. New users at $10/month are fundamentally different from the professionals who paid hundreds of dollars.

When This Applies

regularly, especially when your team is becoming too insular. Stripe's study groups and Adobe's shoulder-to-shoulder observation both demonstrate that internalized empathy influences decisions in ways that reports cannot.

Evidence from the Archive

Facebook growth team stopped all product roadmap work in January 2009 for pure data instrumentation

The 7 friends in 10 days (or 10 friends in 14 days) activation metric became the North Star for retention

Naomi Gleit · Head of Product ▶ 00:29:43

Stripe

Weinstein's previous startup went down for 20 minutes and customers barely noticed -- the missed signal cost years

Stripe study groups: 4-8 people roleplay as a company with a problem, practicing empathy without solving

Jeff Weinstein · Product lead ▶ Watch

Stripe, Square, Mutiny

At Stripe, Yien advocates PMs maintain direct text/message relationships with key customers

His decision log practice -- documenting rationale and checking outcomes -- builds product sense through deliberate reps

Kevin Yien · Product lead ▶ 00:00:13

Adobe, Behance

At Adobe, Belsky broke down boundaries between design, product, and engineering to ship better experiences

Photoshop at $10/month attracts fundamentally different users than at hundreds of dollars -- requiring first-mile reinvention

Scott Belsky · CPO/CSO ▶ 00:08:56

The Synthesis

The four perspectives converge on a provocative conclusion: formal user research -- the kind with screener surveys, recruited participants, discussion guides, and insight decks -- is often the least valuable form of customer understanding.

Formal Research Paradox

Is formal user research the most or least valuable form of customer understanding?

Deliverables vs. Mental Models

Why does most formal research feel performative?

Embedded and Continuous

How should user research be restructured?

Formal user research -- with screener surveys, recruited participants, discussion guides, and insight decks -- is often the least valuable form of customer understanding. The most valuable forms are direct personal relationships with customers, instrumented behavioral data, and empathy exercises.

Most formal research is structured to produce deliverables (reports, presentations) rather than to change mental models. A 40-page research deck nobody reads is less valuable than a 2-minute customer text exchange that shifts how a PM thinks about a problem.

The practical implication is to restructure research to be continuous (not episodic), embedded (not siloed), and focused on changing how teams think (not on producing deliverables). All three most valuable forms produce understanding integrated into daily decision-making.

Which Approach Fits You?

Answer 3 questions about your situation. We'll match you to the right approach.

Question 1

How close are you to your customers?

Question 2

What is your biggest research challenge?

Question 3

How many users do you have?

Notable Absences

The Bottom Line

The practical implication is not to eliminate formal research -- it is to restructure it. Research should be continuous (not episodic), embedded (not siloed), and focused on changing how teams think (not on producing deliverables).

The non-obvious insight: the reason most formal research feels performative is that it is structured to produce deliverables (reports, presentations, recommendation decks) rather than to change mental models. A 40-page research deck that nobody reads is less valuable than a 2-minute customer text exchange that shifts how a PM thinks about a problem.

Sources

Kevin Yien — "Unorthodox PM wisdom: Automating user insights, unselling job candidates, logging every decision, more | Kevin Yien (Stripe, Square, Mutiny)" — Lenny's Podcast, August 18, 2024
Jeff Weinstein — "Building product at Stripe: craft, metrics, and customer obsession | Jeff Weinstein (Product lead)" — Lenny's Podcast, July 11, 2024
Naomi Gleit — "Meta’s Head of Product (and 29th employee) on working with Mark Zuckerberg, early growth tactics, why PMs are like conductors, and more | Naomi Gleit" — Lenny's Podcast, October 27, 2024
Scott Belsky — "Lessons on building product sense, navigating AI, optimizing the first mile, and making it through the messy middle | Scott Belsky (Adobe, Behance)" — Lenny's Podcast, May 18, 2023