UX Research

in the age of the reproducibility crisis


Our story starts with



Field study

High power poses

Low power poses

Reproducibility crisis

Reproducibility project

You'll never believe what happened when scientists attempted to replicate 100 experiments...

Just wait til you see their results

"97% of the original results showed a statistically significant effect, this was reproduced in only 36% of the replication attempts"


And then this happened...
"...the idea became a shorthand for flashy social psychological work that could not be replicated..."


What's to blame?

  • "Publish or perish"
  • No replication studies
  • Clickbait


Sound similar to UXR?

  • "Prove me right"
  • Demanding shortcuts
  • Preferring 'hard data' over qual data
UXR has different goals to Science


Chipping away at the crystal of knowledge


Reducing business risk

So does reproducibility even matter in UXR?

Well yes...

we are

Science lite

Said with ❤️

Debunked ideas

Method/problem mismatches

Make the hard problem of internal buy-in harder

So, should UX Research be reproducible?

My answer is...


*It depends

At least three levels to reproducibility

1. Experiment design

  • Model & Methodology
  • Sample size
  • Sample selection

2. Data captured

3. Interpretation

UX Research types

Qualitative -vs- Quantitative


  1. Experiment design
  2. Data captured ×
  3. Interpretation ×


  1. Experiment design
  2. Data captured
  3. Interpretation
  4. Hopefully 🤞

How do we do it better?

Get inspired by the Open Science movement

Within the constraints of our current workplaces...

Crunching numbers

So you want statistical significance

  • Work on your research design hygiene
  • Get better at understanding the strength of your signal

Embrace uncertainty

Like a scientist

  • Avoid speaking in absolutes
  • Cultivate your curiousity
  • Reward "I don't know"

Define your study
you start

Plan for no conclusion
(unclear results)

Replication studies

Start by identifying a good candidate study


  • expect the effect to be consistent over time
  • the hypothesis is important to business model functioning
  • product has been consistent since last study


  • don't expect the effect to be consistent
  • low risk to business model or low business priority
  • UX or UI in flux; too many variables to control for

Qual data

Consider trying synthesis and interpretation with multiple groups of researchers

Fact check

  • Learn to read references
  • Paper abstract is a good start
  • Develop your sniff test

Open UX...?

Continuous discovery...?


  • Build culture for continuous learning
  • Embrace uncertainty
  • Replication studies (maybe)
  • Spend more time on research design & analysis
  • Uncertain results are ok
"In short, be sceptical, pick a good question, and try to answer it in many ways. It takes many numbers to get close to the truth."

It’s time to talk about ditching statistical significance


Get the slides

Power Pose

A/B Tools

Further reading