UX Research

in the age of the reproducibility crisis



summerscope.github.io/slides/art-vs-science

Our story starts with

TED

ted.com/talks/amy_cuddy_your_body_language_shapes_who_you_are

Field study

High power poses

Low power poses

Meanwhile...
Reproducibility crisis

Reproducibility project

You'll never believe what happened when scientists attempted to replicate 100 experiments...

Just wait til you see their results

"97% of the original results showed a statistically significant effect, this was reproduced in only 36% of the replication attempts"

digest.bps.org.uk

And then this happened...
"...the idea became a shorthand for flashy social psychological work that could not be replicated..."

tandfonline.com/doi/full/10.1080/23743603.2017.1309876

What's to blame?

  • "Publish or perish"
  • No replication studies
  • Clickbait

P-hacking

blogs.discovermagazine.com/neuroskeptic/2015/11/10/reproducibility-crisis-the-plot-thickens/#.XKv_2Ov7TUJ

Sound similar to UXR?

  • "Prove me right"
  • Demanding shortcuts
  • Preferring 'hard data' over qual data
UXR has different goals to Science

Science

Chipping away at the crystal of knowledge

UXR

Reducing business risk

So does reproducibility even matter in UXR?

Well yes...

we are

Science lite

Said with ❤️

Debunked ideas

Method/problem mismatches

Make the hard problem of internal buy-in harder

So, should UX Research be reproducible?

My answer is...

Sometimes!*

*It depends

At least three levels to reproducibility

1. Experiment design

  • Model & Methodology
  • Sample size
  • Sample selection

2. Data captured

3. Interpretation

UX Research types

Qualitative -vs- Quantitative

Qualitative

  1. Experiment design ✔
  2. Data captured ×
  3. Interpretation ×

Quantitative

  1. Experiment design ✔
  2. Data captured ✔
  3. Interpretation ✔
  4. Hopefully 🤞

How do we do it better?

Get inspired by the Open Science movement

Within the constraints of our current workplaces...

Crunching numbers

So you want statistical significance

  • Work on your research design hygiene
  • Get better at understanding the strength of your signal

nngroup.com/articles/which-ux-research-methods/

Embrace uncertainty

Like a scientist

  • Avoid speaking in absolutes
  • Cultivate your curiousity
  • Reward "I don't know"


Define your study
before
you start

Plan for no conclusion
(unclear results)

Replication studies

Start by identifying a good candidate study

Good

  • expect the effect to be consistent over time
  • the hypothesis is important to business model functioning
  • product has been consistent since last study

Poor

  • don't expect the effect to be consistent
  • low risk to business model or low business priority
  • UX or UI in flux; too many variables to control for

Qual data

Consider trying synthesis and interpretation with multiple groups of researchers

Fact check

  • Learn to read references
  • Paper abstract is a good start
  • Develop your sniff test

Open UX...?

Continuous discovery...?

Recommendations

  • Build culture for continuous learning
  • Embrace uncertainty
  • Replication studies (maybe)
  • Spend more time on research design & analysis
  • Uncertain results are ok
"In short, be sceptical, pick a good question, and try to answer it in many ways. It takes many numbers to get close to the truth."

It’s time to talk about ditching statistical significance

Thanks!

Get the slides
summerscope.github.io/slides/art-vs-science

Power Pose

A/B Tools

Further reading

@summerscope