Running experiments in your product will tell you what your customers are doing, but they don't often tell you why they're doing it. So even after experimenting, you can be left wondering: which option would be better to ship? One way that Atlassian has tackled this is by taking the "concept testing" technique to a whole new level. Get a first-hand look into how we run concept tests to extract the right insights – before a single line of code has been written. You'll learn how they can work side-by-side with quantitative experimentation to help you get smarter metrics, and have greater confidence in knowing how to take your product from good to great.

1. BEN CROTHERS • PRINCIPAL DESIGN STRATEGIST • @BENCROTHERS
Going from Good to Great
with Concept Testing

2. CHALLENGE:
Moving beyond
iteration to innovation

3. CHALLENGE:
Making product
decisions confidently

4. CHALLENGE:
Creative confidence

5. CONCEPT TESTING:
Helping you gain more
creative confidence

6. EXAMPLES IN ACTION
WHAT IS CONCEPT TESTING
WORKING IT INTO YOUR PROCESS
BEING MORE CREATIVE AND CONFIDENT
What I’ll cover

7. What is
concept testing?

8. Open inquiry
Getting customers to use raw low-
fidelity concepts and prototypes.
Facilitated groups
Structured activities and discussions
about how people work, and why.
Raw prototypes
Inviting new ideas, and uncovering
happy accidental insights.
What is
concept
testing?

9. Removing risk
Gathering data about customers’
jobs, tasks, goals and mental models.
Validating early
Busting assumptions, checking if we
solve the right problem, not just how.
Customer context
Failing refining early before spending
serious $$$ on development.
What is
concept
testing
for?

10. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace
Capturing real goals and
work patterns

11. • Big cool statistic
• 2,56
9
• Add-Ons in MarketplaceFeedback about
what works,
not what they like

12. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace
Green for useful,
red for not useful

13. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace
Everyone’s invited
to observe

14. Results:
Welcome
done wrong

15. Results:
Welcome
done better

16. Results:
Workflow
editing done
wrong

17. Results:
Workflow
editing done
better

18. This or that…?
Testing various interface designs for
specific tasks (usability testing).
Button colours
Testing conversions based on fine-
grain interface differences (A/B T).
Testing screen design
It’s not a bake-off, but about
strengths and weaknesses of options.
What is
concept
testing
not for?

19. Where does concept testing happen?
PRODUCT BACKLOG
PRODUCT STRATEGY AND DESIGN
CONCEPT TESTING

20. How can
concept testing give you
creative confidence?

21. How can
concept testing give you
creative confidence?

22. How do we make
ourselves
less creative?

23. Stifling creativity #1: FEASIBILITY ANXIETY
We want to make sure our
ideas meet all 3 areas…

24. Lies, Ben!
Lies!

25. Stifling creativity #1: FEASIBILITY ANXIETY
We want to make sure our
ideas meet all 3 areas… …but Desirability
often lucks out.

26. Stifling creativity #2: REPUTATION ANXIETY
We want to unearth
THE BIG ONE…

27. Lies, Ben!
Lies!

28. Stifling creativity #2: REPUTATION ANXIETY
…but we end up under-rating
all the little ones.
We want to unearth
THE BIG ONE…

29. Stifling creativity #3: PERFORMANCE ANXIETY
We want our ideas to WIN…
…but the wins feel like a
let-down… like bingo.

30. Ideas
are a fluid collection of
connections, sparks,
meanderings and
questions

31. Innovation is a series of stones across a river
Problem
Solution Solution Solution

32. We want to come up
with ideas to learn
(not to win)

33. Example:
Real-time chat
while looking at a
task board?

34. Concept
testing helps
you cross
your river
It makes conversation
for a greater range of creative
possibilities.
It lets you be open
to new and unexpected insights and
directions.
It frees you up
to involve customers in your creative
problem-solving process

35. I’m in, Ben!
What’s one thing
I can try?

36. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace
Try it internally first

37. Thank you
BEN CROTHERS • PRINCIPAL DESIGN STRATEGIST • @BENCROTHERS

38. • Getty Images
• Pinterest.com
• Huffingtonpost.com
Image credits