Presented at the Design Research 2007 conference.

1. Osaka, Japan
Several years ago, I did a research project in Japan for a major retailer who wanted to expand
overseas to Japan. We went to Osaka in order to understand shopping.

2. Umeda Marketplace

3. Denden Town, Nipponbashi

4. Gunkan Apartments

5. On-The-Street Interviews

6. Signage and Iconography

7. Interesting Observations

8. Cultural Norms

9. Design Principles
Use colorful iconography to attract and
comfort customers.
The street and store interiors are not as
heavily separated in Japan as they are
in the US. Bring the street into the store.
Display space should reflect those of a street
market, with items tightly pressed together
to create impressions of bounty and variety.
Here are several design principles we came up with.

10. 1. Don’t do any design research. Make it all up.
You just got your first lesson and rule #1. All the images you just saw, I collected off Flickr in one
afternoon. Voila, saved myself a trip to Osaka. I’ve never even been to Japan.

11. How to Lie
with Design Research
Dan Saffer
If you are reading this note and you aren’t me, please understand that this presentation is meant to
be delivered Stephen-Colbert style, as deadpan comedy. Please treat the following as entirely
tongue in cheek.

12. TIP: Don’t go into the field unless you have to.
Why do research when you don't really need to? Most of the time companies are just looking to have their
ideas validated. Why not give them what they want using carefully chosen photos and quot;storiesquot; from the
internet.

13. TIP: Wacky cultural practices always impress.
For quot;internationalquot; research, be sure to throw in a couple of unexpected cultural practices to make
people feel that they've really taken the time to consider diverse perspectives.

14. The Jayson Blair Method
Named after the New York Times reporter who did his on-site reporting from his apartment in
Brooklyn. I don’t recommend this method. Why?

15. TIP: Don’t lie about the easily (dis)provable.
Why? Because like Mr. Blair, you are eventually going to get caught! There are better--although not
EASIER ways of lying with research. A really good lie is one that is untraceable and cannot be pinned
on you or easily disproved.
But let’s back up a minute.

16. Why you’re really here
I know why you all came to this fine conference with such fantastic speakers like myself. It’s really a
privilege for all of you, isn’t it? But even though I’m sure you sold this to your boss as “helping
users” or “making better products” or even “user experience” (whatever that is), it’s really all about
money isn’t it? It’s ok, you can admit it. You came because you wanted to jump on the design
research bandwagon before we’re found out and it rolls away. And I’m here to help you do that.
Lying makes it easier to cash in! We got paid 500 grand for that Osaka study!

17. Why would you ever lie about research?
Now I know some of you are saying, Why would I ever lie about design research? People lie for the
same reason they design--to turn an undesirable situation into a better one. I have an idea I want
you to act on. I don’t want to be uncomfortable or embarrassed or feel stupid. I want you to think
that I really like that shirt. I want you to think that I’m really smart. I want you to do this project.

18. Research gives your ideas more credibility.
Suppose I have an idea for this cool hypnotic lamp. Wouldn’t it be great if I had some research that
said, “Users want to be hypnotized at their desks?” That’s not really lying, is it? Distorting perhaps.
But I bet right now, you are thinking, Dan, how (theoretically) might one go about doing that?

19. Let’s assume you have a good reason for lying.
But I’m not here to judge, but to teach and heal. I’m a uniter.

20. 2. Skew the research subjects in your favor.
Rule Number 2. In design research, your data is only as good as your participants. Choose subjects
that are going to bolster your case. The smaller and more alike the sample size, the better!

21. TIP: Diversity is overrated.
Research people who aren’t diverse. Diversity is overrated when stacking the deck in your favor. If
you are making a website for an automobile manufacturer, why not interview all auto enthusiasts?
Then you can say, without lying, “Everyone we talked to said they have spent hours online looking
at car sites!”
Get bonus points for calling this not diverse group an “Unfocus Group.”

22. TIP: Make unconscious bias, conscious.
We do this anyway, right? If we have two people to choose from, we often unconsciously choose the
person that we think is going to be the most cooperative, and astonishingly, that person often looks
just like us! If we’re talking to people, why in the world would we even want to talk to people who
are different from us?

23. What Would Johnny Do?
Pick your research subjects like the late defense attorney Johnny Cochrane picked juries. You want
the right mix of people who are going to give you the outcome you want. No wild cards. Subjects
who are going to give you the results that you want when you ask them for their stories and ideas.

24. 3. Don’t be objective.
Rule number 3. Leave the objectivity to the scientists and reporters. We’re designers. Why is there
this veneer that we need to be objective in our research? We’re not out there in our lab coats like
some pinhead trying to discover a cure for cancer. Who is lying to who now?

25. My concept:leading questions. great?
Ask really great or just
If we’re not objective, we can ask leading questions. [CLICK] Then you can put in your research
findings that “Subject X thought this was a great concept.” It’s not lying if they said it, right?

26. TIP: Use carefully considered wording to lead.
“Wasn’t that a confusing experience?”
“Isn’t this a great conference?”
Remember that your research subjects are people in the unfamiliar situation of being interviewed
and observed. They want to give you what you want and are looking to you for cues. Thus, you can
easily influence their responses through the phrasing of your questions.

27. 4. Toss out data you don’t like.
Don’t let a research subject ruin your design idea. When it comes time to analyze the data, it is
essentially about culling your data, separating the wheat from the chaff. You have to use some
criteria to make those calls. Some people use things like prevalence or, on the other hand,
uniqueness. Why not use your own preferences? Don't make the mistake of letting rigor or a quest
for objectivity lead you to keep data with implications you don't want to deal with.

28. TIP: Having data doesn’t mean using it.
Why do you have to use all the data? I mean, if you didn’t ask that question or if you weren’t sitting
there observing, you might not know that piece of data at all, right? Just because something is
known, doesn’t mean we have to use it!

29. quot;[A]s we know, there are known knowns; there are
things we know we know. We also know there are
known unknowns; that is to say we know there are
some things we do not know. But there are also
unknown unknowns, the ones we don't know we
don't know.quot; —Donald Rumsfeld
All you need to do is turn a known known in a known unknown. Simple enough.

30. TIP: See only what you want to see.
One way to avoid having to toss out data is to never observe it at all. Or at least not write it down.
Only look in the places you think you are going to get the data you want.

31. 5. Deliberately misrepresent the data.
Ok, once you have data, from one place or another, it’s time to present it. And it’s here that the
best lies can be told all in the name of “storytelling.” Your charts, graphs, and other visualizations
are just ways of telling a story about some data that was collected. Make sure it is YOUR story, not
the story of your subjects or the story of the data!

32. 100
90
80
70
60
50
40
30
20
10
0
2006 2007
How about this graph? Interest in adding a new feature that you want to add only went up 5 percent
over a year. It’s informative, yes. Perhaps too informative. Different approaches allow you to
highlight different aspects of the data. Don't miss this opportunity to emphasize how you were right
and suppress evidence that you weren't.

33. 100
90
55
80
70
60
50
40
30
20
10
0
2006 2007
Once you chop the top and bottom and the pesky numbers off of it, now we’re talking. That looks
much better. The old saying, “A difference is only a difference if it makes a difference” doesn’t mean
anything to me.

34. TIP: Bring the zazz!
Now this graph showing why people join record clubs has a little more zazz to it. If you make your
charts colorful and fun, you pull attention away from the actual data. That way, you can spin it
however you want!
[Graph courtesy of The Onion.]

35. TIP: Make the data appear meaningful even if it isn’t.
Prove your points through nifty diagrams that mean very little but sure look awesome. We’re
designers! We’re good at making something pretty.

36. TIP: More specific = More credible
$311,294,070,513
The more specific the number you quote is, the more people think it must be correct. Anyone know
what this number is? [Click] It’s the exact amount down to the dollar Jakob Nielsen said the whole
world would save by implementing his usability recommendations. I’m surprised he didn’t put cents
in there as well. Most businesses can’t get their budgets down to dollars, but here he does it for the
entire world! Just masterful. My hat is off to you, sir.

37. ☛TIP: Make the viewer feel good.
http://www.odannyboy.com ☚
This is from Jakob Nielsen again, from his article about how everyone should write essays, not blog
posts. Very insightful, a must-read. I found it on a blog. But he says the bloggers on the left are
bozos, and those on the right are clever. Wow, he must be reading MY blog. [CLICK]

38. TIP: Trend lines going up make for good drama.
If you can show everything is moving up, up, up you’ll make your clients feel better and you’ll
probably get more work out of it! It doesn’t matter if you can’t really prove it. It’s the future, so no
one knows what is going to happen!

39. 6. Willfully confuse correlation and cause.
Rule number 6. Just because something happens in a sequence or is connected doesn’t mean the
one thing causes the other. It might be a coincidence. But coincidences don’t make for good
research presentations, do they?

40. Pick the reason you like the most.
If there are several reasonable explanations for a phenomenon, pick the one that suits you the
most.

41. TIP: With a small sample, correlations are easy!
If there are several reasonable explanations for a phenomenon, pick the one that suits you the
most. With a small enough sample, you can make a correlation between just about anything. In this
case, more cars that are blue are hit by rocks every year.

42. 7. Deliberately misinterpret data.
As anyone who has analyzed research data knows, you can take the exact same data and draw from
it multiple conclusions. This is even more true with the qualitative research that designers most
often do. Look at this image, taken on a research trip. You could look at this sign and say, Wow, the
subway really cares about its passengers, giving them this helpful sign. You could also look at this
and say, Wow, only two trips from this platform. Why is the subway so poorly marked it needs this
sign to let us know this information??? It all depends on how you want to spin the data.

43. TIP: Make unlike things seem alike in comparisons.
Death rate in Chicago, 2006: 20 per 1000
Death rate in Iraq, 2006: 10 per 1000
Thus, Chicago is more dangerous than Iraq.
Now clearly, Chicago and Iraq aren’t quite the same, but by comparing apples to oranges, I can get
you to believe that they are AND skew the data.
Oh, And I made those figures up, so don’t bother checking them.

44. “I really love using this product.”
If you divorce words from their context, they are ripe for misinterpretation. There is a big difference
between this [CLICK] and this. But you can say, “Customers said they love using this product.”

45. “The“The subjects this task task nearly impossible.”
subjects said said this was was difficult.”
Language is great for distorting data. Note the subtle difference. Makes it much more impactful,
doesn’t it?

46. 8. Answer questions you weren’t asked.
When you go out into the field (if you bother to go) you are likely to bump into something interesting but
unrelated to the business problem. When that happens, simply change the scope of the research. If you
can’t find what you are looking for, change the scope and then find what you need.

47. TIP: Call them“Emergent Patterns.”
If you can, call it an quot;emergent
pattern.quot;

48. TIP: Don’t get caught.
Amateurs like James Frey get caught because their lies are too obvious and too easily disproved. Be
sure you follow my rules for distorting your research in your favor.

49. “It’s a subjective art!”
And if you do get caught, simply explain, Hey, it’s a subjective art! It’s a magic phrase that will get
out out of all kinds of trouble and it is impossible to argue about.

50. It’s better to be a winner than to tell the truth.
And that, ladies and gentlemen, is...

51. How to Lie
Thanks.
with Design Research
Dan Saffer
http://www.odannyboy.com
dan@odannyboy.com
Thanks to Flickr photographers for the images and to Darrell
Huff’s classic book How to Lie with Statistics for inspiration.
Special thanks to the staff of Adaptive Path and especially Todd Wilkens and Brandon Schauer for
their insights and contributions to this talk.