Over the past two years, the Puppet Labs UX team has run user tests en-masse at our annual conference in San Francisco. The first year, we ran almost 50 user tests, and made a lot of mistakes. This past year, we took those lessons, improved our testing process, and ran twice as many user tests, successfully. Additionally, we’ve recruited hundreds of users at these events for future user testing opportunities. As the person responsible for making this happen, I’d like to talk about how other UX teams can avoid our mistakes and replicate our successes. The latest version of this presentation is at http://slidesha.re/1qm8yj3

1. How to Run 100 User Tests in Two Days
Daniel Sauble, UX Designer at Puppet Labs

2. Hi

5. a suggestion as to a course of action
idea (n):
Julian Santacruz
juliansantacruz.com

6. Why do this?
Richard Winskill
flickr.com/photos/richard_winskill

7. P

8. P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P

9. S

10. P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
S

11. P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
S × 20

12. P P P P P P P P P P
P P P P P P P P P P
P P P P P P P P P P
P P P P P P P P P P
P P P P P P P P P P
2012
P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P
S × 6
2013
P P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P P
S × ?
2014
S × 5

13. Why a conference?

14. P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P

15. V

16. P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
V

17. P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
V × 16

18. Find Recruit Schedule Test

19. mistake (n):
here are some things to avoid
Dean Michael Dorman
flickr.com/photos/bestraft/

20. 1
Don’t write every study yourself

21. Time
Complexity

22. “I didn’t have time to write a short
letter, so I wrote a long one instead.”
!
- Mark Twain

23. Trivial

24. Complex

25. Simple

26. QAWriting

27. QAWriting Training

28. This will take 4x longer than you think

29. 2
Don’t pick a booth in the busiest hallway
Linda Tanner
flickr.com/photos/goingslo/

31. Keynote room
Hallway
Conference rooms
Stairs to
sponsor
booths
Us

32. 12 feet

35. 3
Don’t skip on-the-spot analysis

36. P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P

37. P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P

38. P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
P P P P P P P P P P P P P P P P P P P P
“
”

39. 4
Don’t ignore your product roadmap
David Clark
dcclark.net

40. Research value
Time

41. 6 months?

42. R

43. S R→

44. ←S R

45. S R

46. 6 mo

47. 6 mo
Ship before research

48. 6 mo
Ship after research expires

49. 6 mo
Ship window

50. 5
Don’t leave your laptops unattended
Radu
flickr.com/photos/negora/

51. Record

52. Backup

53. Secure

54. success (n):
here are some things to try
Marina del Castell
flickr.com/photos/marinadelcastell/

55. 1
Pick an event five months in advance
Ivan Serra
flickr.com/photos/ivanilluuu/

56. Where?

57. rocor
flickr.com/photos/rocor/

59. What?

60. Who?

61. 2
Consider the legal ramifications

62. NDAs

63. Do you really need one?

64. E-signatures

65. 3
Recruit volunteers from different teams
J. Ollas
flickr.com/photos/jokke_kuvat/

66. Roles

67. Finding people

68. 4
Define your process (i.e. guardrails)
Stew Dean
flickr.com/photos/stewdean/

69. Our process
the cliff-notes version

77. Crowd control

79. Entrance interview

83. Testing

85. Swag

87. Exit interview

90. 5
Train your volunteers
Centrum/Jim Levitt
flickr.com/photos/ptcentrum/

91. Schedule it

92. Share your documents

93. Repeat yourself

94. 6
Acquire equipment and materials
Stealing-Beauty Photography
flickr.com/photos/stealingbeauty/

95. Laptops

96. Microphones

97. Three-ring binders

98. Paper-based tests

99. 7
Do the thing
Philip R Jones
flickr.com/photos/pj_photos_/

100. Have fun

101. Talk to people

102. Don’t hover

103. What doesn’t kill you…

104. 8
Hold a retrospective
Paul Downey
flickr.com/photos/psd/

105. Stop

106. Keep

107. Try

108. conundrum (n):
something that is puzzling or confusing
Robert S. Donovan
flickr.com/photos/booleansplit/

109. 1
Scheduling is hard

110. 2
Data tends to evaporate
Jason Cross
flickr.com/photos/jasoncross/

111. 3
Quantitative tests?
Gavin Brogan
flickr.com/photos/heavyweightgeek/

112. 4
Software misbehaves

113. 5
Groupthink

114. QA
@djsauble

115. Resources
• docusign.com
• silverbackapp.com
• nngroup.com/topic/user-testing/
• links.puppetlabs.com/puppet-labs-test-pilots/