1. TESTING TECHNOLOGY PRODUCTS
Websites, Apps, Software etc…
March 2013

2. OVERVIEW
METHODOLOGIES
- EXPLORATORY TESTING
- USABILITY TESTING
- “REAL-WORLD” TRIALS
CONCLUSION

3. Reasons to Test Tech Products
 Ensure efficient development of useful and
appealing products
– Begin with a well-conceived concept
– Add features and functions systematically
– Conduct periodic “reality checks” with users
– Avoid “spider web” syndrome with changes
– Increase confidence in reliability, relevance and
appeal of product as release approaches
– Be prepared for an effective roll-out: sales story,
3 user training, competitive awareness etc…

4. Reasons for Not Testing Tech
4

5. Reasons for Not Testing Tech
 Lack of time
 Lack of money
 Overconfidence
– Unaware of potential problems
– Disconnected from users true needs, attitudes or
skill level
 Sometimes not necessary
– minor changes, simple product, previously well
researched, based on something similar, etc…
5

6. Types of Tech Product Testing
 EXPLORATORY:
– What digital content or functionality do users need
and/or desire?
 USABILITY:
– How easy or hard is it to use the digital product?
 “REAL-WORLD” TRIALS:
– How well does the digital product actually fulfill
the users needs?
6

7. OVERVIEW
METHODOLOGIES
- EXPLORATORY TESTING
- USABILITY TESTING
- “REAL-WORLD” TRIALS
CONCLUSION
7

8. Methodologies
Exploratory Testing
 Usually groups
– Live or Online groups
– Online Bulletin Board
 Typical flow of discussion:
– Current technology practices
– Speculate on future needs
– Response to technology product ideas & options
 Concept statements
 Wireframes or other mock-ups of ideas
8  Existing websites or applications

9. OVERVIEW
METHODOLOGIES
- EXPLORATORY TESTING
- USABILITY TESTING
- “REAL-WORLD” TRIALS
CONCLUSION
9

10. Methodologies
Usability Testing
 Most complex but not more important
 Common practices:
– Multiple iterations
– Each consisting of 8-12 In-depth interviews
 Depending on types users
– 45-60-minute sessions
– Conducted live in the early phases
 Computer lab provides stable digital environment
1 – Conducted online in later stages
0  Real-world digital environment - varied OS and browsers

11. Methodologies
Usability Testing
 Four “phases” of usability testing
– Wireframe
– Alpha
– Beta
– Pre-release or Post-release
 Dependent on level of completion and
functioning of test digital product
 Phases may not be clearly defined
1
1  Nice but not necessary to test each phase

12. Methodologies
Usability – Wireframe Testing
 Participants navigate through hyperlinked
screen shots with only minimal content and
sketchy design elements
 Demonstrates the degree to which the
website is organized and functions logically
 Best conducted live, in professional market
research facilities or computer lab
1
2

13. Methodologies
Usability – Alpha Testing
 Product functions roughly and is populated
with just enough data for participants to
complete the most essential tasks
 Design is evident but unrefined
 Evaluate how well the key navigational and
basic design elements align with users
expectations
 Best conducted live, in professional market
1
research facilities or computer lab
3

14. Methodologies
Usability – Beta Testing
 Product is reasonably stable and
approaching full functionality with at least
one complete segment of content
 Gauges extent to which navigation, design
and other conventions will work when
product is complete
 Good time to test “optional” features
 This phase of research is typically conducted
1
online to experience “real-world” conditions
4

15. Methodologies
Usability – Pre or Post Release
 Product is fully functional and content is
complete
 Simulates “real world” use to isolate bugs,
impact of combined actions, identify minor
refinements to improve user experience,
uncover training needs
 Usually conducted online
– Tasks are more open-ended to see how new
1 users learn and problem-solve
5

16. Methodologies
Usability
1
6

17. Methodologies
Usability – When Are You Done?
 Digital products are tools…
– Should fulfill the user’s needs in an efficient and
simple manner that is often “unremarkable”
– “The absence of feedback is feedback”
– High praise = “It worked just as I expected”
 Ready for release when… almost all test
participants successfully complete the
mission critical tasks with minimal problems
1
7

18. OVERVIEW
METHODOLOGIES
- EXPLORATORY TESTING
- USABILITY TESTING
- “REAL-WORLD” TRIALS
CONCLUSION
18

19. Methodologies
"Real-World" Trials
 Multiple phase process
– Individual trial and reporting of experience
 Online Journal
– Can use video, audio or text (smartphone saves time)
– Group discussion
 Live or Online or through Online Bulletin Board
 Options for depth and duration
– Task specific – 2-3 weeks
1 – Short-term panel – 1-2 months
9 – Long-term panel – 3 or more months

20. Methodologies
"Real-World" Trials
 Options for testing
– Individual product
– Comparison of competing products
 Options for participants (Consider comparisons)
– First users of new product
– New users of established product
– Veteran users of established product
– Pre-trained users
2 – Self-trained users
0
– Users of competitive products

21. Methodologies
Trials vs. Usability Testing
"Real-World" Trials Usability Testing
Documented by posts of users Directly observed in real-time during
recollections to an asynchronous one-on-one interviews
online bulletin board
Macro view Micro view
- Tasks are broad “create a lesson” -Tasks are more specific “find a
- Feedback is non-specific lesson about cell division”
- Feedback is specific
Exposes common problems with Exposes all problems and provides
only vague or speculative insight precise insight into users
into users expectations expectations
Helps to determine general Specifically identifies the problems
2 functions that are problematic related to each tested function
1

22. Methodologies
Learning Curve vs Usability Issues
Learning Curve Issues Usability Issues
Challenge is learning the Challenging to learn and to repeat
convention or action
Once learned – repeated Users don’t learn – process is
completion of task is easy flawed
Improved with tweaks to labeling, Requires redesign of process.
design, button position etc… Sometimes creates a “ripple effect”
Also, training, tutorials etc… that impacts other parts of platform.
Can be identified in both usability Requires usability testing.
and "Real-World" Trials Trial testing feedback is generally
not specific enough to provide
insight into these issues
2
2

23. Methodologies
"Real-World" Trials (related)
 Corporate “Online Presence” Testing
– Harvest available data
 Online comments
 Customer communications
 Technical support data
 Website metrics
– Enhance and deepen insights with qualitative
 Live or online groups – primed with pre-group task
 Online Bulletin Board – users respond to web presence
2 of various companies and then “discuss”
3

24. OVERVIEW
METHODOLOGIES
- EXPLORATORY TESTING
- USABILITY TESTING
- “REAL-WORLD” TRIALS
CONCLUSION
24

25. About Dave Kreimer &
Next Step Consulting
 Next Step Consulting consists of a team of professionals
supporting the efforts of Dave Kreimer, the company's
principal.
 Dave Kreimer's background in psychology, and years of
facilitation and marketing experience enhance his market
research skills. Since 1993, Dave has successfully designed,
conducted, and documented roughly 500 research projects
involving thousands of focus groups and personal interviews.
The company’s most loyal clients include medical,
educational, publishing, technology, and finance firms.
2
5

26. Contact Us
 eMail: 2dave@nextstepconsult.com
 Phone: (206) 782-6002
 NSC Website
 NSC Blog
 LinkedIn
 Facebook
 Twitter
2
6

27. Graphic- Thank You
2
7