Nudges are small, deliberate tactics we can use to help our test-takers (and our SMEs) do the things they want to do. Our testing programs have many points that can derail candidates, but through small and subtle changes we can help them persist through the life cycle of application, testing (and retesting), and ongoing certification. Nudges can also be used with SMEs to increase JTA survey response rates and committee volunteer numbers. Join us for a panel discussion with researchers and practitioners using nudges in testing.

1. Using Nudges for
More Effective Exam Programs
Cynthia G. Parshall, PhD, Touchstone Consulting
Ellen Julian, PhD, Julian Consulting
Dot Horber, PhD, NBOME
Sarita Parikh, GED Testing Service
Presented at the annual meeting of ATP, Orlando, FL, March 18-20, 2019.

2. What do we mean by nudges?
▪ And what is behavioral design?

3. Behavioral design
▪ Nudges are small changes to an environment that can
have a powerful effect in influencing the choices
someone makes.
▪ Behavioral design is about understanding how people
think
▪ Then, identify the friction points that make bad decisions
more likely
▪ Then, apply nudges to make good decisions more likely

4. Agenda
Examples and ideas
How to get started
Case study #1
Case study #2
Group discussion

5. Examples from Other Fields
and Ideas for Us
Ellen Julian, PhD, Julian Consulting

6. If you wanted people to drink more water…
1/29/2019 6
At Google kitchen, they moved the water bottles
to eye level.
Would that affect your choice?
Water selection increased by almost 50%!
Technique: Make it Hard to Miss

7. How can we use Make it Hard to Miss?
▪ Add Practice Test promotions
to exam-registration process
▪ Put a Volunteer Now! button
on the annual renewal page

8. If you wanted hotel guests to reuse towels…
A research study compared messages:
• emphasizing the environment
• emphasizing what other hotel guests do
What results would you predict?
Technique: Social Proof
Those getting the ‘other hotel guests’ sign used >25% fewer towels!

9. How can we use Social Proof?
▪ 90%* of other registrants have already renewed
▪ 95%* of successful candidates studied for the test
▪ Most of those who volunteered to write items have already
submitted theirs.
Avoid Negative Proof: Don’t talk about the cheaters; talk about
the honest majority!
*Only use real numbers

10. If you want more organ donations?
Start with the donation box checked ‘Yes,’ allowing them to
opt out.
Technique: Defaults
More than 90% donate when the default is Yes.
Only 15% donate when the default is No.

11. Or you want more people to save?
Changing 401k enrollment from opt-in to opt-out
increased the number of new employees enrolled…
from 34% to 90%!

12. How can we use Defaults?
▪ Sign up new registrants for reminders, texts, newsletters…
and let them opt out.
▪ Assume registrants will want to renew and just send them
an invoice, rather than making them remember, go online,
and hunt down “how to renew.”
▪ Add a donation to your educational foundation to their
renewal checkout page.

13. Nudge-fest for Failers
This isn’t the end! [framing] You’ve worked hard and overcome
obstacles to get this far [identity]; one more try may be all it takes.
Register now for your next attempt and be done with that decision.
[reduce friction]
Many people who became experts in the field had to take the exam
more than once [social proof], but they didn't give up. Once you
pass, it doesn't matter how many tries it took - you'll be a
{professional}! [amplify benefits] Don’t allow one bad day to
change your life and career [loss aversion]. We look forward to
seeing you on your next test date!
13

14. What would you like to impact?
Group Activity:
On the back of your business card,
write a chronic frustration
with examinees, volunteers, or staff
where a nudge might be able to help.
We’ll collect and crowdsource some ideas!

15. Behavioral Design –
What It Is & How To Get Started
Cynthia G. Parshall, PhD, Touchstone Consulting

16. Think of your program’s biggest pain point…
▪ Does it relate to human behavior?
▪ If so, a nudge can probably help!

17. Why it matters
“The goal is to help people do things that they have struggled
with in the past.”
Steve Wendel. Designing for Behavior Change, 2014.
“Small interventions in the environment … can encourage
people to make better decisions.”
Financial Times, 2017.
“By knowing how people think, we can make it easier for them
to choose what is best for them, their families, and society.”
Thaler & Sunstein. Nudge: Improving Decisions about Health, Wealth, and Happiness, 2008.

18. Some of the recent research
There have been
research findings from
psychology, usability,
behavioral economics,
gaming, and
other fields.
18

19. A few behavioral design tools & examples (1)
▪ Framing
▪ ex: By volunteering, you can help determine what content
the test will cover.
▪ Identity
▪ ex: I’m the sort of person who helps my friend study –
but not the sort who helps them cheat.
▪ Reduce friction
▪ simplify text; reduce the number of clicks;
minimize choices

20. A few behavioral design tools & examples (2)
▪ Amplify benefits
▪ improve aesthetics; reward small wins;
give positive reinforcement
▪ Social proof
▪ ex: Many invited participants have already completed
the survey.
▪ Loss aversion
▪ ex: Renew now and avoid extra fees.

21. A behavioral design research process
▪ For best results, take these steps

22. How to research your nudge – 3 steps
1. Diagnose
2. Experiment
3. Learn
For more information about conducting behavioral design
research, see Parshall & Johnson, 2018 at
http://www.proftesting.com/blog/2018/09/19/
developing-nudges-exam-program/

23. How to research your nudge – Diagnose
1. Diagnose:
▪ Select the key behavior you want to change
▪ Think “chronic frustration” here
▪ Conduct a behavioral audit
▪ Find all the friction points
▪ Craft your nudge

24. How to research your nudge – Experiment
2. Experiment:
▪ Could be a qualitative usability study
▪ Or a quantitative A/B study

25. How to research your nudge – Learn
3. Learn:
▪ Need to iterate?
▪ Or ready to implement?

26. Getting Started with Nudges
Dot Horber, PhD
National Board of Osteopathic Medical Examiners

27. NBOME Mission
The NBOME’s mission is
to protect the public by
providing the means to
assess competencies for
osteopathic medicine
and related health care
professions.
27

28. Down with No-Shows!
▪ Problem:
▪ An increasing number of candidates “No Shows” for the
COMLEX-USA Level 2-PE.
▪ Impact of the Problem:
▪ Inconvenience and rescheduling fee for the candidate, delayed
testing
▪ Intervention:
▪ Email confirmation 7 days in advance of test date.
▪ Result:
▪ Decrease in number of “No Shows” from 17 (5001 tested) in
2012 to 10 (5623 tested) in 2013.
2/19/2019 28

29. Down with No-Shows!
▪ Problem:
▪ Following the decrease in No-Shows, the No-Show rate began
to increase again the following year.
▪ Impact of the Problem:
▪ Inconvenience and rescheduling fee for the candidate, delayed
testing
▪ Intervention:
▪ Research: Reminder emails were reinstated after having been
accidentally omitted.
▪ Result:
▪ Decrease in number of No Shows: 0 in next 3 months.
2/19/2019 29

30. Can we Make Reminders Better?
▪ Include reminders to help prepare for the examination
▪ Orientation Guide
▪ Orientation Video
▪ Google Maps
2/19/2019 30

31. Impact of Social Media
▪ Problem:
▪ Notification of score release occurred via email.
▪ Students preferred instant notification on social media.
▪ Intervention:
▪ Beginning 11/16, COMLEX L2-CE score releases were
announced on Facebook and Twitter.
▪ In 8/17, all COMLEX-USA score releases were announced on
social media posts.
▪ Impact:
▪ Range of reactions based on their satisfaction with scores.
▪ Some express joy on social media
▪ Some need support from our Client Services team.
31

32. Can We Make Score Notification Better?
2/19/2019 32

33. Student Comments on Social Media
There isn’t an error right?
2/19/2019 33
91st percentile on my
COMLEX-CE! YASSSSSSS
Thanks for the response!
Now, can you please make sure I’ve passed when the results are released?
Hi there, can the interns/PGY-1s get our Level 3 results today? Pretty please!
Thanks.

34. Can We Enhance the Examination??
2/19/2019 34
▪ Problem:
▪ Prior to 2017, normal lab values were present in separate
exhibits.
▪ Intervention:
▪ In 2017, lab values were embedded in exam content.
▪ Impact:
▪ More realistic presentation for candidates.
▪ No time required to navigate exhibit of normal ranges.

35. Can We Enhance the Examination?
2/19/2019 35

36. Thoughts to share...
2/19/2019 36
▪ Get started!
▪ Collect data
▪ Review and improve
▪ Collect data
▪ Meet your test-takers where they are!

37. Nudges in Practice:
GED Assessment
Sarita Parikh, GED Testing Service

38. Everything you need...
38
● Recruiting
● Retention
● Culture & PR
● Everything you need!
● Free!
● Mission and Business

39. 39
Desire
&
Capability
The Means
to Do it
?!+ =

40. A few behavioral design tools…
▪ Framing
▪ Identity
▪ Reduce friction
▪ Amplify benefits
▪ Social proof
▪ Loss aversion

41. Removing Friction
41

42. 42

43. Framing
Study 3 hours a week
Study 20 minutes a day
43

44. Identity
44
Identity
Now you are
a GED
student!
Sarita Parikh

45. Loss Aversion
45
As long as you
are an active
student...
Sarita Parikh

46. Novelty and Rewards
46

47. Social Proof
47

48. Results
48
300%
Student persistence
300→ 3600
GEDWorks graduates
+5%
User conversions

49. Activity --
The Group’s Issues
Crowd-sourcing nudges
for the most common causes
of chronic frustration

50. Sources of chronic frustration
50
Group’s list here…

51. What are your take-aways from this session?
And now...

52. Thank you!
▪ For more information –
▪ See the series starting with
“Action Design for Instruction and Assessment”
at FromtheItemBank.com
(or via https://lnkd.in/gJpHhPT )
▪ Cynthia G. Parshall, PhD
▪ CParshall@Touchstone.Consulting
▪ www.linkedin.com/in/cynthia-parshall