The webinar summarized key lessons from several summer testing conferences regarding test security. It discussed conferences focused on statistical detection of cheating, state assessment, and international testing. Top takeaways included a growing recognition of security importance, attention to standards, and using data to detect misbehavior. The webinar also summarized the National College Testing Association conference, which featured several sessions on detection, prevention, and responding to breaches. It concluded that awareness of security is high but cheating remains a challenge requiring innovative detection and prevention strategies.
2. “The Endless Security Summer”
Steve Addicott
• Vice President
• Caveon Test Security
John Fremer, Ph.D.
• President
• Caveon Consulting Services
Lou Woodruff, Ph.D.
• Senior Security Consultant, Caveon
• Former President, National College Testing Association
2
3. Agenda
• Conference on Statistical Detection of Potential
Test Fraud, May 23-24
• National Conference on Student
Assessment, June 26-29
• International Test Commission, July 3-5
• Dutch Exam Society, July 6
• National College Testing Association, August 1-4
• Closing Thoughts
3
5. Conference on Statistical Detection of
Potential Test Fraud
• University of Kansas’ CETE; Lawrence, KS
• First-ever Data Forensics Conference
• Gathering of measurement experts &
researchers
– Certification
– Education
– Vendors
– Test Publishers
– Academics
5
6. Conference on Statistical Detection of
Potential Test Fraud
• Most important takeaway
THIS CONFERENCE EXISTS!!
• Second most important take away
Protecting our tests grows ever harder, and
ever more important
6
7. National Conference on Student
Assessment (NCSA)
• June 26-29 in Minneapolis, MN
• Council of Chief State School Officers (CCSSO)
• 2012 Theme: Teaching, Learning, and
Assessment through the Next Decade
7
9. NCSA Key Takeaways
#3—Bombshell!
CCSSO is partnering with Center for
Assessment to “re-focus” the conference
– Reduced attendance?
– Budgets?
– Changing priorities?
9
10. NCSA Key Takeaways
#2—A number of dynamics in state assessments
impacting security
– Teacher Accountability
• Legislatures passing laws
• Using existing assessments to now grade teaching
performance
– Assessment Consortia
– Technology
• Classrooms
• Administration
• Learning
10 • Assessments
11. #1 NCSA Takeaway
• Despite all these changes,
THERE WAS NOT ONE SESSION FOCUSED
ON TEST SECURITY.
NOT ONE.
11
12. Topics That Could Have Covered…
• Thoughtful piloting of security strategies
• By 2014-2015, all state assessments will be CBT
– Colossal logistical challenge
– Too few PCs and test centers
• Blending individual test programs
– Dozens into two
– Exams will be shared across states and time zones
– Administration windows will last weeks
– More technology, more sharing?
12
13. International Test Commission
• July 3 – 5, 2012
• Group representing testing organizations and
national agencies world wide
• Amsterdam, The Netherlands
13
14. ITC – Major Takeaways
• Growing recognition throughout the world of the
importance of protecting IP and preventing
cheating
• Attention to standards – ATP, ITC, ANSI, Caveon
• More attention to how to use data to detect
misbehavior
• “Trust but verify”
14
15. CITO & Caveon Session
• CITO – very major provider of testing services
to Europe and many non-European countries
• Collaborating with other Dutch testing
agencies to promote security standards
• Caveon – evidence of importance of security
– Internationalization of testing
– Challenges of technologically savvy cheats
– Notorious US cases, especially Atlanta PS
15
16. Faking and Test Security
• Steve Addicott Contribution
• Dealing with Media
– Be prepared
– Draw on all parts of your organization –
communications, legal, membership, etc.
– Have a spokesman
– Tell the story from your perspective
– Don’t expect the story to go away
16
17. Dutch Exam Society (NVE) Security Summit
• NVE has 400+ members from dozens of test
programs in The Netherlands
• Eight leaders gathered to:
– Address Test Security concerns
– Consider adoption of Security Standards for
members
• Caveon invited to play a supporting role
17
18. Top NVE Security Summit Takeaways
#3—Times are changing
– Test Security was not, but is now, a front-burner
issue
#2—Security Risks are global
#1—Rising tide lifts all boats
– Organizations which are considered
“competitive” can collaborate and create positive
change
18
20. ncta-testing.org
National College Testing Association
11th Annual Conference
Minneapolis 2012
20
21. ncta-testing.org
College testing administrators and staff
representing over 600 colleges
More than 50 companies that do business
in the testing field
21
22. ncta-testing.org
Consortium of College Testing Centers
Listing of hundreds of college testing centers
offering proctoring services for individuals
needing to take a secure, proctored examination
Access the listing for free on the NCTA website
22
23. Six Sessions Devoted to Test Security:
The Keynote Address
The General Session
Four Breakout Sessions
23
24. Keynote Address
Mark Poole
Product Strategist and Security Expert
Pearson VUE
“The Future of Secure Testing in a Digital World”
24
25. Highlighted five key trends that are
shaping the next decade of testing:
o a global cheating culture
o high tech test fraud
o testing anywhere securely
o candidate authentication
o data forensics
25
26. Breakout Session:
“Demystifying Data Forensics: An Overview of the
Logic Underlying Cheating Detection Techniques”
Jim Wollack, Ph.D.
Director, Testing and Evaluation Services
University of Wisconsin, Madison
Presented the basic principles of applied data
forensics in a way that everyone could understand.
Reviewed some forensics approaches to identify
examinees who copy answers, had prior access to
test content, or hired surrogates to test for them
26
27. Breakout Session:
“Gimme Shelter: Weathering the Media Storm
of a Security Breach”
Steve Addicott
Vice President, Client Services
Caveon Test Security
Ray Nicosia
Executive Director of the Office of Testing Integrity
Educational Testing Services (ETS)
Faisel Alam
Test Security Specialist
Law School Admission Council (LSAC)
27
28. This panel of security experts shared key aspects of
media strategy for testing programs and techniques
for positioning incidents to various audiences
High profile security breaches were reviewed,
highlighting details of what went well and what did not
The use and effectiveness of various media tools was
analyzed, and a checklist of media “Do’s and Don'ts”
was presented
28
29. Breakout Session:
“Security Strategies for Every Stage of the Testing
Process”
DeDe Hedlund Jeff Place
Creighton University Questionmark
Looked at best practices for colleges to protect test
content and minimize cheating
Demonstrated the potential for using mobile test
centers with examinees testing on tablets such as
the Apple iPad
29
30. Breakout Session:
“Innovations in Online Proctoring”
Rick Beaudry Denis Gravel
CEO Director, Workforce Development
B Virtual Inc. School of Applied Technology
Presented a proctoring service that proctored test
takers in any location via web-cam and microphone,
with a proctor monitoring the session remotely
30
31. General Session
“Detection and Prevention of Test Fraud
The State of the Art”
John Fremer
President
Caveon Consulting Services
Lou Woodruff
Senior Security Consultant
Caveon Consulting Services
31
32. o Perspectives on the Perpetrators
o Test Security Anatomy
o Contemporary Test Security Responses
o The Technology of Detection
o Large Scale Security Investigations
o Future Test Security Challenges for Higher Ed
o Implications for College Testing Professionals
32
33. Survey data on the prevalence of cheating among the
nation’s youth was shared which showed an
alarming level of cheating
Cheating has grown from a strategy for increasing
test scores to the manifestation of a growing social
norm that accepts dishonesty in many areas of life
The cheating/detection arms race will continue with
advances in biometric, data forensic and test
delivery techniques enhancing prevention and
detection
33
34. Result of Using Data Forensics
in a State Assessment Program
Reduction in Data Forensic Analyses of
"Irregularities" school districts statewide
6.0%
5.0% Decisive actions taken on
the forensic results
4.0%
3.0% Two years of follow-up test
2.0% administration monitoring
1.0%
Found a 72% reduction in
0.0%
test security irregularities
Report 1
Report 2
Report 3
Report 4
Report 5
Report 6
Report 7
Report 8
Report 9
34
36. Mandated assessments
tied to federal funding
The Perfect
Schools and teachers Test Security
evaluated on test scores
Storm
Administrators feeling
unjustly treated
Test security left in the hands of
the teachers and the principals
36
37. Atlanta Public Schools Test Security Case
Governor brought in the GA Bureau of Investigation
Sixty-one agents on the case with subpoena power
District Superintendent, all Deputy Superintendents,
and many Principals and Teachers were removed
Who do you trust? Sometimes – no one!
37
38. Take-Aways from the NCTA Conference
College testing professionals are highly aware of
and strongly interested in test security
NCTA is a valuable platform for promoting security
related initiatives and partnerships between
colleges the testing industry
Colleges are and will remain a rich resource for
implementing cheating detection and prevention
strategies
38
40. Today’s Presenters
Steve Addicott
steve.addicott@caveon.com
John Fremer
john.fremer@caveon.com
Lou Woodruff
LCWoodruff@aol.com
40
Hinweis der Redaktion
lou
lou
lou
Polls:-Has there been a larger focus on security at your org. in last 5 years? before lou after Steve. -Did you attend any of these? (Select all that apply) right after agenda.Note: John’s last slide last before Q&A.Steve to add anima. And tweaks.Skyler to compile and add polls.At end of each speaker, ask questions if any come up.