Discover security lessons learned from the program sessions and face-to-face meetings with state assessment and national organization staff at the June 2013 National Conference on Student Assessment in National Harbor, Maryland. Caveon Leaders John Fremer and Steve Addicott summarize the test security ideas and strategies that they drew from the conference. Among the points covered are the recommendations included in the new TILSA Test Security Guidebook which was released at the conference.

1. UPCOMING CAVEON EVENTS
• EATP – European Association of
Test Publishers - September 25-27,
St. Julian’s, Malta.
• ICE - Institute for Credentialing
Excellence - November 11-14,
Amelia Island, FL
• ATP 2014 - March 2-5, 2014,
Scottsdale, AZ

2. CAVEON ONLINE
• Caveon Security Insights Blog
– http://www.caveon.com/blog/
• Twitter
– @Caveon
– Follow us!
• LinkedIn
– Caveon Company Page
– Caveon Test Security Group
• Please contribute!
• Facebook
– Will you be our ―friend?‖
– ―Like‖ us!
www.caveon.com

3. "Preventing and Detecting Cheating: Lessons from
the CCSSO National Conference on Student
Assessment"
Steve Addicott Dr. John Fremer
Vice President President
Caveon Test Security Caveon Consulting Services
July 17, 2013
Caveon Webinar Series:

4. AGENDA FOR TODAY
• State Assessment Directors Meeting
• CCSSO TILSA Guidebook
• Greg Cizek Recommendations
• ―The Critical Role of Policy in Test Security‖
• CCSSO/ATP Operational Best Practices
• Ongoing Challenges for School Districts
• Other events
– ITC Security Standards
– 2nd Annual CSDPTF

5. STATE ASSESSMENT DIRECTORS
• State Assessment Directors
Meeting
• ―Hallway‖ Conversations
• ―Plan B‖ Dominant Theme
– Virtually all Consortium States have one
– One year extension of current pattern very
common alternative
– Some states have not done much fleshing out an
alternative
– Security and test difficulty could be major
problems

6. CCSSO TILSA GUIDEBOOK SESSION
• Context
– Charlene Tucker – TILSA Perspective
– John Olson – Project Leader’s Overview
• State Perspectives
– Kentucky
– West Virginia
• Key Recommendations in Guidebook
• Wonderful Greg Cizek Discussant
Comments

7. KENTUCKY TEST SECURITY –
FIRST STEPS
1. Commitment From Senior Leadership
2. Professional Audit

8. KENTUCKY: POLICIES & PRACTICE -
PREVENTION
• Security Plan
o Findings: The Department of Education does
not maintain a single, comprehensive security
plan/manual
o Action Tasks:
• Developing a comprehensive security
plan/manual
• Providing security plan/manual to all key
staff and portions to outside contributors as
appropriate

9. KENTUCKY: POLICIES & PRACTICE -
DETECTION
Data Forensics Analysis
o Action Tasks:
• Adding additional data forensics analysis
• Erasure Analysis
• Similarity Analysis
• Score Growth and Losses

10. WEST VIRGINIA PERSPECTIVE –
AN IMPORTANT ADDITION
• A compilation of ―Best Practices‖
• 3 Key Areas
– Prevention
– Detection
– Investigation
• Key Implications:
– Potential source to
conduct a gap analysis
– Preparation for any test security
requirements

11. WEST VIRGINIA - STATE AWARENESS
TIMELINE
• Test security has always been of interest to
SEAs
• Security breaches have thrust assessment
issues into the spotlight
• SEAs, LEAs, and schools now have a
dedicated interest in test security and
assessment integrity

12. WEST VIRGINIA - A STATE’S
POTENTIAL CONTEXT
• The 3 Ps of Assessment Integrity
– Policy
– Politics
– Practical Implications (i.e., reliability of
results)
• What is the driver?
– Timing + Severity + Pervasiveness +
Climate =
– Crapshoot

13. WEST VIRGINIA - WHERE ARE
WE NOW?
• Waiting for the next AJC article
• (Quality?) Information everywhere about test
security practices, audits, and requirements
• The guidebook providing a comprehensive look at
the three major categories
• Taking recommendations to create actionable
information that practitioners can use to revise
practice and policy

14. OVERVIEW OF FREMER PRESENTATION
Recommendations (Sample Presented)
Prevention
Detection
Test Security Investigations

15. PREVENTION
• Communicate Zero Tolerance
1. Clear
2. Consistent
3. Frequently emphasized messages
• Develop a Security Plan
– Comprehensive
– Up to date

16. DETECTION
• Use Multiple Data Forensics Methods
1. Unusual Gains
2. Similarity
3. Erasures (Answer Changing)
4. Person Fit Analyses
• Introduce Data Forensics Very Carefully
– Think Through Planned Uses
– Prepare LEAs (and others) for Data
Forensics Use
– Planful, Multi-step Public Information
Program

17. TEST SECURITY INVESTIGATIONS
• Areas Needing Attention
– Level of evidence warranting an
investigation
– Roles for state, district, and other staff
– Timelines
– Model investigations kit

18. TEST SECURITY INVESTIGATIONS (cont)
• Model Investigations Kit
– Clear definition of responsibilities
– Procedures for evaluating evidence
– Planning and conducting interviews
– Developing a report with
recommendations

19. TEST SECURITY INVESTIGATIONS (cont.)
– Focus Primarily on Improving Security in
the Future
• Follow ―Innocent Until Proven Guilty‖
Approach, but be Firm in your Questioning
• Give Most Attention to the ―Worst of the
Worst‖
• Consider Other Possible Explanations for
Results

20. GREG CIZEK RECOMMENDATIONS
A SAMPLE
• More Cheating than You Think!
– “Anyone who believes that his or her
program doesn’t have a serious cheating
problem has simply not investigated the
concern deeply enough.”
• Don’t do Cheating Statistical Analyses until
you have a Plan for Using the Results
• Focus on Score Validity. Look to Skilled
Investigators for Help if Needed

21. SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
Best thing —This session was on program!
• Presentations from Mississippi, Florida, and
Maryland DOEs
• 3 different perspectives on
security policies

22. SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
Mississippi Department of Education (MDE)
• James Mason & Walt Drane
• Shared three incidents
• Important lessons learned
– No going back!
– Have multiple enforcement options
– Consistent use of data and policies
– Play to win
– Plan for the future

23. SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
Florida Department of Education
• Formal Data Forensics Program in Place
– Student level and school level
– Analyses drive invalidations (students) and
investigations (schools)
– Appeals process implemented
– ―Measure to Manage‖
– Make resources available to all stakeholders

24. SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
Maryland State Department of Education
• Dr. Tamara Lewis, State Test Security Officer
• State Test Administration & Security
Committee (STASC)

25. SESSION: THE CRITICAL ROLE OF POLICY
IN MAINTAINING TEST SECURITY
MSDE Mission ―POSSIBLE‖
• Policy on test security
• Opportunities for training
• School security plans
• Standards for detecting infractions
• Investigation of suspected infractions
• Being prompt in reporting infractions
• Levying appropriate action/sanction
• Examining areas for improvement

26. SESSION: CCSSO/ATP OPERATIONAL
BEST PRACTICES UPDATE
 Christopher Hanczrik – State of WA, Office of
Superintendent of Public Instruction (Co-Chair)
 Pat Roschewski – Data Recognition Corporation
 Sarah Susbury – Virginia Department of
Education
 Lisa R. Ehrlich – Measured Progress (Co-Chair)

27. SESSION: CCSSO/ATP OPERATIONAL
BEST PRACTICES UPDATE
Chapter 8: Test Security
Define and design a comprehensive security plan
 Internal processes
 Computer systems
 Electronic transfers
 Test administration
BEST ADVICE
 Develop the Plan
 Implement it!!

28. SCHOOL DISTRICT CHALLENGES
Non-session oriented takeaways
• School District leaders are in a crucible
– Tons of Exams
– Ever greater stakes
• Incumbent on us to provide the tools
they need to measure and manage
security risks
• Provide the right resources to
investigate

29. OTHER SIGNIFICANT EVENTS
International Test Commission (ITC)
• Publishing ―Best Practices‖
• Adapted from Caveon Security Audit
Conference on Statistical Detection of
Potential Test Fraud
• October 17-19, 2013
• University of Wisconsin at Madison
• Program: http://bit.ly/1b6jM42
• Registration: http://bit.ly/18b1fzM

30. RESOURCES
• CCSSO/ATP – Operational Best Practices
• ATP Security Committee
• Caveon Test Security
– Blog
– Webinars
• NCME Guidelines
– http://ncme.org/default/assets/File/Committee%20Docs/Test%20Score%
20Integrity/Test%20Integrity-
NCME%20Endorsed%20(2012%20FINAL).pdf
• TILSA Guidebook for State Assessment Directors on
Data Forensics
– Buy the Guidebook at https://www.createspace.com/4302978
– Print for free at
http://www.ccsso.org/Resources/Publications/TILSA_Test_Security
_Guidebook.html
– Read Greg Cizek’s observations – http://www.caveon.com/tilsa-
test-security-guidebook-next-steps/

31. Caveon’s 10 Year Anniversary Video,
Take a look and celebrate with us!
http://lnkd.in/XFBKR7

32. Thank you!
Steve Addicott Dr. John Fremer
Vice President President
Caveon Test Security Caveon Consulting Services
Steve.addicott@caveon.com john.fremer@caveon.com
- Follow Caveon on twitter @caveon and
- (Fremer @TestSecurityGuy)
- Check out our blog…www.caveon.com/blog/
- LinkedIn Group – Caveon Test Security