This webinar discusses on-site monitoring of standardized test administrations. On-site monitoring involves trained monitors visiting schools during tests to document test security practices, observe administrators and students, and ensure compliance with procedures. Monitoring supports test validity, provides quality assurance of the testing program, and complies with federal requirements. It was recommended that monitoring be both announced and unannounced. Observations are documented using checklists and any issues require follow-up. Monitoring data should be analyzed, compared across schools, and used to improve training, policies and future monitoring. The webinar outlined six steps for districts to establish an effective monitoring program.
3. Today’s Questions
• What is on-site monitoring?
• Why implement on-site monitoring?
• How do we get started with on-site
monitoring?
• What do we do with the information from
monitoring?
www.caveon.com3
5. What Is On-site Monitoring?
Trained monitors visit schools and testing rooms
during assessment administrations.
Monitors will:
Review and document school-level test security
compliance practices and procedures.
Observe test administrators, proctors and students.
Review all parts of the testing day.
Capture observations using pre-defined checklists for later
review.
www.caveon.com5
6. Why Have A Monitoring Program?
• Support the validity of assessment results.
• Provide quality assurance for your testing program.
– Gather information about implementation and efficacy of training
programs.
– Measure compliance with test administration processes and
protocols.
– Identify areas for improvement of process/policies.
• Comply with USED Peer Review Requirements.
www.caveon.com6
8. Should We Announce Monitoring?
• Announced
– Pro: Administrators are more likely to ensure
compliance with policies and procedures.
– Con: School personnel on “best behavior,” so not a
true indicator or what happens during assessments
that are not monitored.
www.caveon.com8
9. Should We Announce Monitoring?
• Unannounced
– Pro: Provides a true indicator of what happens
during testing and encourages increased
compliance at all times
– Con: Added stress for test administrators
www.caveon.com9
10. During On-Site Monitoring
Look at all parts of the testing day
–Training logs and proctor agreements.
–School-level secure testing materials management.
–Proctor and student login procedures.
–Actual testing sessions in testing rooms.
–Final sign-off/ secure testing materials management.
www.caveon.com10
11. During On-Site Monitoring
Focus on process and people
– Test administration process.
– School staff/proctor roles and
behaviors.
– Student behaviors.
– Testing environment
(classroom, computer lab).
www.caveon.com11
12. During On-Site Monitoring
• Use objective checklist or form
– Yes/no (binary) checklist items to eliminate
discretion of monitors and allow for data analytics
across schools.
– Require explicit narrative descriptions of any
observed potential irregularity or deviation from
policies.
– Clear procedures for data capture and management.
– Archiving and evidence preservation.
www.caveon.com12
13. During On-Site Monitoring
• Consider electronic tools for incident
management
– Real-time data capture.
– Data analytics across schools.
– Workflow tools for incident management.
– Archiving and evidence preservation.
www.caveon.com13
14. What to do after On-Site
Monitoring
www.caveon.com14
15. Use of Monitoring Data and Results
• Do irregularities or potential violations
observed during monitoring require follow up
investigation?
• What level of monitoring data and results will
be shared with schools and staff?
• Should public reports of monitoring be
released?
www.caveon.com15
16. Use of Monitoring Data and Results
• Evaluate with other test security data sources.
–Incident or irregularity reports.
–Data Forensics or other test results analysis.
• Compare monitoring data points across
schools.
• Evaluate monitoring data within overall test
security context.
www.caveon.com16
17. Take Action
• Provide feedback to staff at observed sites.
• Conduct follow up investigations if warranted.
• Select sites for future monitoring.
• Improve test materials, instructions, or training.
www.caveon.com17
19. Three questions to ask
• What are the goals of monitoring?
– Quality assurance.
– Ensuring compliance.
– Evaluating training effectiveness.
• What resources are available for monitoring?
• What will we do with the data from the
monitoring?
www.caveon.com19
21. Six Steps
1. Set purpose and goals.
2. Determine budget/funding source.
3. Select sites.
4. Evaluate staffing options.
5. Develop audit forms.
6. Set periodic program reviews.
www.caveon.com21
22. Other Important Things to Consider
• Communication plans to stakeholders.
• Post test surveys of stakeholders about testing
experience.
• Staffing.
www.caveon.com22
Hinweis der Redaktion
Richelle
Richelle
Steve
Monitoring classrooms is an important part of an overall test security program. This webinar will cover the resons for implementing a moitoring program, how to set up a program, and managing and acting on the information gained from monitoring.
Steve will hand off to Marc for story - LA DOE—issues in media, demonstrate commitment to fair & valid testing.
Steve
This slide defines what we mean by onsite monitoring. Need a brief story of what this actually looks like (e.g. MS using their literacy coaches as monitors with training and checklists developed by the state)
Steve
Need examples , maybe training?
Steve
Steve
Announced/Disadvantages of Announced and Unannounced
Announced Monitoring
Pros
Administrators likely to ensure compliance with policies and procedures
First Hand Knowledge (Marc, not sure what this means)
School Personnel on “Best Behavior”
Cons
School Personnel on “Best Behavior”
Unannounced Monitoring
Pros
Eliminates one-time behavior
Potential for increased compliance at all times
Cons
Cannot address observed issues immediately
Added stress for test administrators
Steve will hand off to Marc for story - MS no monitors on day one, monitors on day 2, reduced number of anomalies
Announced/Disadvantages of Announced and Unannounced
Announced Monitoring
Pros
Administrators likely to ensure compliance with policies and procedures
First Hand Knowledge (Marc, not sure what this means)
School Personnel on “Best Behavior”
Cons
School Personnel on “Best Behavior”
Unannounced Monitoring
Pros
Eliminates one-time behavior
Potential for increased compliance at all times
Cons
Cannot address observed issues immediately
Added stress for test administrators
Steve
Here is where we can emphasize the need to go into classrooms.
Steve
Steve will hand off to Marc for timing story…proctor had no watch
Steve will hand off to Marc for story - state & LEA providing exception to a standard policy regarding electronic devices
Shameless plug for Caveon Core
Steve
Steve
Steve
Steve
Steve
Need a success story of one monitoring program startup
Steve will hand off to Marc - additional goal—transparency and integrity of the accountability process.
Steve
Steve
The details of these steps will be in the one pager that we distribute after the call.