Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
Â
Feasibility of Using RATE with Teacher Candidates
1. 2016
ANN BULLOCK, KRISTEN CUTHRELL,
ELIZABETH FOGARTY, JOY STAPLETON
EAST CAROLINA UNIVERSITY
C O L L E G E O F E D U C A T I O N
D E P A R T M E N T O F E L E M E N T A R Y E D U C A T I O N A N D
M I D D L E G R A D E S E D U C A T I O N
Feasibility of Using RATE with
Teacher Candidates
3. Context
ï Importance of assessment system with multiple
measures of effectiveness
ïĄ Our home-grown assessments found to be invalid or
unreliable measures of performance (Henry et al., 2013)
ï·Portfolio assessment
ï·Dispositions
ï·Observation Instrument
4. Rapid Assessment of Teacher Effectiveness
ïĄ RATE is a validated observation instrument in use
with practicing teachers
ïĄ Draws on existing teacher effectiveness
constructs
ïĄ Observe classroom lesson videos; Results in less
expensive teacher observation
ïĄ Focuses on direct instruction
ïĄ RATE design has 6 items, each item is assessed on a 3
point scale (Gargani and Strong,2014)
6. Research Questions
RQ 1: Does the RATE observation instrumentâs
reliability and validity hold when used with pre-
service teachers?
RQ2: Is the RATE observation instrument a feasible
instrument to be used by large teacher education
programs?
RQ3: Do you have to use two raters in order for the
rater scores to remain reliable and valid?
7. Procedures - Videos
ï Preservice Teachers:
ïĄ ELEM candidates submit full lesson videos to an electronic
portfolio system during final semester of internship
ïĄ identify their subject, topic, and 20 consecutive minutes of
direct instruction to be viewed
ï 179 videos were used for scoring in this study
8. Procedures- Training
ï March 2015
ïĄ Strong and Gargani held an orientation session with 38
faculty, university supervisors, instructional coaches and COE
administrators
ïĄ Focus was on introducing faculty to the RATE instrument
ï May 2015
ïĄ Day 1: 38 participants received training on the RATE
instrument
ïĄ Day 2: participants scored videos
9. Procedures - Scoring
ï 8 30 minute sessions were held on Day 2
ïĄ Within each 30 minute session a participant would score the
20 minute video individually then meet with the other person
who scored the video.
ïĄ Scores were compared and reconciled
ï 150/179 videos were scored twice using both rater
pairs and solo raters
ï At a later date- RATE personnel evaluated all videos
for comparison purposes.
13. Conclusions
ï RQ 1: Discrepancies in paired scores exist but rater
reliability was higher than in initial RATE validation study
ïĄ Biggest discrepancy: clarity and flow
ï RQ 2: Instrument is shorter and less complex than
existing instrument
ïĄ However, having two individuals rate one video seems
prohibitive in a large program.
ï RQ 3: To maintain reliability and validity two raters are
needed.
14. Next Steps
ï Pilot RATE instrument with preservice teachers in
live observations (spring 2016)
ï Determine predictive nature of RATE with
preservice teacher observation
ïĄ Explore relationship between RATE and CLASS instruments
ïĄ Explore relationship and possible predictive nature of RATE
with edTPA
15. References
Cuthrell, K., Stapleton, J., Bullock, A., Lys, D., Smith, J., and Fogarty, E. (2014). Mapping the
journey of reform and assessment for an elementary education teacher preparation program. Journal of
Curriculum and Instruction, 8(1).
Gargani, J., & Strong, M. A. (2014). Can we identify a successful teacher better, faster, and
cheaper? Evidence for innovating teacher observation systems. Journal of Teacher Education. Pre-print
copies available online.
Good, T., & Lavigne, A. (2015). Response to "Rating Teachers Cheaper, Faster, and Better: Not
So Fast": It's about evidence. Journal of Teacher Education, 0: 0022487115587110v1-
22487115587110
Henry, G. T., Campbell, S. L., Thompson, C. L., Patriarca, L. A., Luterbach, K .J., Lys, D. B., &
Covington, V. M. (2013). The predictive validity of measures of teacher candidate programs and
performance: Toward an evidence-based approach to teacher preparation. Journal of Teacher
Education 64, 439-453.
Sawchuk, S. (2011). University of Michigan project scales up "high leverage" teaching practices.
Education Week. Retrieved from the www on August 31, 2012 at
http://blogs.edweek.org/edweek/teacherbeat/2011/10/high_leverage_teaching_practices.html
Strong, M., Gargani, J., & HacifazlioÄlu, Ă. (2011). Do we know a successful teacher when we
see one? Experiments in the identification of effective teacher. Journal of Teacher Education, 62(4),
367-382.
Strong, M. (2011). The Highly Qualified Teacher: What is Teacher Quality and How Do We
Measure It? New York: Teachers College Press.
COE Innovations were borne out of a 9 million dollar TQP grant and allowed for the evaluation partnership with EPIC mentioned earlier. The goals of the
The lack of ability to make reliable assumptions about our candidates based on available data drove both program leadership and faculty to look for a more reliable method for assessing the readiness of teacher candidates
edTPA, Dispositions form that is enacted 3 times during the program, by multiple evaluators
Follow up conversations with university supervisors revealed that they were reluctant to score a teacher candidate down in an area on the instrument because they were trying to be supportive, thus not using the observation instrument as an evaluation tool but rather a developmental tool with teacher candidates. The use of the observation instrument for different purposes led to grade documentation issues with teacher candidates. For example, many times at the end of the semester the âscoresâ on the observation instrument would not show any deficits but the supervisor would want to give the teacher candidate a lower grade. Without evidence of reduced performance from the observation instrument, giving the intern a lower grade was not always possible. This data-driven knowledge led us to investigate other reliable and valid observation instruments. Â
RATE design consists of six items relating to the lesson objective, instructional delivery mechanisms, teacher questioning strategies, clarity of presentation of concepts, time on task, and level of student understanding. Each item is assessed on a 3-point scale, and evaluators are encouraged to jot down notes to support their scores.
Widely accepted frameworks include Danielson (2011) Framework for Teaching; Teaching Works Framework (Sawchuck, 2011), the Gates Foundation Measures of Effective Teaching (MET) project (Danielson, 2010); and Marzanoâs Classroom Instruction that Works (Marzano, Pickering, and Pollock, 2009).
In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
As a program requirement, teacher candidates submitted a final teaching video at the end of their internship semester. Full lesson videos were submitted to Taskstream, an electronic portfolio platform. Upon submission, candidates were asked to identify their subject, topic, and 20 consecutive minutes of direct instruction captured in the full lesson video. These videos were the subject of this research project.
In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
In March 2015, Strong and Gargani provided an orientation session on the RATE instrument for 38 ECU faculty (across all content areas), university supervisors, school district instructional coaches, and college of education administration faculty. This session focused on introducing faculty to the RATE instrument and how it may be helpful for teacher candidates. Strong and Gargani returned in May 2015. On the first day, thirty-eight university faculty (across all content areas) participated in eight hours of orientation and training on the RATE instrument. In the second day, each participant scored 8 videos in 30 minute time segments (a four hour time period plus breaks). The 30 minute timeframe included viewing the video clip, recording the scores on the RATE instrument, then conferring with a partner who viewed the same video. During the conferring, scores were reconciled. Each of the 150 videos were scored twice using both rater pairs and solo raters. Off site, trained RATE personnel evaluated all videos for comparison purposes.
In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
In March 2015, Strong and Gargani provided an orientation session on the RATE instrument for 38 ECU faculty (across all content areas), university supervisors, school district instructional coaches, and college of education administration faculty. This session focused on introducing faculty to the RATE instrument and how it may be helpful for teacher candidates. Strong and Gargani returned in May 2015. On the first day, thirty-eight university faculty (across all content areas) participated in eight hours of orientation and training on the RATE instrument. In the second day, each participant scored 8 videos in 30 minute time segments (a four hour time period plus breaks). The 30 minute timeframe included viewing the video clip, recording the scores on the RATE instrument, then conferring with a partner who viewed the same video. During the conferring, scores were reconciled. Each of the 150 videos were scored twice using both rater pairs and solo raters. Off site, trained RATE personnel evaluated all videos for comparison purposes.
In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
In March 2015, Strong and Gargani provided an orientation session on the RATE instrument for 38 ECU faculty (across all content areas), university supervisors, school district instructional coaches, and college of education administration faculty. This session focused on introducing faculty to the RATE instrument and how it may be helpful for teacher candidates. Strong and Gargani returned in May 2015. On the first day, thirty-eight university faculty (across all content areas) participated in eight hours of orientation and training on the RATE instrument. In the second day, each participant scored 8 videos in 30 minute time segments (a four hour time period plus breaks). The 30 minute timeframe included viewing the video clip, recording the scores on the RATE instrument, then conferring with a partner who viewed the same video. During the conferring, scores were reconciled. Each of the 150 videos were scored twice using both rater pairs and solo raters. Off site, trained RATE personnel evaluated all videos for comparison purposes.
In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
ECU faculty tended to rate teacher candidates higher on the lesson objective and student thinking portion of the videos. They tended to score teacher candidates lower on checking for understanding, clarity of delivery, flow of instruction, and time on task.
While there was good agreement among raters in most areas, flow of instruction seemed to show the biggest discrepancy. The cause of this discrepancy is not immediately apparent and warrants further examination.
It appears that discussing ratings with another person brings the extreme raters closer back to the norm. The closer the score was to the norm, the less it moved. The closer it was to the norm, the more it moved. This finding especially holds for ECU raters. RATE team scores were less variable across the rater.
Open circle is independent score
Close is revised sum scores
Even though discrepancies existed, rater reliability was higher in this study than in the RATEâs (VS7) validation study and the MET study conducted by the Gates Family Foundation. The biggest discrepancies occurred in the ECU facultyâs rating of clarity and flow of the lesson. Further exploration into the facultyâs interpretation of these categories is needed before that discrepancy can be explained.
The instrument itself is shorter and less complex than the current version of the instrument and requires only a 20 minute observation. One concern coming into the proof of concept study was that two raters were needed for each video. Given the large number of individuals within the program, having two individuals rate each person seems prohibitive. However, data indicate that having two raters for each video tends to hold weakly justified extreme scores in check. Since the raters are trained to do each session in a 30 minute time period, the process may actually end up taking the same amount of time, if not less time.
Continuing research is being conducted on observation practices that are effective and whether RATE is an observation instrument that can best support candidate growth in challenging clinical experiences. Being able to use the RATE instrument in live setting with teacher candidates would help teacher education programs have a reliable and valid outcome measure of their teachers. Â This measure in addition to other reliable outcomes measures, such as, the edTPA, would strengthen programsâ ability to predict the success of their graduates.
