SlideShare ist ein Scribd-Unternehmen logo
1 von 37
Downloaden Sie, um offline zu lesen
JISC Assessment and Feedback Programme
Webinar
Analysis of Feedback
24 June 2013 12.00 - 13.00
Holly Smith IoE Assessment Careers H.Smith@ioe.ac.uk
Anne Jones QUB e-AFFECT a.m.jones@qub.ac.uk
Maria Fernandez-
Toro
Open University eFEP Maria.Fernandez-Toro@open.ac.uk
Peter Chatterton Daedalus e-
World
Critical Friend peter.chatterton@daedalus-e-world.com
Institute of Education
www.ioe.ac.uk/assessmentcareers
Queen’s University Belfast
http://www.qub.ac.uk/directorates/AcademicStudentAffairs/CentreforEducationalDevelopment/e-AFFECTproject/
Open University http://www.open.ac.uk/blogs/efep/
Speakers
To explore the following:
• Why analyse feedback?
• Approaches and tools for analysing feedback
• Institutional experiences of using the tools - resulting
feedback profiles and audits
• Benefits, impact and challenges from using the tools
Webinar objective
Why analyse feedback?
Why analyse feedback?
• Widely inconsistent practice in feedback (quality & quantity)
• Lack of learner understanding of, engagement with, and
dialogue/action & on feedback
• High teacher effort - low efficiency
• Not utilising self/peer feedback
• Transmitted feedback creates dependency on teacher
• Reduced staff satisfaction as evidence of feed-forward not seen
• NSS scores
Dependent learners
feedback “done to them”
Independent learners
capable of self-review
Approaches and tools
for analysing feedback
IOE Coding framework
• The score is the number of times a classification
appears in the feedback
• The default unit for analysis was the sentence
• Where a sentence contains clauses that make
distinct points, it was split into separate clauses,
each of which was classified separately.
• Neutral comments that for example describe the
piece of work, but do not make any judgement are
unclassified
IOE Feedback tool
P1 Giving praise
P2 Recognising progress or ipsative feedback
Criticisms
C1 Correction of errors
C2 Factual criticisms
C3 Criticism of approach
Giving advice
A1 Specific to content current assignment
A2 General skills in current assignment
A3 For future assignments
Q Clarifications and questions
O Other unclassified statements
Adapted from Orsmond & Merry,
2011 including Hughes, 2011.
e-AFFECT - What the students say
Last year their feedback
pointed out spelling
mistakes or referencing
mistakes, but we were
not told how to do things
right
What do they mean
by clear and concise?
Sometimes they say
something to
encourage that is not
really true – am I
excelling at it or are you being nice?
I liked feedback which
helped me improve my
work the next time. Despite
this, I felt that my marks
never really changed much
and they tended to stay at
the same level
What do ‘?’ and
‘What?’ in the
margin mean?
e-AFFECT - Analysis of coursework
• Content analysis of depth of feedback using adapted
categories developed by Glover and Brown (2006)
• Indication that there is an strength/error/weakness or
omission (Level 1)
• Provides correction or appropriate response/indication why a
strength (Level 2)
• Provides explanation as to why the student’s response was
incorrect or inappropriate or why suggestion was preferable or
how a strength can be built upon (Level 3)
Essays Lab reports Total
Year 1 40 26 66
Year 2 61 24 85
Total 101 50 151
Analysis of feedback webinar
• Analysis criteria based around two dimensions:
o Whether feedback focuses on strengths or weaknesses
o How much information the feedback provides, cf ‘depth’ of
feedback (Brown & Glover 2006) layers of scaffolding
• Analysis tool: Feedback Analysis Chart for Tutors (FACT)
Provides a visual ‘profile’ of a tutor’s feedback
Analysing assignment feedback
Screencast description: http://www.open.ac.uk/blogs/efep/?page_id=114
Layers Comments focusing on weaknesses Comments focusing on strengths
1
Error identified only Strength identified only
2
Error categorised, but not corrected Strength categorised or described as per
marking criteria
3
Error corrected Illustrated with specific example from
student’s
performance
4
Explanation given Explanation given
5
Advice given on how to prevent errors in
future performance
Advice given on how to develop
existing strengths in future
Manos blancosManos blancos Good workGood work ????
Layers of scaffolding in assignment feedback
Manos blancos AgreementManos blancos Agreement
Although it ends in O, ‘mano’ is
a feminine noun.
Although it ends in O, ‘mano’ is
a feminine noun.
Manos blancos blancasManos blancos blancas
Revise section 6.1 of
your grammar book
Revise section 6.1 of
your grammar book
You use a wide range of
language structures
You use a wide range of
language structures
No digo que quieran...
Good use of the subjunctive
No digo que quieran...
Good use of the subjunctive
This connector makes it very clear
that a new section is starting here.
This connector makes it very clear
that a new section is starting here.
Good, you could
also look up...
Good, you could
also look up...
Feedback Analysis Chart for Tutors (FACT)
Possible uses:
Enables us to compare...
 feedback relating to different criteria
 feedback given by different tutors
 feedback given to more/less proficient students
 feedback related to different types of assignment
(e.g. spoken presentations vs. written essays)
 feedback delivered through different media
(e.g. written vs. audio-recorded feedback)
Language
Weaknesses Strengths
Identified only Identified only
Categorised
Categorised /
Described
Corrected /
Modelled
Exemplified
Explained Explained
Future-oriented Future-oriented
Example of FACT analysis grid
(tutor A: beginner assignment)
Weaknesses Strengths
Identified
only
6
Identified
only
Categorised 2
Categorised /
Described
Corrected /
Modelled
8 Exemplified
Explained 3 Explained
Future-
oriented
Future-
oriented
Weaknesses Strengths
Identified only Identified only
Categorised 2
Categorised /
Described
Corrected /
Modelled
1 Exemplified
Explained Explained
Future-oriented Future-oriented
Weaknesses Strengths
Identified
only
Identified
only
Categorised 3
Categorised /
Described
Corrected /
Modelled
Exemplified
Explained Explained
Future-
oriented
1
Future-
oriented
Content Feedback form
Notes on script
Language
Weaknesses Strengths
Identified only Identified only
Categorised 3
Categorised /
Described
Corrected /
Modelled
6 Exemplified
Explained 2 Explained
Future-oriented Future-oriented
Example of FACT analysis grid
(tutor B: advanced assignment)
Weaknesses Strengths
Identified
only
4
Identified
only
Categorised 1 5
Categorised /
Described
Corrected /
Modelled
2 6 Exemplified
Explained
5
Explained
Future-
oriented
Future-
oriented
Weaknesses Strengths
Identified only Identified only
Categorised 6
Categorised /
Described
Corrected /
Modelled
4 2 Exemplified
Explained 1 1 Explained
Future-oriented Future-oriented
Weaknesses Strengths
Identified
only
Identified
only
Categorised 1 6
Categorised /
Described
Corrected /
Modelled
2 Exemplified
Explained 2 Explained
Future-
oriented
Future-
oriented
Content Feedback form
Notes on script
Questions?
Institutional experiences
of using the tools
IOE Data
• Analysed formative and summative assessment
feedback for modules on 5 postgraduate programmes
at the IOE (total 228 pieces)
• Recorded the total number of comments in each
category and the average per script
• Ranked the categories to obtain a feedback profile at
programme level as well as an aggregate profile of the
5 programmes.
IOE Profile for Summative Assessment
Total comments for 5 programmes
Summative assessment (N= 165)
Category of feedback Average per script Rank
P1 Praise 4.4 1
P2 Ipsative (progress) 0 (negligible) 5
C1-C3 Critique 2.7 2
A1-A3 Advice for current or future
assignments
1.9 (mostly for current
assignment)
3
Q Questions and clarification
requests
0.1 4
e-AFFECT - Comments on essays
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
Student response Student skills Achievement
%
Distribution of comments at Year 1
Strengths
Weaknesses
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
50.0
Student response Student skills Achievement
%
Distribution of comments at Year 2
Strengths
Weaknesses
e-AFFECT - Further learning comments
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
Source
materials
Dialogue
and
reflection
Future work
%
% Year 1 work
% Year 2 work
e-AFFECT - Depth of feedback
Depth of feedback
Indication that there is an
strength/error/weakness or omission (Level 1)
Provides correction or appropriate
response/indication why a strength (Level 2)
Provides explanation as to why the student’s
response was incorrect or inappropriate or why
suggestion was preferable or how a strength can
be built upon (Level 3)
-30 -20 -10 0 10 20
1
2
3
%
Depth of
feedback
Year 2
Weakness Strength
-15 -10 -5 0 5 10 15
1
2
3
%
Depth of
feedback
Year 1
Weakness Strength
-40 -20 0 20 40
1
2
3
%
Depth of
feedback
Year 1 Module A
Weakness Strength
-30 -20 -10 0 10 20 30
1
2
3
%
Depth of
feedback
Year 1 Module B
Weakness Strength
-60 -40 -20 0 20 40
1
2
3
%
Depth of
feedback
Year 2 Module C
Weakness Strength
-40 -20 0 20 40
1
2
3
%
Depth of
feedback
Year 2 Module D
Weakness Strength
e-AFFECT – Module depth of feedback
e-AFFECT - Feedback workshop
Analysing what?
Feedback on language assignments at the OU
Our sample:
 100 writing assignments
 100 speaking assignments
 4 levels (9 tutors per level)
 108 students (3 per tutor)
Feedback consists of:
 200 e-feedback forms
 100 annotated scripts
 100 audio files
Tutor
Student
Well done!
Summary of FACT analysis results: Use of the four media
E-feedback forms (writing/speaking)
Contain the lowest proportion of…
 Comments on strengths
 Comments on content
 Categorised strengths/weaknesses
 Advice for future (proportion on script = 0%).
Quite low everywhere else (both in the audio
feedback and on the e-feedback forms)
Script annotations/Audio feedback
Contain the highest proportion of…
Comments on weaknesses (often
adressed at more than one depth)
Comments on language
Corrected errors
Explanations (especially high occurrence
in the audio feedback)
Contain the lowest proportion of…
 Comments on weaknesses
 Comments on language
 Corrected errors/Exemplified strengths
 Explanations
All of these occur a bit more frequently for
the speaking assignment
Contain the highest proportion of…
 Comments on strengths
 Comments on content
 Categorised strengths/weaknesses
All of these occur even more frequently
for the written assignment
Questions?
Benefits, impact and challenges
from using the tools
IOE Benefits, Impact and Challenges
• Enables initiation of discussion of feedback at the
programme team level
• Facilitates reflection on the purpose of feedback
within programme teams
• Feedback practices are very entrenched and resistant
to change
e-AFFECT
Benefits, impacts and challenges from using the tools
Benefits
• Staff seeing the real issue rather than an imagined issue
• Engendering dialogue
• Raising awareness of feedback messages
Impacts
• Work in progress
Challenges
• Reaching consensus of the level and quantity of
feedback
• Useful research tool: Overall patterns of use of different media for giving
feedback on language assignments at the OU
• Results need to be interpreted with caution (e.g. ‘deeper’ feedback is not
necessarily the most appropriate in all contexts)
• Not suitable for quantitative evaluation by practitioners: Coding requires
complex guidelines in order to be reliable
• Suitable for awareness-raising purposes in staff training events. Materials
include: sample of marked assignments + coding grids + student webcasts
giving their ‘feedback on feedback’
(Online training event: 67% ‘very useful’ – 33% ‘possibly useful’)
• FACT criteria now also presented as a simplified checklist for reflection
• Informs new research strand focusing on feedback alignment
Benefits, impact and challenges
from using the FACT analysis tool
Questions?
Further info on projects
More info on projects
Assessment Careers-Institute of Education
www.ioe.ac.uk/assessmentcareers
http://youtu.be/VSaGbPoXPh0
References:
1. Brown, E. & Glover, C. (2006) Evaluating written feedback. in: B. C. & K. Klegg
(Eds) Innovative assessment in higher education. London, Routledge), 81-91.
2. Hattie, J. & Timperley, H. (2007) The Power of Feedback. Review of Educational
Research, 77, 81-112.
3. Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative
Assessment in Higher Education Studies in Higher Education 36 (3): 353 – 367.
4. Orsmond, P. & Merry, S. (2011) Feedback alignment: effective and ineffective
links between tutors’ and students’ understanding of coursework feedback.
Assessment & Evaluation in Higher Education. 36(2): 125-126.
Holly Smith H.Smith@ioe.ac.uk
More info on projects
Queen’s University Belfast
e-AFFECT
http://www.qub.ac.uk/directorates/AcademicStudentAffair
s/CentreforEducationalDevelopment/e-AFFECTproject/
http://blogs.qub.ac.uk/e-affect/
Anne Jones a.m.jones@qub.ac.uk
More info on projects
Open University
The eFeedback Evaluation Project (eFEP)
http://www.open.ac.uk/blogs/efep/
Screencast description:
http://www.open.ac.uk/blogs/efep/?page_id=114
Maria Fernandez-Toro Maria.Fernandez-Toro@open.ac.uk

Weitere ähnliche Inhalte

Was ist angesagt?

Introduction to the IELTS
Introduction to the IELTSIntroduction to the IELTS
Introduction to the IELTSZakaria Rmidi
 
C2 πινακες δεξιοτητων και αξιολογησεων εξετασεων αγγλικης γλωσσας
C2 πινακες δεξιοτητων και αξιολογησεων εξετασεων αγγλικης γλωσσαςC2 πινακες δεξιοτητων και αξιολογησεων εξετασεων αγγλικης γλωσσας
C2 πινακες δεξιοτητων και αξιολογησεων εξετασεων αγγλικης γλωσσαςPetros Manzieris
 
English placement test
English placement test English placement test
English placement test Daniel Sanchez
 
Get ready for ielts listening free ebook
Get ready for ielts listening   free ebookGet ready for ielts listening   free ebook
Get ready for ielts listening free ebookssuser5396af
 
Fce speaking overview_activity
Fce speaking overview_activityFce speaking overview_activity
Fce speaking overview_activityelenainforb
 
TOEFL iBT basic guidelines!
TOEFL iBT basic guidelines!TOEFL iBT basic guidelines!
TOEFL iBT basic guidelines!Jihad Ghafur
 
Cambridge english first__fce__handbook
Cambridge english first__fce__handbookCambridge english first__fce__handbook
Cambridge english first__fce__handbookDinh Thi Diep Thao
 
Arief Firdaus An Insight Into Toefl & Ielts Day 1
Arief Firdaus An Insight Into Toefl & Ielts Day 1Arief Firdaus An Insight Into Toefl & Ielts Day 1
Arief Firdaus An Insight Into Toefl & Ielts Day 1Arief Firdaus
 
IELTS General Introduction
IELTS General IntroductionIELTS General Introduction
IELTS General IntroductionIELTSBackup
 
Language testing - Contrastive analysis
Language testing - Contrastive analysis Language testing - Contrastive analysis
Language testing - Contrastive analysis King Saud University
 

Was ist angesagt? (20)

Introduction to the IELTS
Introduction to the IELTSIntroduction to the IELTS
Introduction to the IELTS
 
IELTS ORIENTATION
IELTS ORIENTATIONIELTS ORIENTATION
IELTS ORIENTATION
 
C2 πινακες δεξιοτητων και αξιολογησεων εξετασεων αγγλικης γλωσσας
C2 πινακες δεξιοτητων και αξιολογησεων εξετασεων αγγλικης γλωσσαςC2 πινακες δεξιοτητων και αξιολογησεων εξετασεων αγγλικης γλωσσας
C2 πινακες δεξιοτητων και αξιολογησεων εξετασεων αγγλικης γλωσσας
 
English placement test
English placement test English placement test
English placement test
 
Testing grammar
Testing grammarTesting grammar
Testing grammar
 
Get ready for ielts listening free ebook
Get ready for ielts listening   free ebookGet ready for ielts listening   free ebook
Get ready for ielts listening free ebook
 
Fce speaking overview_activity
Fce speaking overview_activityFce speaking overview_activity
Fce speaking overview_activity
 
Sip2
Sip2Sip2
Sip2
 
Sip
SipSip
Sip
 
TOEFL iBT basic guidelines!
TOEFL iBT basic guidelines!TOEFL iBT basic guidelines!
TOEFL iBT basic guidelines!
 
Feedback m.cheraghi
Feedback  m.cheraghiFeedback  m.cheraghi
Feedback m.cheraghi
 
Cambridge english first__fce__handbook
Cambridge english first__fce__handbookCambridge english first__fce__handbook
Cambridge english first__fce__handbook
 
Ielts overview presentation
Ielts overview presentationIelts overview presentation
Ielts overview presentation
 
Overview of IELTS
Overview of IELTSOverview of IELTS
Overview of IELTS
 
Arief Firdaus An Insight Into Toefl & Ielts Day 1
Arief Firdaus An Insight Into Toefl & Ielts Day 1Arief Firdaus An Insight Into Toefl & Ielts Day 1
Arief Firdaus An Insight Into Toefl & Ielts Day 1
 
Igcse speaking test3
Igcse speaking test3Igcse speaking test3
Igcse speaking test3
 
Ielts workshop
Ielts workshopIelts workshop
Ielts workshop
 
IELTS General Introduction
IELTS General IntroductionIELTS General Introduction
IELTS General Introduction
 
Fce Exam Overview
Fce Exam OverviewFce Exam Overview
Fce Exam Overview
 
Language testing - Contrastive analysis
Language testing - Contrastive analysis Language testing - Contrastive analysis
Language testing - Contrastive analysis
 

Andere mochten auch

Andere mochten auch (8)

Presenting information presentation.
Presenting information presentation.Presenting information presentation.
Presenting information presentation.
 
Motivational Feedback
Motivational FeedbackMotivational Feedback
Motivational Feedback
 
Effective Feedback
Effective FeedbackEffective Feedback
Effective Feedback
 
Feedback
FeedbackFeedback
Feedback
 
Giving Effective Feedback.ppt
Giving Effective Feedback.pptGiving Effective Feedback.ppt
Giving Effective Feedback.ppt
 
Decision Making In Management
Decision Making In ManagementDecision Making In Management
Decision Making In Management
 
Decision Making Process
Decision Making ProcessDecision Making Process
Decision Making Process
 
Team Effectiveness
Team EffectivenessTeam Effectiveness
Team Effectiveness
 

Ähnlich wie Analysis of feedback webinar

Assessment in Distance Education
Assessment in Distance EducationAssessment in Distance Education
Assessment in Distance EducationSanjaya Mishra
 
SOC-520 Assessment AnalysisWorksheetIn order to ef.docx
SOC-520 Assessment AnalysisWorksheetIn order to ef.docxSOC-520 Assessment AnalysisWorksheetIn order to ef.docx
SOC-520 Assessment AnalysisWorksheetIn order to ef.docxpbilly1
 
SOC-520 Assessment AnalysisWorksheetIn order to ef.docx
SOC-520 Assessment AnalysisWorksheetIn order to ef.docxSOC-520 Assessment AnalysisWorksheetIn order to ef.docx
SOC-520 Assessment AnalysisWorksheetIn order to ef.docxrosemariebrayshaw
 
SOC-520 Assessment AnalysisWorksheetIn order to effectiv.docx
SOC-520 Assessment AnalysisWorksheetIn order to effectiv.docxSOC-520 Assessment AnalysisWorksheetIn order to effectiv.docx
SOC-520 Assessment AnalysisWorksheetIn order to effectiv.docxpbilly1
 
Rubric Development
Rubric DevelopmentRubric Development
Rubric DevelopmentSteven Baule
 
How to test grammar email
How to test grammar emailHow to test grammar email
How to test grammar emailjuniato
 
How to test grammar email
How to test grammar emailHow to test grammar email
How to test grammar emailjuniato
 
How to test grammar email
How to test grammar emailHow to test grammar email
How to test grammar emailjuniato
 
Unit 2 poetry mark scheme (higher)
Unit 2 poetry mark scheme (higher)Unit 2 poetry mark scheme (higher)
Unit 2 poetry mark scheme (higher)Mrs McMinn
 
Classroom assessment
Classroom assessmentClassroom assessment
Classroom assessmenterlynmg
 
Providing Feedback to Improve Performance
Providing Feedback to Improve PerformanceProviding Feedback to Improve Performance
Providing Feedback to Improve PerformancePooja Ganesh
 
Meister & Martinez The Rest of the 7 Student-Centered Strategies of Assessmen...
Meister & Martinez The Rest of the 7 Student-Centered Strategies of Assessmen...Meister & Martinez The Rest of the 7 Student-Centered Strategies of Assessmen...
Meister & Martinez The Rest of the 7 Student-Centered Strategies of Assessmen...Jeremy
 
Poetry across time jan 2012 mark scheme
Poetry across time jan 2012 mark schemePoetry across time jan 2012 mark scheme
Poetry across time jan 2012 mark schemeTpaisey
 
Assessment in he 2014 webinar 5
Assessment in he 2014 webinar 5Assessment in he 2014 webinar 5
Assessment in he 2014 webinar 5Rachel Forsyth
 
(Wolf) The Rest of the 7 Student-Centered Strategies of Assessment for Learning
(Wolf) The Rest of the 7 Student-Centered Strategies of Assessment for Learning (Wolf) The Rest of the 7 Student-Centered Strategies of Assessment for Learning
(Wolf) The Rest of the 7 Student-Centered Strategies of Assessment for Learning Jeremy
 
Assesing speaking skills
Assesing speaking skillsAssesing speaking skills
Assesing speaking skillssyed ahmed
 

Ähnlich wie Analysis of feedback webinar (20)

Assessment in Distance Education
Assessment in Distance EducationAssessment in Distance Education
Assessment in Distance Education
 
Rubric Development for Teachers
Rubric Development for TeachersRubric Development for Teachers
Rubric Development for Teachers
 
SOC-520 Assessment AnalysisWorksheetIn order to ef.docx
SOC-520 Assessment AnalysisWorksheetIn order to ef.docxSOC-520 Assessment AnalysisWorksheetIn order to ef.docx
SOC-520 Assessment AnalysisWorksheetIn order to ef.docx
 
SOC-520 Assessment AnalysisWorksheetIn order to ef.docx
SOC-520 Assessment AnalysisWorksheetIn order to ef.docxSOC-520 Assessment AnalysisWorksheetIn order to ef.docx
SOC-520 Assessment AnalysisWorksheetIn order to ef.docx
 
SOC-520 Assessment AnalysisWorksheetIn order to effectiv.docx
SOC-520 Assessment AnalysisWorksheetIn order to effectiv.docxSOC-520 Assessment AnalysisWorksheetIn order to effectiv.docx
SOC-520 Assessment AnalysisWorksheetIn order to effectiv.docx
 
Rubric Development
Rubric DevelopmentRubric Development
Rubric Development
 
My Seminar 3
My Seminar 3My Seminar 3
My Seminar 3
 
How to test grammar email
How to test grammar emailHow to test grammar email
How to test grammar email
 
How to test grammar email
How to test grammar emailHow to test grammar email
How to test grammar email
 
How to test grammar email
How to test grammar emailHow to test grammar email
How to test grammar email
 
Unit 2 poetry mark scheme (higher)
Unit 2 poetry mark scheme (higher)Unit 2 poetry mark scheme (higher)
Unit 2 poetry mark scheme (higher)
 
Classroom assessment
Classroom assessmentClassroom assessment
Classroom assessment
 
Providing Feedback to Improve Performance
Providing Feedback to Improve PerformanceProviding Feedback to Improve Performance
Providing Feedback to Improve Performance
 
Meister & Martinez The Rest of the 7 Student-Centered Strategies of Assessmen...
Meister & Martinez The Rest of the 7 Student-Centered Strategies of Assessmen...Meister & Martinez The Rest of the 7 Student-Centered Strategies of Assessmen...
Meister & Martinez The Rest of the 7 Student-Centered Strategies of Assessmen...
 
Poetry across time jan 2012 mark scheme
Poetry across time jan 2012 mark schemePoetry across time jan 2012 mark scheme
Poetry across time jan 2012 mark scheme
 
Designing classroom language tests
Designing classroom language testsDesigning classroom language tests
Designing classroom language tests
 
Assessment in he 2014 webinar 5
Assessment in he 2014 webinar 5Assessment in he 2014 webinar 5
Assessment in he 2014 webinar 5
 
Assessment1
Assessment1Assessment1
Assessment1
 
(Wolf) The Rest of the 7 Student-Centered Strategies of Assessment for Learning
(Wolf) The Rest of the 7 Student-Centered Strategies of Assessment for Learning (Wolf) The Rest of the 7 Student-Centered Strategies of Assessment for Learning
(Wolf) The Rest of the 7 Student-Centered Strategies of Assessment for Learning
 
Assesing speaking skills
Assesing speaking skillsAssesing speaking skills
Assesing speaking skills
 

Mehr von jisc-elearning

Developing digital capability for the future
Developing digital capability for the futureDeveloping digital capability for the future
Developing digital capability for the futurejisc-elearning
 
Strategic Approaches to Digital Literacies: Gregynog Colloquium 2014
Strategic Approaches to Digital Literacies: Gregynog Colloquium 2014Strategic Approaches to Digital Literacies: Gregynog Colloquium 2014
Strategic Approaches to Digital Literacies: Gregynog Colloquium 2014jisc-elearning
 
The changing face of assessment and feedback: how technology can make a diffe...
The changing face of assessment and feedback: how technology can make a diffe...The changing face of assessment and feedback: how technology can make a diffe...
The changing face of assessment and feedback: how technology can make a diffe...jisc-elearning
 
Feed-forward approaches for enhancing assessment and feedback
Feed-forward approaches for enhancing assessment and feedbackFeed-forward approaches for enhancing assessment and feedback
Feed-forward approaches for enhancing assessment and feedbackjisc-elearning
 
Support assessment and feedback practice with technology: a view of the UK la...
Support assessment and feedback practice with technology: a view of the UK la...Support assessment and feedback practice with technology: a view of the UK la...
Support assessment and feedback practice with technology: a view of the UK la...jisc-elearning
 
Approaches to changing assessment and feedback practice
Approaches to changing assessment and feedback practiceApproaches to changing assessment and feedback practice
Approaches to changing assessment and feedback practicejisc-elearning
 
Learning analytics for assessment and feedback
Learning analytics for assessment and feedbackLearning analytics for assessment and feedback
Learning analytics for assessment and feedbackjisc-elearning
 
Changing student practices and digital literacies
Changing student practices and digital literaciesChanging student practices and digital literacies
Changing student practices and digital literaciesjisc-elearning
 
Exploring employability through assessment and feedback
Exploring employability through assessment and feedbackExploring employability through assessment and feedback
Exploring employability through assessment and feedbackjisc-elearning
 
Jisc webinar: Implementing the UKPSF in the digital university
Jisc webinar: Implementing the UKPSF in the digital universityJisc webinar: Implementing the UKPSF in the digital university
Jisc webinar: Implementing the UKPSF in the digital universityjisc-elearning
 
Jisc webinar: developing staff digital literacies
Jisc webinar: developing staff digital literaciesJisc webinar: developing staff digital literacies
Jisc webinar: developing staff digital literaciesjisc-elearning
 
Evaluation of a comms and tutoring system at Dundee
Evaluation of a comms and tutoring system at DundeeEvaluation of a comms and tutoring system at Dundee
Evaluation of a comms and tutoring system at Dundeejisc-elearning
 
Peer review and the development of evaluative skills
Peer review and the development of evaluative skillsPeer review and the development of evaluative skills
Peer review and the development of evaluative skillsjisc-elearning
 
Using open source assessment and feedback tools
Using open source assessment and feedback toolsUsing open source assessment and feedback tools
Using open source assessment and feedback toolsjisc-elearning
 
Current issues and approaches in developing digital literacy
Current issues and approaches in developing digital literacyCurrent issues and approaches in developing digital literacy
Current issues and approaches in developing digital literacyjisc-elearning
 
CAA 2012 Closing Address
CAA 2012 Closing Address CAA 2012 Closing Address
CAA 2012 Closing Address jisc-elearning
 
Assessment and Feedback Cluster A IBLC 2012
Assessment and Feedback Cluster A IBLC 2012Assessment and Feedback Cluster A IBLC 2012
Assessment and Feedback Cluster A IBLC 2012jisc-elearning
 
Assessment and Feedback programme update (April 2012)
Assessment and Feedback programme update (April 2012)Assessment and Feedback programme update (April 2012)
Assessment and Feedback programme update (April 2012)jisc-elearning
 
Assessment and Feedback start-up meeting Oct 2011
Assessment and Feedback start-up meeting Oct 2011Assessment and Feedback start-up meeting Oct 2011
Assessment and Feedback start-up meeting Oct 2011jisc-elearning
 
Winning hearts and minds: tools and techniques to engage staff in curriculum ...
Winning hearts and minds: tools and techniques to engage staff in curriculum ...Winning hearts and minds: tools and techniques to engage staff in curriculum ...
Winning hearts and minds: tools and techniques to engage staff in curriculum ...jisc-elearning
 

Mehr von jisc-elearning (20)

Developing digital capability for the future
Developing digital capability for the futureDeveloping digital capability for the future
Developing digital capability for the future
 
Strategic Approaches to Digital Literacies: Gregynog Colloquium 2014
Strategic Approaches to Digital Literacies: Gregynog Colloquium 2014Strategic Approaches to Digital Literacies: Gregynog Colloquium 2014
Strategic Approaches to Digital Literacies: Gregynog Colloquium 2014
 
The changing face of assessment and feedback: how technology can make a diffe...
The changing face of assessment and feedback: how technology can make a diffe...The changing face of assessment and feedback: how technology can make a diffe...
The changing face of assessment and feedback: how technology can make a diffe...
 
Feed-forward approaches for enhancing assessment and feedback
Feed-forward approaches for enhancing assessment and feedbackFeed-forward approaches for enhancing assessment and feedback
Feed-forward approaches for enhancing assessment and feedback
 
Support assessment and feedback practice with technology: a view of the UK la...
Support assessment and feedback practice with technology: a view of the UK la...Support assessment and feedback practice with technology: a view of the UK la...
Support assessment and feedback practice with technology: a view of the UK la...
 
Approaches to changing assessment and feedback practice
Approaches to changing assessment and feedback practiceApproaches to changing assessment and feedback practice
Approaches to changing assessment and feedback practice
 
Learning analytics for assessment and feedback
Learning analytics for assessment and feedbackLearning analytics for assessment and feedback
Learning analytics for assessment and feedback
 
Changing student practices and digital literacies
Changing student practices and digital literaciesChanging student practices and digital literacies
Changing student practices and digital literacies
 
Exploring employability through assessment and feedback
Exploring employability through assessment and feedbackExploring employability through assessment and feedback
Exploring employability through assessment and feedback
 
Jisc webinar: Implementing the UKPSF in the digital university
Jisc webinar: Implementing the UKPSF in the digital universityJisc webinar: Implementing the UKPSF in the digital university
Jisc webinar: Implementing the UKPSF in the digital university
 
Jisc webinar: developing staff digital literacies
Jisc webinar: developing staff digital literaciesJisc webinar: developing staff digital literacies
Jisc webinar: developing staff digital literacies
 
Evaluation of a comms and tutoring system at Dundee
Evaluation of a comms and tutoring system at DundeeEvaluation of a comms and tutoring system at Dundee
Evaluation of a comms and tutoring system at Dundee
 
Peer review and the development of evaluative skills
Peer review and the development of evaluative skillsPeer review and the development of evaluative skills
Peer review and the development of evaluative skills
 
Using open source assessment and feedback tools
Using open source assessment and feedback toolsUsing open source assessment and feedback tools
Using open source assessment and feedback tools
 
Current issues and approaches in developing digital literacy
Current issues and approaches in developing digital literacyCurrent issues and approaches in developing digital literacy
Current issues and approaches in developing digital literacy
 
CAA 2012 Closing Address
CAA 2012 Closing Address CAA 2012 Closing Address
CAA 2012 Closing Address
 
Assessment and Feedback Cluster A IBLC 2012
Assessment and Feedback Cluster A IBLC 2012Assessment and Feedback Cluster A IBLC 2012
Assessment and Feedback Cluster A IBLC 2012
 
Assessment and Feedback programme update (April 2012)
Assessment and Feedback programme update (April 2012)Assessment and Feedback programme update (April 2012)
Assessment and Feedback programme update (April 2012)
 
Assessment and Feedback start-up meeting Oct 2011
Assessment and Feedback start-up meeting Oct 2011Assessment and Feedback start-up meeting Oct 2011
Assessment and Feedback start-up meeting Oct 2011
 
Winning hearts and minds: tools and techniques to engage staff in curriculum ...
Winning hearts and minds: tools and techniques to engage staff in curriculum ...Winning hearts and minds: tools and techniques to engage staff in curriculum ...
Winning hearts and minds: tools and techniques to engage staff in curriculum ...
 

Kürzlich hochgeladen

Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....Riddhi Kevadiya
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxraviapr7
 
How to Send Emails From Odoo 17 Using Code
How to Send Emails From Odoo 17 Using CodeHow to Send Emails From Odoo 17 Using Code
How to Send Emails From Odoo 17 Using CodeCeline George
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxraviapr7
 
Protein Structure - threading Protein modelling pptx
Protein Structure - threading Protein modelling pptxProtein Structure - threading Protein modelling pptx
Protein Structure - threading Protein modelling pptxvidhisharma994099
 
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...M56BOOKSTORE PRODUCT/SERVICE
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfMohonDas
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesCeline George
 
How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17Celine George
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapitolTechU
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxiammrhaywood
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...CaraSkikne1
 
Vani Magazine - Quarterly Magazine of Seshadripuram Educational Trust
Vani Magazine - Quarterly Magazine of Seshadripuram Educational TrustVani Magazine - Quarterly Magazine of Seshadripuram Educational Trust
Vani Magazine - Quarterly Magazine of Seshadripuram Educational TrustSavipriya Raghavendra
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?TechSoup
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17Celine George
 
A gentle introduction to Artificial Intelligence
A gentle introduction to Artificial IntelligenceA gentle introduction to Artificial Intelligence
A gentle introduction to Artificial IntelligenceApostolos Syropoulos
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICESayali Powar
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfMohonDas
 

Kürzlich hochgeladen (20)

Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....Riddhi Kevadiya. WILLIAM SHAKESPEARE....
Riddhi Kevadiya. WILLIAM SHAKESPEARE....
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptx
 
How to Send Emails From Odoo 17 Using Code
How to Send Emails From Odoo 17 Using CodeHow to Send Emails From Odoo 17 Using Code
How to Send Emails From Odoo 17 Using Code
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptx
 
Protein Structure - threading Protein modelling pptx
Protein Structure - threading Protein modelling pptxProtein Structure - threading Protein modelling pptx
Protein Structure - threading Protein modelling pptx
 
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdfPersonal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
 
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...KARNAADA.pptx  made by -  saransh dwivedi ( SD ) -  SHALAKYA TANTRA - ENT - 4...
KARNAADA.pptx made by - saransh dwivedi ( SD ) - SHALAKYA TANTRA - ENT - 4...
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdf
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 Sales
 
How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptx
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...
 
Vani Magazine - Quarterly Magazine of Seshadripuram Educational Trust
Vani Magazine - Quarterly Magazine of Seshadripuram Educational TrustVani Magazine - Quarterly Magazine of Seshadripuram Educational Trust
Vani Magazine - Quarterly Magazine of Seshadripuram Educational Trust
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17
 
A gentle introduction to Artificial Intelligence
A gentle introduction to Artificial IntelligenceA gentle introduction to Artificial Intelligence
A gentle introduction to Artificial Intelligence
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICE
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdf
 

Analysis of feedback webinar

  • 1. JISC Assessment and Feedback Programme Webinar Analysis of Feedback 24 June 2013 12.00 - 13.00
  • 2. Holly Smith IoE Assessment Careers H.Smith@ioe.ac.uk Anne Jones QUB e-AFFECT a.m.jones@qub.ac.uk Maria Fernandez- Toro Open University eFEP Maria.Fernandez-Toro@open.ac.uk Peter Chatterton Daedalus e- World Critical Friend peter.chatterton@daedalus-e-world.com Institute of Education www.ioe.ac.uk/assessmentcareers Queen’s University Belfast http://www.qub.ac.uk/directorates/AcademicStudentAffairs/CentreforEducationalDevelopment/e-AFFECTproject/ Open University http://www.open.ac.uk/blogs/efep/ Speakers
  • 3. To explore the following: • Why analyse feedback? • Approaches and tools for analysing feedback • Institutional experiences of using the tools - resulting feedback profiles and audits • Benefits, impact and challenges from using the tools Webinar objective
  • 5. Why analyse feedback? • Widely inconsistent practice in feedback (quality & quantity) • Lack of learner understanding of, engagement with, and dialogue/action & on feedback • High teacher effort - low efficiency • Not utilising self/peer feedback • Transmitted feedback creates dependency on teacher • Reduced staff satisfaction as evidence of feed-forward not seen • NSS scores Dependent learners feedback “done to them” Independent learners capable of self-review
  • 6. Approaches and tools for analysing feedback
  • 7. IOE Coding framework • The score is the number of times a classification appears in the feedback • The default unit for analysis was the sentence • Where a sentence contains clauses that make distinct points, it was split into separate clauses, each of which was classified separately. • Neutral comments that for example describe the piece of work, but do not make any judgement are unclassified
  • 8. IOE Feedback tool P1 Giving praise P2 Recognising progress or ipsative feedback Criticisms C1 Correction of errors C2 Factual criticisms C3 Criticism of approach Giving advice A1 Specific to content current assignment A2 General skills in current assignment A3 For future assignments Q Clarifications and questions O Other unclassified statements Adapted from Orsmond & Merry, 2011 including Hughes, 2011.
  • 9. e-AFFECT - What the students say Last year their feedback pointed out spelling mistakes or referencing mistakes, but we were not told how to do things right What do they mean by clear and concise? Sometimes they say something to encourage that is not really true – am I excelling at it or are you being nice? I liked feedback which helped me improve my work the next time. Despite this, I felt that my marks never really changed much and they tended to stay at the same level What do ‘?’ and ‘What?’ in the margin mean?
  • 10. e-AFFECT - Analysis of coursework • Content analysis of depth of feedback using adapted categories developed by Glover and Brown (2006) • Indication that there is an strength/error/weakness or omission (Level 1) • Provides correction or appropriate response/indication why a strength (Level 2) • Provides explanation as to why the student’s response was incorrect or inappropriate or why suggestion was preferable or how a strength can be built upon (Level 3) Essays Lab reports Total Year 1 40 26 66 Year 2 61 24 85 Total 101 50 151
  • 12. • Analysis criteria based around two dimensions: o Whether feedback focuses on strengths or weaknesses o How much information the feedback provides, cf ‘depth’ of feedback (Brown & Glover 2006) layers of scaffolding • Analysis tool: Feedback Analysis Chart for Tutors (FACT) Provides a visual ‘profile’ of a tutor’s feedback Analysing assignment feedback Screencast description: http://www.open.ac.uk/blogs/efep/?page_id=114
  • 13. Layers Comments focusing on weaknesses Comments focusing on strengths 1 Error identified only Strength identified only 2 Error categorised, but not corrected Strength categorised or described as per marking criteria 3 Error corrected Illustrated with specific example from student’s performance 4 Explanation given Explanation given 5 Advice given on how to prevent errors in future performance Advice given on how to develop existing strengths in future Manos blancosManos blancos Good workGood work ???? Layers of scaffolding in assignment feedback Manos blancos AgreementManos blancos Agreement Although it ends in O, ‘mano’ is a feminine noun. Although it ends in O, ‘mano’ is a feminine noun. Manos blancos blancasManos blancos blancas Revise section 6.1 of your grammar book Revise section 6.1 of your grammar book You use a wide range of language structures You use a wide range of language structures No digo que quieran... Good use of the subjunctive No digo que quieran... Good use of the subjunctive This connector makes it very clear that a new section is starting here. This connector makes it very clear that a new section is starting here. Good, you could also look up... Good, you could also look up...
  • 14. Feedback Analysis Chart for Tutors (FACT) Possible uses: Enables us to compare...  feedback relating to different criteria  feedback given by different tutors  feedback given to more/less proficient students  feedback related to different types of assignment (e.g. spoken presentations vs. written essays)  feedback delivered through different media (e.g. written vs. audio-recorded feedback)
  • 15. Language Weaknesses Strengths Identified only Identified only Categorised Categorised / Described Corrected / Modelled Exemplified Explained Explained Future-oriented Future-oriented Example of FACT analysis grid (tutor A: beginner assignment) Weaknesses Strengths Identified only 6 Identified only Categorised 2 Categorised / Described Corrected / Modelled 8 Exemplified Explained 3 Explained Future- oriented Future- oriented Weaknesses Strengths Identified only Identified only Categorised 2 Categorised / Described Corrected / Modelled 1 Exemplified Explained Explained Future-oriented Future-oriented Weaknesses Strengths Identified only Identified only Categorised 3 Categorised / Described Corrected / Modelled Exemplified Explained Explained Future- oriented 1 Future- oriented Content Feedback form Notes on script
  • 16. Language Weaknesses Strengths Identified only Identified only Categorised 3 Categorised / Described Corrected / Modelled 6 Exemplified Explained 2 Explained Future-oriented Future-oriented Example of FACT analysis grid (tutor B: advanced assignment) Weaknesses Strengths Identified only 4 Identified only Categorised 1 5 Categorised / Described Corrected / Modelled 2 6 Exemplified Explained 5 Explained Future- oriented Future- oriented Weaknesses Strengths Identified only Identified only Categorised 6 Categorised / Described Corrected / Modelled 4 2 Exemplified Explained 1 1 Explained Future-oriented Future-oriented Weaknesses Strengths Identified only Identified only Categorised 1 6 Categorised / Described Corrected / Modelled 2 Exemplified Explained 2 Explained Future- oriented Future- oriented Content Feedback form Notes on script
  • 19. IOE Data • Analysed formative and summative assessment feedback for modules on 5 postgraduate programmes at the IOE (total 228 pieces) • Recorded the total number of comments in each category and the average per script • Ranked the categories to obtain a feedback profile at programme level as well as an aggregate profile of the 5 programmes.
  • 20. IOE Profile for Summative Assessment Total comments for 5 programmes Summative assessment (N= 165) Category of feedback Average per script Rank P1 Praise 4.4 1 P2 Ipsative (progress) 0 (negligible) 5 C1-C3 Critique 2.7 2 A1-A3 Advice for current or future assignments 1.9 (mostly for current assignment) 3 Q Questions and clarification requests 0.1 4
  • 21. e-AFFECT - Comments on essays 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 45.0 Student response Student skills Achievement % Distribution of comments at Year 1 Strengths Weaknesses 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 45.0 50.0 Student response Student skills Achievement % Distribution of comments at Year 2 Strengths Weaknesses
  • 22. e-AFFECT - Further learning comments 0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 Source materials Dialogue and reflection Future work % % Year 1 work % Year 2 work
  • 23. e-AFFECT - Depth of feedback Depth of feedback Indication that there is an strength/error/weakness or omission (Level 1) Provides correction or appropriate response/indication why a strength (Level 2) Provides explanation as to why the student’s response was incorrect or inappropriate or why suggestion was preferable or how a strength can be built upon (Level 3) -30 -20 -10 0 10 20 1 2 3 % Depth of feedback Year 2 Weakness Strength -15 -10 -5 0 5 10 15 1 2 3 % Depth of feedback Year 1 Weakness Strength
  • 24. -40 -20 0 20 40 1 2 3 % Depth of feedback Year 1 Module A Weakness Strength -30 -20 -10 0 10 20 30 1 2 3 % Depth of feedback Year 1 Module B Weakness Strength -60 -40 -20 0 20 40 1 2 3 % Depth of feedback Year 2 Module C Weakness Strength -40 -20 0 20 40 1 2 3 % Depth of feedback Year 2 Module D Weakness Strength e-AFFECT – Module depth of feedback
  • 26. Analysing what? Feedback on language assignments at the OU Our sample:  100 writing assignments  100 speaking assignments  4 levels (9 tutors per level)  108 students (3 per tutor) Feedback consists of:  200 e-feedback forms  100 annotated scripts  100 audio files Tutor Student Well done!
  • 27. Summary of FACT analysis results: Use of the four media E-feedback forms (writing/speaking) Contain the lowest proportion of…  Comments on strengths  Comments on content  Categorised strengths/weaknesses  Advice for future (proportion on script = 0%). Quite low everywhere else (both in the audio feedback and on the e-feedback forms) Script annotations/Audio feedback Contain the highest proportion of… Comments on weaknesses (often adressed at more than one depth) Comments on language Corrected errors Explanations (especially high occurrence in the audio feedback) Contain the lowest proportion of…  Comments on weaknesses  Comments on language  Corrected errors/Exemplified strengths  Explanations All of these occur a bit more frequently for the speaking assignment Contain the highest proportion of…  Comments on strengths  Comments on content  Categorised strengths/weaknesses All of these occur even more frequently for the written assignment
  • 29. Benefits, impact and challenges from using the tools
  • 30. IOE Benefits, Impact and Challenges • Enables initiation of discussion of feedback at the programme team level • Facilitates reflection on the purpose of feedback within programme teams • Feedback practices are very entrenched and resistant to change
  • 31. e-AFFECT Benefits, impacts and challenges from using the tools Benefits • Staff seeing the real issue rather than an imagined issue • Engendering dialogue • Raising awareness of feedback messages Impacts • Work in progress Challenges • Reaching consensus of the level and quantity of feedback
  • 32. • Useful research tool: Overall patterns of use of different media for giving feedback on language assignments at the OU • Results need to be interpreted with caution (e.g. ‘deeper’ feedback is not necessarily the most appropriate in all contexts) • Not suitable for quantitative evaluation by practitioners: Coding requires complex guidelines in order to be reliable • Suitable for awareness-raising purposes in staff training events. Materials include: sample of marked assignments + coding grids + student webcasts giving their ‘feedback on feedback’ (Online training event: 67% ‘very useful’ – 33% ‘possibly useful’) • FACT criteria now also presented as a simplified checklist for reflection • Informs new research strand focusing on feedback alignment Benefits, impact and challenges from using the FACT analysis tool
  • 34. Further info on projects
  • 35. More info on projects Assessment Careers-Institute of Education www.ioe.ac.uk/assessmentcareers http://youtu.be/VSaGbPoXPh0 References: 1. Brown, E. & Glover, C. (2006) Evaluating written feedback. in: B. C. & K. Klegg (Eds) Innovative assessment in higher education. London, Routledge), 81-91. 2. Hattie, J. & Timperley, H. (2007) The Power of Feedback. Review of Educational Research, 77, 81-112. 3. Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative Assessment in Higher Education Studies in Higher Education 36 (3): 353 – 367. 4. Orsmond, P. & Merry, S. (2011) Feedback alignment: effective and ineffective links between tutors’ and students’ understanding of coursework feedback. Assessment & Evaluation in Higher Education. 36(2): 125-126. Holly Smith H.Smith@ioe.ac.uk
  • 36. More info on projects Queen’s University Belfast e-AFFECT http://www.qub.ac.uk/directorates/AcademicStudentAffair s/CentreforEducationalDevelopment/e-AFFECTproject/ http://blogs.qub.ac.uk/e-affect/ Anne Jones a.m.jones@qub.ac.uk
  • 37. More info on projects Open University The eFeedback Evaluation Project (eFEP) http://www.open.ac.uk/blogs/efep/ Screencast description: http://www.open.ac.uk/blogs/efep/?page_id=114 Maria Fernandez-Toro Maria.Fernandez-Toro@open.ac.uk