This paper, by comparing and contrasting between two different formative assessment protocols used in a first year undergraduate module, investigates the formation of student self-assessment skills. We operationalise the concept of self-assessment skills by measuring the relationship between student attainment and student confidence in their own performance. We find that, whilst this understanding of student confidence is related to attainment levels, there is a significant asymmetry across the two protocols adopted. Independent of the formative assessment type, high-attainment students display a consistent positive association between confidence and attainment. In contrast, low-attainment students display a relationship between confidence and attainment in only one of the two formative assessment set-ups. We conclude that self-assessment skills are tied to the assessment format.

1. 1
Assessing Student Self-Assessment:
An Additional Argument for
Blended Learning
Fabio R. Aricò
Duncan Watson
Chris Thomson
NRHE SRHE – Dec 2014

2. ACKNOWLEDGEMENTS
UEA-HEFCE Widening Participation Teaching Fellowship
HEA – Teaching Development Grant Scheme
2

3. OUTLINE
1. Overview of the “When Student Confidence Clicks Project”
2. Introduction to 2 learning environments to be compared:
seminars and workshops
Learning analytics:
the relationship between attainment & self-assessment
4. Final remarks: assessing self-assessment
5. Further research
3

4. ETHICAL REMARK
You will be presented with data collected during teaching sessions.
Students involved have given informed consent for me to analyse
their responses and present the results of this analysis.
I can assist with ethical queries as well, please ask me.
4

5. 1. When Student Confidence
Clicks: the role of Academic
Self-Efficacy in Learning
5

6. PROJECT DEVELOPMENTS
Project Resources: https://sites.google.com/site/fabioarico/hea_tdg
Project Workshop: Wednesday 3 Sep 2014 - University of East Anglia
6

7. BLENDED LEARNING ENVIRONMENT
• Intense use of Student Response Systems (clickers).
• Focus on increasing students’ Academic Self-Efficacy beliefs
 Self-Assessment is a fundamental component.
• Intense use of SRS + VLE
 dialogue with the students – closing feedback loop
 blended learning, but also blended surveying!
• Learning analytics to disentangle relationship between
different aspects of the learning experience.
7

8. DATASETS
Student Q1 Q2 Q3 … 
1 0 1 1
2 1 0 0
3 1 1 …
…
 performance per question
confidence by question
performanceperstudent
confidencebystudent
8
longitudinal study
- across all lectures
- across all seminars
- across all workshops
Intermediate and final
attainment outcomes
- course test
- final exam

9. EVIDENCE-BASED PRACTICE
9
ATTAINMENT
ENGAGEMENT CONFIDENCE
Gender
Satisfaction
Previous
attainmentDomicilePOLAR
Perceived
difficulties

10. 2. Workshops and Seminars
Attainment & Self-Assessment
10

11. MOTIVATION and CORE IDEAS
• This is a by-product of my HEA TDG project, which is taking life on its own
 Self-Assessment as an important determinant of Self-Efficacy beliefs
(Bandura, 1977 – Pajares, 1996)
• Self-Assessment skills contribute to form autonomous students
 HE learning and teaching practice should develop these
(Boud, 1995 – Boud and Falchikov, 2006/7 – Evans, 2013)
• Empirical evidence seems to identify positive association between
self-assessment skills and performance in academic settings
 Dunning et.al. (2003), Zell & Krizan (2014)
• Challenge: does (self)assessment design have a role to play?
11

12. TEACHING PROTOCOL – the module
Introductory
Macroeconomics  Level 1 – compulsory year-long module - 170 students
Lectures  traditional frontal-teaching (10 per sem.)
Seminars  small group, pre-assigned problem sets (4 per sem.)
Workshops  large group, problem-solving sessions (4 per sem.)
Support Sessions  non-compulsory drop-in sessions (4 per sem.)
12

13. WORKSHOPS – learning environment
• Large flipped-class environment (2 Groups - 85 students each)
• 1st part: Multiple choice questions (4 choices) - 2nd part: problem-solving
• Iterated algorithm:
1. Ask Question 1st round
 4 choices (distribution not revealed)
2. Ask Self-Assessment Question
 4 levels Likert-scale (strongly/agree – strongly/disagree)
3. Peer-Instruction
4. Ask Question 2nd round
 show answers – teach correct answer and give explanations.
13

14. WORKSHOPS – data coding
For each session (7 in a year):
• Code 1st response: 1 = correct
0 = incorrect
• Code confidence in response: 1 = strongly/agree
0 = strongly/disagree
• Compute average score and average confidence per student.
attainment self-assessment
measure measure
14

15. WORKSHOPS – data coding
For each student (in each session):
If student average score > session average score  high-attainment
otherwise low-attainment
METHOD A:
If student average conf. > session average conf.  high-confidence
otherwise low-confidence
METHOD B:
If student average conf. > 0.5  high-confidence
otherwise low-confidence
15

16. WORKSHOPS – data analysis
Cross-tabulate results:
16

17. WORKSHOPS - results
What is the relationship between attainment and confidence?
17

18. WORKSHOP - results
What is the relationship between attainment and confidence?
18

19. SEMINARS – learning environment
• Small-class environment (Groups of 13-15 students each)
• 1st part: Multiple choice quiz (4 choices) - 2nd part: problem-solving
• Structure of the Quiz:
- Paper based
- 5-6 minutes duration
- 3 or 4 formative questions
- 1 self-assessment question on the quiz
- 1 self-efficacy question about future beliefs.
19

20. SEMINARS – data coding
For each session (7 in a year):
• Code each answer: 1 = correct
0 = incorrect
• Code confidence in response: 1 = strongly/agree
0 = strongly/disagree
• Compute average score attainment measure
There is just one confidence response confidence measure
20

21. SEMINARS – data coding
For each student (in each session):
If student average score > session average score  high-attainment
otherwise low-attainment
If student confidence = 1  high-confidence
otherwise low-confidence
Note the asymmetry with workshops.
This was addressed this year, asking about confidence question by question.
21

22. SEMINARS - results
What is the relationship between attainment and confidence?
22

23. SEMINARS - results
What is the relationship between attainment and confidence?
23

24. 3. Final Remarks
Assessing Self-Assessment
24

25. PREVIOUS LITERATURE
• In literature there is a prevailing idea:
low-attainment students are not able to self-assess their performance.
• Dunning, D., Johnson, K., Ehrlinger, J., and Kruger, J., (2003), “Why People Fail
to Recognise their Own Incompetence”, Current Directions in Psychological
Science, 12:83, 83-87.
• But the results find in my research seem to question the learning
environment within which self-assessment is being observed.
 the learning environment affects the relationship attainment-confidence.
25

26. OVERALL RESULTS
• In Seminar Quizzes:
 high-attainment students display higher confidence
 low-attainment students not able to self-assess their performance.
• In Workshop sessions:
 high-attainment students display higher confidence
 low-attainment students display lower confidence.
• How to interpret this asymmetry?
26

27. OVERALL RESULTS
• In Seminar Quizzes:
 3 or 4 questions, paper-based quiz, 5-6 minutes, not anonymous, small class
 1 confidence assessment for overall performance.
• In Workshop sessions:
 5-10 questions, clicker response, slower pace, quasi-anonymous, large class
 1 confidence assessment for each question asked.
27

28. OVERALL RESULTS
• Low-attainment students encounter more difficulties in
self-assessing their performance in an environment where:
 they self-assess their ‘overall’ performance on a composite task
 they are exposed to questions for a shorter period of time
 they are exposed to fewer questions, not anonymously.
• Focus group interviews (differentiated by attainment groups)
confirm that low-attainment students display poorer self-assessment skills.
• So what is happening in workshops?
28

29. 4. Further Research
29

30. FURTHER RESEARCH
• Introduce confidence evaluation question-by-question in Seminars.
• Explore the relationship between objective and subjective evaluation
 relationship between response entropy and student confidence.
• “Peer-Instruction Unveiled”
 relationship between 1st and 2nd response – ‘learning gains’
 relationship between learning gains and student confidence.
• Learning analytics at student-level
 investigate the role of demographics
 investigate the impact of formative assessment on summative assessment.
30