22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence” Afternoon Presentations

22 January 2018 HEFCE
open event “Using data to
increase learning gains and
teaching excellence”.
https://twitter.com/LearningGains
#learninggainsOU
https://abclearninggains.com/

Presentations in the afternoon
Sonia Ilie
University of Cambridge
Heike Behle
Warwick
University
Paul Mcdermott
University of East
Anglia
Fabio Arico
University of East Anglia
Fiona Cobb
University of London

Learning gain:
from concept to measurement
LEGACY – Cambridge Strand – Jan Vermunt, Anna Vignoles & Sonia Ilie

LEGACY aims
Develop a context-appropriate theoretical understanding of
learning gain
Develop and test an instrument to measure learning gain:
reliability, validity and at-scale usability
Test a longitudinal model of learning gain in relation to student
background characteristics, contextual factors, and existing
measures of academic success

Learning Gain
Complex concept
Multidimensional
A wide variety of
definitions
Requires theoretical
grounding
Difficult to measure
Student engagement
Data quality

Conceptual framework
Developed based on:
Qualitative student
interviews
Review of the
theoretical literature
Review of available
measurement tools
Focus on:
Comprehensiveness,
and
Practicality

Developing measures
Twelve measurement instruments used in the survey
newly-developed, from qualitative work
adapted from other sources
unchanged from other sources
Piloting before full administration

Developing measures
Piloting: main problem: length
Reliability: overall good, but issues already identified:
11 of the 12 measures adequately reliable
1 measure: epistemological beliefs: low reliability
additional scale used for Round 2
Administration of measures: online

Longitudinal design
Round 1
6,275
Round 2
3,189
Round 3
…
Data
HESA
matching
Oct
2016
June
2017
Feb-Mar
2018

Example #1
Relating and Structuring (from ILS, Vermunt & Vermetten, 2004, with minor adaptations)
Item M(SD) Factor loading
1 I try to construct an overall picture of a course for myself 3.06(1.22
)
0.47
2 I compare the conclusions drawn in different academic sources. 3.03(1.17
)
0.61
3 I try to see the connection between the topics discussed in different academic
subjects
3.25(1.08
)
0.75
4 I try to discover similarities and differences between theories. 3.04(1.11
)
0.70
5 I try to combine subjects that are dealt with separately into a whole. 3.18(1.16
)
0.56
6 I relate specific facts to the main issue in a chapter or article. 3.16(1.08
0.62

Example #3
Reasoning ability (individual items from ICAR, 2015, own assembly)
Item % Correct Factor loading Item % Correct Factor loading
1 Rotation 1
57.95% 0.57 7
Letters and Numbers
1
52.36% 0.47
2 Rotation 2
46.19% 0.49 8
Letters and Numbers
2
65.60% 0.47
3 Rotation 3
34.84% 0.51 9
Letters and Numbers
3
67.50% 0.53
4 Verbal 1 88.21% 0.46 10 Matrix 1 71.79% 0.46
5 Verbal 2 61.92% 0.36 11 Matrix 2 72.69% 0.37
6 Verbal 3 80.82% 0.47 12 Matrix 3 68.99% 0.45
Final model fit: RMSEA=0.043, CFI=0.944, SRMR=0.028, TLI=0.929, α=0.769

Lessons learned
Theoretical framework essential
Using qualitative work to inform scale development is useful
Adapting measures tricky, and not always successful
Measurement quality an issue: e.g. epistemology
Student participation: important, but not trivial to achieve
Quality of measurement scales fundamental to assessing change

Looking ahead: further analysis
Students’ self-regulatory behaviours, learning patterns,
engagement levels, attitudes towards research, and other
abilities and attitudes may change over time
learning gain

Thank you.
Contact: learning-gain@educ.cam.ac.uk

Learning Gain and Confidence Gain
Through Peer-instruction: the
role of pedagogical design
Fabio R. Aricò
@FabioArico
Open University
Jan 2018

YOUR PRESENTER
Fabio Aricò
Senior Lecturer in Macroeconomics
National Teaching Fellow 2017
School of Economics – University of East Anglia, UK
Research fields
• Higher Education policy and practice (widen. access, satisfaction)
• Technology Enhanced Learning
• Self-Assessment and Academic Self-Efficacy
Twitter: @FabioArico
20

ACKNOWLEDGEMENTS
HEFCE Piloting and Evaluating Measures of Learning Gain
UEA Students, Alumni, and Research Assistants
HEA – Teaching Development Grant Scheme
21

OUTLINE
Part 1 Introduction to main concepts
• Peer-instruction, Self-efficacy & Self-assessment, Learning Gain.
Part 2 Description of my active learning pedagogy
Introduction to my research questions
• Operationalising learning/confidence gain
Part 3 Empirical methodology and results
• Regression outputs and discussion.
22

1. Introduction to
main concepts
23

FLIPPED CLASS and PEER-INSTRUCTION
• Flipped classroom & Peer-Instruction  pre-reading + student interaction
 Mazur (1997)
 Henderson and Dancy (2009)
 well-developed research in Physics and STEM.
• Learning analytics for Peer-Instruction
 Learning gain: Mazour Group - Bates & Galloway (2012)
 Student satisfaction: Hernandez Nanclares & Cerezo Menendez (2014).
• There is not much literature on the links with self-assessment skills
Open field, with many unanswered questions
 e.g. role of demographics, language, previous background
 Pedagogically: self-assessment blends with flipping and Peer-instruction.
24

SELF-EFFICACY and SELF-ASSESSMENT
Academic Self-Efficacy = confidence at performing academic
tasks and/or attaining academic goals.
Bandura (1977) 1. Mastery of experiences
2. Vicarious experiences
3. Verbal persuasion
4. Environment and settings
See also: Pajares (1996) and Ritchie (2015).
Idea: Students should develop their self-efficacy to master their
learning experience. Measure learning gain along with
increased self-efficacy: ‘confidence gain’.
25

FRAMEWORK
26
Active Learning
Peer-instruction Self-Assessment
Learning Gain Confidence Gain
TEL
learning analytics

2. A description of my
active learning pedagogy
Research questions
27

ACTIVE LEARNING ENVIRONMENT
Introductory Macroeconomics (2015-16 & 2016-17)
• year-long module (compulsory 1st year)
• 250 students (about 250 over past 2 years)
• 22 lectures (2hrs per week)
• 8 seminars (every second week)
• 8 workshops (every second week)
Students endowed with individual Audience Response Systems (clickers)
 continuous data collection facilitated by technology;
 comprehensive ethical approval obtained beforehand.
28

WORKSHOPS – teaching algorithm
29
Round 1
- formative question
- 4 choices
- no information
- no answer
Self-Assessment 1
- confidence question
- 4 level Likert-scale
- information shared
Peer-Instruction
- students talk
- compare answers
- explain each other
Round 2
- Identical to R1
- correct answer
Self-Assessment 2

RESEARCH QUESTIONS
30
1. Is the pedagogy developing good self-assessment skills?
Are students self-assessing correctly over Round 1/2?
2. Is peer-instruction able to generate learning/confidence gain?
How does gain relate to initial knowledge/confidence (Round 1)?
3. Is learning gain associated to a confidence gain?
Is this association affected by the structure of the teaching algorithm?
2016 Vicarious of Experience Scenario (VES) only
2017 Mastery of Experience Scenario (MES)
contrasted with VES (4 sessions each).

WORKSHOPS – contrast 2 teaching algorithms
31
Round 1
- 4 choices
- no information
- no answer
Self-Assessment 1
Peer-Instruction
- students talk
- compare answers
- explain each other
Round 2
- Identical to R1
- correct answer
Self-Assessment 2
VES MES

3. Empirical methodology
and Results
32

DATASETS AND CODING
Student Q1 Q2 Q3 … 
1 0 1 1
2 1 0 0
3 1 1 …
…
 performance per question
confidence by question
performanceperstudent
confidencebystudent
33
Formative questions
1 = correct
0 = incorrect
Confidence questions
1 = strongly/agree
0 = strongly/disagree

OPERATIONALISING TWO GAINS
For each 1st and 2nd response to formative assessment questions:
% correct R2  % correct R1
Normalised Learning Gain (NLG) =
100%  % correct R1
For each 1st and 2nd response to self-assessment questions:
% confident R2  % confident R1
Normalised Confidence Gain (NCG) =
100%  % confident R1
34

CHANGE IN CONFIDENCE: rough descriptives
35
0%
20%
40%
60%
80%
100%
1 3 5 7 2 4 6 8
Week
2016: only VES
C.score
K.score
0%
20%
40%
60%
80%
100%
1 3 5 7 2 4 6 8
Week
2017: VES & MES
C.score
K.score
1st
2nd
MESVES
Average confidence levels per session
 Confidence gain is always positive
 Confidence gain higher with VES (vicarious) and lower with MES (mastery)

EMPIRICAL METHODOLOGY
Regression analysis at class-level N=140 (2016) and N=136 (2017)
Dependent variables: %confident responses, NLG, NCG
Independent variables: %correct responses, NLG
Workshop Group dummy for 2
Workshop Session dummy for 8
VES/MES Scenario dummy for 2
Functional form: checking for polynomial forms
Robustness checks: Robust-regression with White-correction
(ML estimation of standard errors)
36

37
% confident
Round 1
% correct Round 1
RESULT – Self-assessment Skills Round 1
2016 VES =0.429*** R²=0.56
0.33
0.44
2017 VES =0.367*** R²=0.59
2017 MES =0.09***

38
% confident
Round 2
% correct Round 2
RESULT – Self-assessment Skills Round 2
2016 VES =0.282*** R²=0.26
0.57
0.60
2017 VES =0.322*** R²=0.64
2017 MES =1.44***

RESULT – Normalised Gains in 2016
39
NLG
% Correct/Confident
in Round 1
2016 0.37 0.48
0.85
0.52
NCG
NLG R²=0.39
NCG R²=0.19

RESULT – Normalised Gains in 2017
40
NLG
% Correct/Confident
in Round 1
0.40 0.42
0.67
NCG
NLG R²=0.21
NCG R²=0.42
=0.132**

RESULT – Learning Gain & Confidence Gain
41
NCG
NLG
2016 R²=0.16
MES =0.125**
-1
0.31
0.28
2017 VES =0.19*** R²=0.39

SUMMARY of RESULTS
• In both Round 1&2, confidence levels are lower under MES.
However, performance and confidence are consistently positively correlated
 students self-assess correctly, irrespectively of MES/VES scenario.
• Peer-instruction generates higher learning/confidence gain when
knowledge/confidence in the classroom is neither too high or too low
 ‘sweet-spot’ pattern does not depend on MES/VES scenario.
• Under VES scenario confidence gain is lower, compared to MES scenario.
However, confidence gain is consistently positively correlated to learning gain
 students develop more confidence as they learn, irrespectively
of MES/VES scenario.
The pedagogy appears to be robust to teacher intervention  Active Learning!
42

43
F.Arico@uea.ac.uk
@FabioArico
“Promoting Active Learning Through Peer-Instruction and Self-
Assessment: A Toolkit to Design, Support and Evaluate Teaching”,
Educational Developments, SEDA, 17.1, 15-18.
STAY IN TOUCH!

22/01/2018 45
Self-Efficacy
“Confidence is the Pivot to
Success”
Techsavvywomen.net
Dr Paul McDermott, School of
Pharmaceutical Sciences, UEA
Dr Robert Jenkins, Norwich
Business School, UEA

22/01/2018 HEA Annual Conference 2017 46
Warm Up
GMYMTNOLMERERO
LONG TERM MEMORY

Self-Assessment
22/01/2018
Closely aligned to the construct of self-efficacy
Self assessment can be defined as information about the learners provided
by the learners themselves4.
Good self assessment will provide ACCURATE data about:
• The learner’s abilities
• The progress they think they are making
• What they think they can or cannot do with the material they have
covered in the course
The greater a learner’s self assessment ability in relation to a task the more
likely it is they will develop a feeling of mastery over that task (self-efficacy).
(4). Blanche, P., & Merino, B. Language Learning, 1989, 39, 313-340
“I can’t”
“I know”
- Suzanne Fergus

Self-Assessment
22/01/2018
Closely aligned to the construct of self-efficacy
There have been many reported studies in the education research literature
for the measurement of learners’ self-assessment.
The most popular
methodology employed
across these studies is a
multiple choice quiz
followed by a confidence
tier questionnaire.
Many of which display the
Dunning-Kruger effect
(5). Ehrlinger, J., et al, Organizational Behaviour and Human Decision Processes., 2008, 105, p98-121.

Self-Assessment
22/01/2018
METRICS have come into sharp focus as we move into a new REF-TEF world
Our Hypotheses:
1). Previous research methodology doesn’t necessarily account for the
differing motivations a student will experience when answering conceptual
questions and then reporting their own confidence in a separate process.
2). If we were able to bring these two processes into one function we would
have a method that allowed us to see through the fog of subjectivity and
irrational optimism.

Self-Assessment
22/01/2018
METRICS have come into sharp focus as we move into a new REF-TEF
world
Our Methodology (Taken from TBL):
This format has been designed to allow
For partial credit when marking MCQ’s
Students distribute 4 marks across the answer
options in a strategic manner to gain the best
possible score.
To mark the answer grid an acetate is placed
over the answer sheet and the values in the
clear boxes are written in the points column.
Teambasedlearning.org

Self-Assessment
22/01/2018
world
Data analysis:
We assigned a code to each answer strategy and ranked each strategy
in order of increasing entropy
S0 - the strategy profile that student has entered is 0,0,0,0 (invalid response)
S1 - the strategy profile that student has entered is 1,1,1,1
S2a - the strategy profile that student has entered is 2,2,0,0
S2b - the strategy profile that student has entered is 2,1,1,0

Self-Assessment
22/01/2018
world
Data analysis:
• 4 points on one answer (S4) corresponds to the entropy minima
• 1 mark in each option (S1) is the entropy maxima
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Q1 Q2 Q3 Q4
Strategies
used (%)
4003-summative Strategies by Student
Quartile
S1 S2a S2b S3 S4
Correlation r = -0.889
P value = 0.000
r2 = 0.362
Entropy Index:
Where:
k = number of categories
fi = relative frequency of class i
 





 k
i
f
i
k
i
f
i
i
i
fk
f
e
1
1
1
1

Self-Assessment
22/01/2018
world
Data analysis:
This pattern held up across a range of different assessments
(both summative and formative)
Name Correlation Pvalue Rsquared Intercept Questions Students Data collection
Formative
course
test
-0.639 0.000 0.314 0.556 15 83 April 2016
Formative
course
test
-0.480 0.000 0.185 0.572 17 101 December 2016
Summativ
e course
test
-0.889 0.000 0.362 0.698 18 114 January 2017
Aromatic
formative
-0.871 0.000 0.302 0.750 10 102 January 2017

Self-Assessment
22/01/2018
world
Data analysis:
This pattern held up across a range of different assessments (both
Summative and formative)
Name Correlation Pvalue Rsquared Intercept Questions Students Data
Collection
Endocrinology 1 -0.997 0.000 0.371 0.656 10 98 November
2015
2015
2016
2016

Self-Assessment
22/01/2018
world
Data analysis:
This pattern held up across a range of different assessments (both
Summative and formative)
Name Correlation Pvalue Rsquared Intercept Questions Students
Bipolar
affective
disorder
-0.717 0.000 0.192 0.592 18 82 February
2016
Anxiety and
depression
-0.787 0.000 0.155 0.605 18 90 February
15/16
Schizophrenia -0.357 0.051 0.04 0.494 17 83 March 2016
Depression
and Anxietiy
-0.434 0.000 0.189 0.701 20 119 February
2017
Bipolar -0.697 0.000 0.242 0.735 20 119 February
2017
Schizophrenia -1.359 0.000 0.264 0.813 9 77 March 2017

Self-Assessment
22/01/2018
world
The students do not know they are giving this self efficacy data
They have a different “motivation” as they are focussed on an effort to
maximise their grade.

Self-Assessment
22/01/2018
world
• This data does not (necessarily) follow a Dunning Kruger pattern
• Very clearly we can see that lower performing students use higher
entropy strategies which we can attribute to lower confidence in their
answers
• Higher performing students use lower entropy strategies which we can
attribute to higher confidence in their answers
• The significant correlation values show that this is accurate self
assessment data with the potential for a number of applications
Data Analysis:

Self-Assessment
22/01/2018
world
Results
97.22%
1 2 3 4 5 6 7 8 9
4
4 4 4
4 4 4
4 4
10 11 12 13 14 15 16 17 18
4 4
4 4 2 4
4 4
4 2

Self-Assessment
22/01/2018
world
Results
72.22%
1 2 3 4 5 6 7 8 9
4
4 2 4 4
2 4 4 4
4
10 11 12 13 14 15 16 17 18
1 1
2 4 2 2 4
4 2 3 4
4 1 2

Self-Assessment
22/01/2018
world
Results
29.17%
1 2 3 4 5 6 7 8 9
1 4 4
2 2 3 2
2 3 2 1 1
4 4 1
10 11 12 13 14 15 16 17 18
2 1 2 1 1
3 1 2 1 1
4 1 1 2 2 3 1
3 2 1 1

Self-Assessment
22/01/2018
world
Results
10 11 12 13 14 15 16 17 18
1
4 4 1
4 4 4 4 4 1
4 1
1 2 3 4 5 6 7 8 9
4
4 4 4 4
4 4 4
4
29.17%

Confidence Measure
22/01/2018
world
Question number 1 2 3 4 5 Mark Grade
Student ID Answer Key X X X X X
Ms A. N. Nonymous A 2 1
B 4 1 2 1 1 11 55.00
C 3 2 1 1 0.00
D 1 0.00
4 1.5 1 0.666667 0.25 1.48 55.00
=MAX(C3:C6)/COUNTIF(C3:C6,">0")
=AVERAGE
Average = 55.4%

Confidence Measure
22/01/2018
world
Average = 64.2% Average = 66.7% Average = 61.7%
2016-17 (SUMMATIVE):
2015-16 (FORMATIVE):
Average = 51.2% Average = 52.3% Average = 45.4%

Confidence Measure
22/01/2018
A bit more…
Calculated
  
yx
ii
i
ss
yyxx
c

 for each student
y
x
ci positive
ci positive
ci negative
ci negative
Max = 3.86
Min = −1.94

Confidence Measure
22/01/2018
A bit more…
Calculated
  
yx
ii
i
ss
yyxx
c

y
x
Above average performance
Below average performance

Confidence Measure
22/01/2018
A bit more…
Calculated
  
yx
ii
i
ss
yyxx
c

y
x

Confidence Measure
22/01/2018
Interpretation/student feedback loop.
  
yx
ii
i
ss
yyxx
c

 Max = 3.86 Min = −1.94
Close to zero indicates close to mean results in confidence or grade or both.
Green, positive: above average in both, closer to 4 more confident high marks.
Green, negative: above average in grade, below average confident
closer to -2 high performing student, low confidence.
Red, negative: below average in both, closer to 4 low confidence, low marks.
Red, positive: below average in grade, above average confident
closer to -2 high confidence, low scores.
Inform student of this, their score and means give reflective opportunity.

Self-Assessment
22/01/2018
world
• This gives us a self-efficacy metric that instantly correlates to actual
student performance
• This methodology can be seamlessly incorporated into a vast range of
teaching and assessment settings and the data is quickly and easily
accessed
• We propose that this methodology allows us to gain insightful self-
assessment data through the direct measurement of subjective
confidence (2nd order construct)
Conclusion:

22/01/2018 TBL Conference 2015 69
THANK
YOU

Using data to increase learning gains and teaching excellence
Milton Keynes
Measuring employability gain in Higher Education.
A case study measuring the impact of R2 Strengths on final years’ employability
Dr Heike Behle, LEGACY, Warwick Institute for Employment Research (IER),
University of Warwick
22 January 2018

@HeikeBehle
Measuring employability gain
• Employability NOT employment
• Student Outcomes and Learning Gain (TEF)
“acquisition of attributes such as lifelong learning skills and others that
allow a graduate to make a strong contribution to society, economy and
the environment”
“progression to further study, acquisition of knowledge, skills and
attributes necessary to compete for a graduate level job that requires the
high level of skills arising from higher education” (DfE, 2017, p. 24)
• Employability – ability to find, keep and progress in
graduate employment
• Requirements for a framework
- Holistic
- Sustainable
- Cover HEIs interventions
- Identify limits

@HeikeBehle
The employability framework
Individual Factors
(demographics, health, skills,
competences, knowledge,
personality)
Enabling Support Systems
(public and private labour market
intermediaries and support agencies)
Individual Circumstances
(household, work culture, resources,
networks)
Labour Market
(demand on the local, national and
international labour markets,
operations and norms, regulations
and institutional factors)
Employability

@HeikeBehle
Interventions to enhance
students’ employability?
• Skills, knowledge, credentials
• International experiences
• Work experiences (internship,
sandwich course, visits to work
places)
• Extra-curricular activities
• Volunteering
• Reputation of HEI
• Careers guidance, training for
job search, R 2 Strength

@HeikeBehle
Realise 2 Strength (R 2 Strength)
 Dynamic
assessment
 Recognises the
role of changing
contexts in the use
of strengths
 Holistic and
integrated
understanding of
an individual
profile
 Different to skills
and knowledge
• Strengths the student enjoys,
is good at and has the
opportunity to use
Realised Strengths
• Strengths the student enjoys
and is good at but doesn’t
use often
Unrealised
Strengths
• Activities the student is
neither good at nor enjoys
Weaknesses
•Strengths which, whilst the
student is proficient in, they are
not energised by them
Learned
Behaviours

Enabling Support
Factors
Individual Circumstances
Individual Factors
Local & National Labour Market Factors
Employability
Degree
Life/ Work
experience
Extra-curricular
communities of
practice
Awareness
Transferable Skills
Personal Qualities,
Efficacy, Self-beliefs
Meta-Cognition
Skilful Practice
Subject
Understanding

@HeikeBehle
Pilot Study: R2 Strength
Two Research Questions
 Does R2Strengths
impact on
employability of
students and
graduates?
 How can we
document the
potential impact of R2
Strengths?
Employability
Method
Impact of
R2
Strengths

@HeikeBehle
Mixed Methods Design
Survey
6 Russell group universities
Each University:
96 final year Home and EU
Undergraduates (realised
sample 524 before, 400 after)
3 groups of students:
Group R2
Strengths
Profile
One-to-One
1 X X
2 X
3 Control group
Qualitative Interviews
36 qualitative semi-
structured interviews with
group 1 participants Self-
selecting participants
Common interview guide
Timeframe of interviews:
March – May 2017
35-50 minutes in length

@HeikeBehle
R2 Strengths Findings: Survey
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Before
After
Before
After
Before
After
Group1Group2Group3
I am aware of my strengths
Strongly disagree Disagree Slightly disagree Neither agree or disagree Slightly agree Agree Strongly agree
Source: R2 Strengths Survey (respondents to both surveys only, n = 400)

R2 Strengths Findings:
Qualitative interviews
“I thought I was someone who knew what I was good at and always thought I knew
I’m good at this or that but I don’t think I reflected on a wider sense. It was more on
specific skills of what I can do. That’s not how strengths were articulated in the project.
It was more of a broader thing.” [Alexa]
Tangible impact
“It has helped me with attaching words to the strengths I have without being restricted
by my own vocabulary. [Lucas]”
Intangible impact
More confident. Definitely. They're more quantified, and they're more visible,
realisable, tangible. Especially my persistence, because I am very persistent. And my
work ethic. [Philip]
Confidence
It was more like "these actually are my strengths." Talking about it made me seem
more engaged […] it legitimised it. [Philip]

@HeikeBehle
Summary
• Holistic Employability Framework
• Employability is not the same as employment rates. “Employability = ability
to find, keep and progress in graduate employment”
• TEF mainly focusses on employability skills, however, a wider focus is
needed.
• Case Study: R2 Strengths.
• Positive employability impacts for students, both in tangible terms
(application processes) but also in intangible terms (increased self-
confidence and self-efficacy) could be evidenced.
• R2 Strengths made a considerable difference to many students in
encouraging awareness of their strengths, thus helping them to articulate
these with increased self-confidence.
• The quantitative survey did not capture evidence of students’ development
in the way the qualitative interviews did. Students were less likely to agree
that they had increased their knowledge, use and ability to articulate their
strengths and their career-readiness over the life of the project

@HeikeBehle
Final remarks
Follow us on @LegacyLGproject ; @HeikeBehle
Heike.Behle@warwick.ac.uk
For further information www.legacy.ac.uk
The LEGACY employability briefing can be downloaded here www.legacy.ac.uk
Click: Resources
Or here: www2.warwick.ac.uk/services/aro/dar/quality/legacy/hp-
contents/employability_behle_h_uow_2016.pdf
Report in preparation
Wilson, Behle and Tassinari et al., (2018) R2 Strengths - Measuring employability gain

82
Copyright © The Careers Group, University of London
Careers Registration
Measuring work-readiness
learning gain
Fiona Cobb
Careers Registration Research Project Coordinator
fiona.cobb@careers.lon.ac.uk
@Careersgroup
Photo by Tom Barrett on Unsplash
Using data to increase learning gains and teaching excellence

83
Aims of this session
 Discuss findings from the Careers
Registration learning gain research project
 Explore Careers Registration as an evidence
based approach for employability support
in Higher Education.
 Consider wider implications of work
readiness learning gain for HE strategic
planning, policy and practice

84
Defining Learning Gain
“the 'distance travelled', or the improvement
in knowledge, skills, work-readiness and
personal development demonstrated by
students at two points in time.”
[HEFCE, 2015]

85
Measuring Learning Gain
“an attempt to measure the improvement in
knowledge, skills, work-readiness and
personal development made by students
during their time spent in higher education”
[HEFCE, 2017]

86
Where are you right
now?
Image: CC0 Public Domain Free for commercial use

87
 3-year project investigating
Careers Registration (CR) as
a measure of learning gain
in work readiness
 Led by The Careers Group,
University of London
Careers Registration
Learning Gain project

88
Project members
Aberystwyth University
City, University of London
Goldsmiths, University of London
King’s College London
Lancaster University
Liverpool John Moores University
Queen Mary, University of London
Royal Veterinary College
School of Oriental and African Studies
St George’s University of London
St Mary’s University
Ulster University
University College London
University of Bristol
University of Edinburgh
University of Exeter
 16 Institutions
 13 fully implemented CR to
date
 2 partially implemented by
Sept 17
 Representation from England,
Scotland, Northern Ireland &
Wales

89
What is Careers Registration?
 Employability-related questions included in
student registration each year — Current &
Connected
 Completed by all new and all re-enrolling
students — Comprehensive & Consistent
 Both cross-sectional and longitudinal data
— Comparable

90
Core CR questions
Career readiness
- Self-assessed readiness to engage
with career management
- Select from 12 statements, e.g.
 I am not ready to start thinking
about my career yet — (Decide)
 I have a career in mind & intend to
gain relevant work experience —
(Plan)
 I am ready to apply for graduate
level / professional opportunities
— (Compete)
 I have a job, further study or my
own business plan confirmed —
(Sorted)
Employability experience
- Self-reported experience
- Choose from list of activities, e.g.
 a placement year
 a summer internship
 volunteering
 position of responsibility in a club
or society
 full time work prior to my course
 self-employment / running my own
business
 no work experience to date
Other questions on sector preference & experience, future plans & enterprise

91
Planning for success
Most important factors in
determining if graduates were in
professional or managerial roles or
non professional roles:
1. knowing exactly what they
wanted to do or having a
good idea about types of jobs
and careers upon completing
university.
2. Having a very targeted
approach to job applications
3. Having undertaken unpaid
work experience
 7,500 students drawn from 27
institutions.
 Study combined data from the
6 month DLHE survey with
follow up data from a survey
conducted two years later
 Aimed to understand the
relative importance of different
behaviours, characteristics and
factors in determining graduate
outcomes

92
Project aims
To assess whether a small
number of questions in student
enrolment can:
 track development of
student employability during
their time in HE (distance
travelled/learning gain)
 evaluate the effectiveness of
employability strategies and
interventions.

93
Research priorities
Analysis
•Cross-sectional
‘Snapshot’ CR
•Longitudinal
‘tracked’ CR
Comparison
•DLHE
•Employability
health check
•HEAR
•NSS
•Self Efficacy (CDSE
tests)
•Individual –
Academic
records/retention
Impact
•Impact evaluation
of existing
implementations
of careers
registration
Application
•Widening
participation
•Interventions
•Consultants use of
data
•Engagement:
Careers services,
Academics,
Students
Recommendations

94
Careers Registration Learning
Gain Project
Year two findings

95
Year Two Analytics
Measure What have we focused on?
Career thinking response ratios year of study, discipline groups, Socio-
demographics (undergraduates only)
Compete Growth Dif. between yr 1-2, yr 2-3 of
Undergraduate programmes, by
discipline groups, WP characteristics
Career thinking movement Between statements, and within and
between categories
Benchmarking
(in progress)
Constructing Career thinking
benchmarks by discipline group and WP
characteristics

96
Compete Growth

97
Compete growth
 How many students are in the ‘compete’
category of career thinking by year 3 of
study vs year 1 of study…
 Data collected from 2 institutions who
implemented CR in 2014 -15, and have
collected data through to 2016/17
 16,587 UG student responses analysed

98
Undergraduate Compete growth
tracking
- Year 1 compete stage career thinking 1.52%
- Year 3 compete stage career thinking 19.8%
- Increase of 18.28% in students career thinking
patterns overall as students progress through
their course of study.

99
2014/5 and 2016/7 – Tracking Undergraduate students
(Year 1 & 3)

100
Subject area
Non Science subjects
- Year 1: .96% compete
- Year 3: 24.10% compete
- Compete growth: 23.14 %
Science subjects
- Compete growth: 13.34 %

101
Subject area compete growth

102
Fee status
Compete growth
Overseas students 26.1%
Home/EU students 17.36%

103
Mode of study
Full time
- Compete growth: 18.74%
Part time
- Compete growth: 3.94% %
More part time students in ‘sorted’
category to begin with….

104
Tracking Compete growth – Part Time (Years 1 & 3 )

105
Compete growth: WP
characteristics
Category Split
Disability Disability/no disability
Gender Male/female (other)
Mature student Under 21/ mature student
Polar 3 Yes/no
(deprivation categories 4+5)
Ethnicity Black, Asian, White, Other

106
WP Category Findings
Disability More Year 3 students with a disability in the compete stage of
their career thinking, 22.10% compared to 19.62% Year 3
students with no disability.
Gender 1.14% difference in compete growth based on gender
Mature student Bigger increase in growth for U21’s (more Mature students in
compete to start with)
Polar 3 low participation neighbourhood background less likely to be
in the compete stage in their career thinking. 12.39% low
participation, 18.8% rest of cohort.
Ethnicity Asian students had the highest growth in compete responses
by year 3 of study (22% increase)

107
Tracked individual statement selection for 2015/16 to 2016/17
From year 1 – year 2, and year 2 – year 3
Changes categorised as ‘positive’, ‘negative’, or ‘no movement’

108
Career thinking movement by Subject Area

109
WP Statement movement
- Minimal differences between year 1 -2 for
all WP characteristics analysed
- Higher % positive movement between
years 2 -3 for:
- Mature students (34% vs 24% Undergraduate)
- Polar 3 (low part. 21%, high participation, 26%)

110
Next steps for analytics
- Interactions and correlations between
variables
- Multinomial logistic regression modelling
- DLHE comparison
- Careers staff readiness to work with the
data survey
Photo by Rodion Kutsaev on Unsplash

111
Careers registration in action
Careers registration data is being used to:
 inform strategic planning and academic
department engagement
 identify individual student careers and support
needs
 promote the services offered by careers
departments to their student bodies.

112
Strategic uses of the data I
Academic departmental
engagement
• partnership agreements
drawn up on the basis
of CR data
• CR data used by careers
consultants to inform
strategy for and
planning of careers
activities
• data provided to
personal tutors for
careers discussions with
students
Strategic planning
• data used in QAA
reporting
• data informing
institutional level policy
and KPIs
Employer engagement
• analysis of
employer/sector data
to identify gaps in
employer engagement
• data used to persuade
new/existing employers
of student interest in
their sectors

113
Strategic uses of the data II
Marketing and
communications
• targeted event
marketing to
students based on
interests expressed
in CR surveys
Final year support
programmes
• targeted event for
final year students
who were identified
as not knowing
where to begin with
their career planning
Widening
participation
initiatives:
• using CR data to
promote relevant
events and
opportunities to WP
students, (data held
in student records
system, responses
can be linked with
WP markers)

114
CR and The TEF
Metrics based on DLHE
‘Employment or further study’ metric
Benchmark factors: subject of study, entry
qualifications, age on entry, ethnicity, sex, level
of study
…Graduate Outcomes…15 months to wait…
Careers registration offers year on year
tracking, data can be slices by benchmark
factors

115
Fundamentals of Careers Registration
Webinar series
Date Webinar Title
21/9/2017
Designing, Planning and Delivering Careers Registration in record time
29/09/2017 Careers registration for postgraduates
31/10/2017 Using registration data in employer engagement
16/11/2017 Transforming careers practitioners professional practice with Careers Registration Data
30/11/2017
The power of dashboards: visualising careers registration data in your careers service
30/01/2018 Tracking the student journey: linking Careers Registration with an exit survey
06/02/2018 Careers Registration, DLHE and Graduate Outcomes
12/03/2018 Work readiness learning gain: what do the Careers registration findings tell us?
https://www.eventbrite.co.uk/o/careers-staff-development-8361679885

116
Project website https://goo.gl/Ami6af
Project Contact Fiona Cobb Fiona.cobb@careers.lon.ac.uk
Twitter @FiCobb @careersgroup #learninggain

Big thanks
Natalie Eggleston Rebecca FergusonMaryja Strickland Jekaterina Rogaten Rhona Sharpe
University of Surrey

Open Discussion
 Anyone has a question, idea, point to raise, concern, or anything else to share?

Open Discussion
 Should we as universities buy-in to the narrative of learning gains?
Are our students customers?
Can we measure learning?
 Is there a “best-way” to measure learning gains?
 Is there a “best-way” to measuring learning gains across disciplines?
 Is there a “best-way” to measuring learning gains across institutions?
 What is the role of data protection/GDPR/ethics?

22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence” Afternoon Presentations

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence” Afternoon Presentations

Ähnlich wie 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence” Afternoon Presentations (20)

Mehr von Bart Rienties

Mehr von Bart Rienties (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence” Afternoon Presentations

Hinweis der Redaktion