Assessment for Effectiveness and Equity: Lessons from a Longitudinal Study By Caine Rolleston Presented at REAL Centre One Day Conference - "Learning from learning assessments to leave no one behind" REAL, University of Cambridge June 15, 2016

1. Assessment for
Effectiveness and Equity:
Lessons from a Longitudinal
Study
Caine Rolleston
REAL, Cambridge
June 15, 2016

2. • Extent of ‘learning crisis’ becoming clearer – e.g. from citizen led
assessments UWEZO etc.
• Cross-sectional data & benchmarking key altho still absent in many
contexts
• But other key questions (especially effectiveness/equity) require more
sophisticated data, including longitudinal or combined household and
school data:
• Which schools/school systems are more effective (add more value)
& why?
• To what extent a crisis of school quality? (not only learning)
• When do gaps arise/develop during the life-course?
• What are the causes of/remedies for poor attainment?
• Cost-effectiveness, intervention choices
• Some potentially require combined longitudinal HH plus longitudinal
school data
– E.g. Are schools equally effective for more and less advantaged
pupils?
ASSESSMENT IN DEVELOPING COUNTRIES

3. Young Lives longitudinal survey of children, households &
communities every 3 years since 2002
• 12,000 index children in two cohorts (now aged 13
& 19)
• Ethiopia, India, Peru, Vietnam
• 20 sentinel sites in each country
• Tested in maths at each round with common items
• Primary school surveys implemented since 2010
• Secondary school surveys from 2016
Allows comparison of
• Learning levels
• Learning trajectories
• Change over time between cohorts
YOUNG LIVES STUDY

4. YOUNG LIVES STUDY

6. BENCHMARKING COMPETENCY LEVELS (ETHIOPIA)
Competency level Number of Pupils Percentage of
Pupils
Cumulative
Percentage
0 (Below Level 1) 853 8.66 8.66
1 (Early Foundational) (~ G1) 2,121 21.54 30.21
2 (Foundational) (~G2-3) 5,152 52.33 82.53
3 (Emerging) (~G3-4) 1,473 14.96 97.49
4 (Grade level) (~G4-5) 247 2.51 100.00
Total 9,846 100.00
0
20406080
100
percent
Rural Urban
5 4 5 4
Level 0 Level 1
Level 2 Level 3
Level 4

7. LONGITUDINAL STUDY WITHIN A STUDY
.0005
.001
.0015
.002
.0025
Density
200 400 600 800
Score
Test 1 Test 2
Maths Scores at First and Second Round Tests
• Mean Test 1 = 500, SD= 100. Mean Test 2=530 (gain 0.3 SD)

8. MEASURING PROGRESS OVER THE SCHOOL YEAR (ETHIOPIA)
INTERVALE-SCALE METRICS
Mathematics Reading Comprehension
October
2012
May
2013
Gain October
2012
May
2013
Gain
Mean 500.0 530.0 30.0 500.0 530.6 30.6
Gender Boy 502.1 532.5 30.4 498.0 527.5 29.5
Girl 497.9 527.6 29.7 501.9 533.6 31.7
Difference 4.2 4.9 0.7 -3.9 -6.1 -2.2
Location Urban 517.6 548.5 30.9 521.1 551.1 30.0
Rural 456.4 481.2 24.8 447.3 476.6 29.3
Difference 61.2 67.3 6.1 73.8 74.5 0.7

9. PREDICTORS OF ATTAINMENT AND PROGRESS OVER TIME (ETHIOPIA)
VARIABLES Maths T1 Maths T2 Reading T1 Reading T2
Girl -10.2031 *** -4.4330 *** 0.1650 4.8107 ***
Has 3+ meals per day 18.2975 *** 6.6820 *** 12.6032 *** 2.1339
PCA pupil durable assets 4.5648 *** 1.7210 *** 4.9427 *** 0.8929 *
% days absence W1-W2 -3.3064 *** -1.8387 *** -3.1255 *** -1.4661 ***
Orphan (single or double) 2.8872 -3.2279 * 2.6308 -3.3137 **
No-one in household literate -9.1339 ** -5.7685 * -14.3049 *** -1.8481
Attended pre-school 2.6854 -0.3207 6.3172 *** 3.3394 **
Ever repeated a grade -39.6920 *** -5.0135 *** -38.7614 *** -4.2894 ***
Ever dropped-out -6.1516 ** -2.1293 -13.4413 *** -3.7113 **
Reads books at home 17.5734 *** 6.2704 *** 17.3460 *** 3.6314 *
Child learns in home language 3.8083 -2.2792 14.5838 *** 5.4488 *
Pastoralist -24.1437 *** -4.3839 -37.9640 *** -8.0455 **
Pupil spends time on paid work -10.4830 *** 0.2447 -12.1998 *** -1.0128

10. SCHOOL-LEVEL VALUE-ADDED (ETHIOPIA)-100
-50
0
50
100
0 20 40 60 80 100
School Rank
Urban Rural

11. DUAL COHORT STUDY WITH A 7 YEAR INTERVAL:
IMPROVING TEST SCORES, MIXED PATTERNS OF EQUITY
Peru Vietnam
Test score gains with/without equity improvement
Children aged 12 in 2006 and 2013

12. DECLINING TEST SCORES WITH WIDENING
INEQUALITY
India (AP) Ethiopia
Children aged 12 in 2006 and 2013

13. 0
20406080
0 20 40 60 80 100
CDA-Q Score R2 %
Ethiopia Peru
India Vietnam
WIDENING GAPS ARE DRIVEN BY DIFFERENCES IN LEARNING
PROGRESS OVER TIME BETWEEN SYSTEMS: AGE 5 TO 8

14. 0.2.4.6
01234
-2 0 2 4 -2 0 2 4
Ethiopia Vietnam
Most Poor Least Poor
Math scores (2009)
Graphs by country
Maths scores
Learning divergence by wealth groups
• Steeper gains by prior score in Ethiopia, broadly similar gaps by wealth
• Pupils typically make more gains over time in Vietnam

15. • Distribution of school quality accentuates disadvantage
• E.g. differences in school quality explain more of the differences
in test scores in India and Peru than Vietnam (schools are more
heterogeneous)
• Differential effectiveness may mean triple disadvantage (two
ways in which schools widen gaps)
• Reasons could include elitism, curriculum, language of
instruction, discrimination
• A recent study Glewwe, Krutikova & Rolleston (EDCC,
forthcoming) compares Vietnam and Peru – using both
longitudinal household and school data
EQUITY ISSUES

16. -0.18
0.41*
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
Vietnam Peru
Proportion of 1 SD of maths test score…
Difference in test score gains by increasing
school quality by 1SD between richest 40%
and poorest 60% of pupils
In Vietnam, schools equally effective in
teaching Maths
In Peru, schools significantly less effective at
teaching children from disadvantaged
backgrounds
WITHIN THE SAME SCHOOL, DISADVANTAGED PUPILS
MAKE LESS PROGRESS IN PERU, BUT NOT IN VIETNAM

17. www.younglives.org.uk
c.rolleston@ucl.ac.uk
FINDING OUT MORE