Forecasting a school's future performance

1. Schoolstradamus
Forecasting a school’s future
performance
Eric Smith
Insight Data Science

2. As a kid, location matters

3. How do you decide which high school to go
to?

4. How do you decide which high school to go
to?
• You can compare information about the current performance of
different high schools

5. How do you decide which high school to go
to?
• You can compare information about the current performance of
different high schools
• But you’re going to be in high school for 4 years: can you compare
the future performance of high schools?

6. (demo)
schoolstradamus.com

7. Auto-regressive model outperforms
historical average (2007 to 2014)
Statistics
RMS error
(my model)
RMS error
(historical average)
• Post-secondary education rate 13% vs. 14%
• Regents exams pass rate 5% vs. 7%
• Student dropout rate 4% vs. 6%

8. Auto-regressive model outperforms
historical average (2007 to 2014)
• Compute root-mean-square (RMS) error from 10-fold cross validation
Statistics
RMS error
(my model)
RMS error
(historical average)
• Post-secondary education rate 13% vs. 14%
• Regents exams pass rate 5% vs. 7%
• Student dropout rate 4% vs. 6%

9. Auto-regressive model outperforms
historical average (2007 to 2014)
• Compute root-mean-square (RMS) error from 10-fold cross validation
Statistics
RMS error
(my model)
RMS error
(historical average)
• Post-secondary education rate 13% vs. 14%
• Regents exams pass rate 5% vs. 7%
• Student dropout rate 4% vs. 6%

10. Auto-regressive model outperforms
historical average (2007 to 2014)
Statistics
RMS error
(my model)
RMS error
(historical average)
• Post-secondary education rate 13% vs. 14%
• Regents exams pass rate 5% vs. 7%
• Student dropout rate 4% vs. 6%
• Compute root-mean-square (RMS) error from 10-fold cross validation

11. Correlations amongpercentage
changes in features over time,
2007 to 2014
• Do different markers of school
performance change together or
separately?
• Decreasing district budget
correlated with increasing number of
low-income students
• Increasing classroom size correlated
with increasing dropout rate
Troubling correlations exist in
features’ trends over time

12. Correlations amongpercentage
changes in features over time,
2007 to 2014
• Do different markers of school
performance change together or
separately?
• Decreasing district budget
correlated with increasing number of
low-income students
• Increasing classroom size correlated
with increasing dropout rate
Troubling correlations exist in
features’ trends over time

13. About me (Eric Smith)

14. About me (Eric Smith)

15. About me (Eric Smith)

17. Challenge:
features not
normal
School district budget Regents Exams pass rate
Student retention rate10th grade class size

18. Cross-correlations
among features not
very descriptive
• Grid: cross-correlation of every
feature’s time traces with respect to
those of every other feature, averaged
over schools
• Each time trace standardized per
school prior to cross-correlation
• Pink, pinker, red shading: p < 1%,
0.01%, 0.0001%
• Difficulties: non-normal features, only
8 time points

19. Why not use a more complicated model?
1. Complicated model: most residuals not normal
2. Complicated model: strength of coefficients wildly different
among different k-folds of cross-validation
3. By eye, the more complicated model is more likely to give bizarre
predictions
4. Occam's razor: increase in predictive power too small to warrant
such complication

20. School district budget Regents Exams pass rate
For the simple
AR(1) model,
residuals are
mostly normal
12th grade population Student retention rate
• (School district budget is the
only statistic with non-normal
residuals)

21. School district budget Regents Exams pass rate
For the complex
AR(4) model,
residuals are
extremely non-
normal
12th grade population Student retention rate
• Model includes other
statistics as features and
uses elastic-net
regularization
• Indicates that the complex
model has lost sight of the
actual structure in the data?
• (In this case, school district
budget is the only statistic
with normal residuals)

22. Correlations amongfeatures (2014)
1. Regents Exam pass rate, college/post-
secondary placement rate correlated
2. Dropout rate, % receiving discount
lunch, teacher turnover rate highly
correlated
• These two groups are anti-correlated to
each other
• (Hard to judge 12th grade population,
teacher number, budget because of
differences in school and school district
sizes)
Clear correlations exist
among features