This webinar focused on current practices for evaluating program effectiveness, the evaluation tools in use and how blended learning schools analyze multiple sources of data to understand program success. Presenters will lead a discussion of important considerations around the ongoing formative data collected to inform teachers and administrators about what contributes to student success in online courses. The panelists will explore how their programs approach collection of data and what methodology they use to organize and present data for school or district leaders. Speakers: Ernie Silva, Director of External Affairs, SIATech Elizabeth Hessom, Director of Education Services, SIATech Mary Esselman, Deputy Chancellor, Instructional Support & Educational Accountability Education Achievement Authority of Michigan

1. Approaches to Evaluating
Blended Learning Programs
• Mary Esselman, Deputy Chancellor, EAA, Michigan
• Ernie Silva, Director of External Affairs, SIATech,
California
• Elizabeth Hessom, Director of Education Services,
SIATech
January 2014

2. Webinar Format
•
•
•
•
Overview
Introductions
Presentation
Type questions in the chat window

3. Blended Learning
• “a formal education program in which a student
learns at least in part through online delivery of
content and instruction with some element of
student control over time, place, path, and/or
pace and at least in part at a supervised brickand-mortar location away from home…”
- (Horn and Staker, 2013)

4. Tech-rich = blended

5. Teaching and Learning
• What the student is
doing and where the
student is.
 What
the teacher is
doing and where the
teacher is.
 What
and where the
content is.

6. Webinar Focus
• Blended Learning Program Evaluation
– What ongoing measures are used to measure
program effectiveness?
– What evaluation tools/processes are used by
teachers to determine student and program
success?
– What evaluation tools/processes are used by
administrators and leadership to determine
student and program success?

7. Today’s Presenters
• Ernie Silva, Director of External
Affairs, SIATech, California
• Mary Esselman, Deputy Chancellor,
EAA, Michigan

8. SIATech Overview
• Appropriate Accountability for Drop
Out Recovery

9. Job
training,
mentors
&
internshi
ps
+
Blended
Learning &
Competency
Based
+
Adaptive Assessments
= Educated
Productive
Graduates

10. Our programs are
recovering drop-outs.
It is that simple.
13,186 standardsaligned diplomas
earned since 1998.
Key lessons have been
Key lessons have been
learned in the past 15 years
learned in the past 15 years
serving aa100% drop-out
serving 100% drop-out
population.
population.

11.
Florida
California
Arizona
[N=1241]
[N=2110]
[N=640]
New
Mexico
Arkansas
[N=261]
[N=302]
Average Age on
Entry
STAR Math Entry
STAR Reading Entry
Socioeconomically
Disadvantaged
19.31
G.E. 6.4
[N=1128]
19.44
19.81
19.06
18.77
G.E. 6.4 G.E. 6.0 G.E. 6.1
[N=1805] [N=430] [N=270]
G.E. 6.0
[N=282]
G.E. 6.9 G.E. 7.2 G.E. 6.7 G.E. 7.4
[N=1133] [N=1806] [N=431] [N=272]
G.E. 6.6
[N=304]
88%
100%
100%
98%
100%

12. Educational Achievement
Authority (EAA) Overview
• Detroit, Michigan

13. dd
Hi
Hi
gg
hh
EE
xx
pp
ee
t
ct
t
at
o
io
nn
ss
SS
l
el
-fPP
aa
i
ci
nn
gg
aa
nn
dd
M
M
aa
t
st
r
er
yy
BB
aa
ss
.
e.
dd
l
Bl
ee
nn
dd
ee
dd
n
“The mission of the EAA, as a catalyst for change,
is to disrupt traditional public schooling and provide a
prototype for next generation learning.”



Year 1
Year 2
Year 3

14. dd
Hi
Hi
gg
hh
EE
xx
pp
ee
t
ct
t
at
o
io
nn
ss
SS
l
el
-fPP
aa
i
ci
nn
gg
aa
nn
dd
M
M
aa
t
st
r
er
yy
BB
aa
ss
.
e.
dd
l
Bl
ee
nn
dd
ee
dd
n
EAA – Year 1 – Five Pillars
1.Students are grouped by readiness, not by age or grade.
2.Students create and assume ownership for their respective
personalized learning paths and are able to communicate their
progress relative to their individualized learning goals.
3.Students are allowed to work at their own pace using a blended
delivery system to master rigorous standards aligned to next
generation readiness.
4.Students provide evidence of mastery through relevant
performance tasks and common assessments.
5.Continuous feedback is provided to students, teachers,
administrators and parents.

15. dd
Hi
Hi
gg
hh
EE
xx
pp
ee
t
ct
t
at
o
io
nn
ss
SS
l
el
-fPP
aa
i
ci
nn
gg
aa
nn
dd
M
M
aa
t
st
r
er
yy
BB
aa
ss
.
e.
dd
l
Bl
ee
nn
dd
ee
dd
n
EAA – Year 2
•
Professional Development was redesigned to follow a
blended, personalized delivery model
•
Schools began to look at new ways to use time, space,
talent and resources.
•
PASE, the SCL Village, the Open Math Lab, and the
Learning Pods were all created within EAA schools to
allow students to move completely at their own pace and
to start leveraging time, space, and talent more efficiently
and effectively.
•
Students began transitioning freely between learning
spaces and time. Students are no longer tethered to one
classroom and one subject.

16. dd
Hi
Hi
gg
hh
EE
xx
pp
ee
t
ct
t
at
o
io
nn
ss
SS
l
el
-fPP
aa
i
ci
nn
gg
aa
nn
dd
M
M
aa
t
st
r
er
yy
BB
aa
ss
.
e.
dd
l
Bl
ee
nn
dd
ee
dd
n
EAA – Year 3:
Next Generation High School
•
Staffing structures will be redesigned to truly leverage
time, talent, and space with a distributed leadership model
rather than a traditional principal led school.
•
Students will demonstrate mastery of Next Generation
Competencies that truly prove they are Next Generation
Ready.
•
Staffing structures will allow for multiple adults, including
full time and part-time teachers, tutors, paras, instructional
assistants, mentors and interventions

17. dd
Hi
Hi
gg
hh
EE
xx
pp
ee
t
ct
t
at
o
io
nn
ss
SS
l
el
-fPP
aa
i
ci
nn
gg
aa
nn
dd
M
M
aa
t
st
r
er
yy
BB
aa
ss
.
e.
dd
l
Bl
ee
nn
dd
ee
dd
n
“The mission of the EAA, as a catalyst for change, is to
disrupt traditional public schooling and provide a prototype
for next generation learning.”
..
EAA Year One
EAA Year One
Five Pillars:
Five Pillars:
1.Students are grouped by
1.Students are grouped by
readiness, not by age or grade.
readiness, not by age or grade.
2.Students create and assume
2.Students create and assume
ownership for their respective
ownership for their respective
personalized learning paths and
personalized learning paths and
are able to communicate their
are able to communicate their
progress relative to their
progress relative to their
individualized learning goals.
individualized learning goals.
3.Students are allowed to work at
3.Students are allowed to work at
their own pace using aa blended
their own pace using blended
delivery system to master rigorous
delivery system to master rigorous
standards aligned to next generation
standards aligned to next generation
readiness.
readiness.
4.Students provide evidence of
4.Students provide evidence of
mastery through relevant
mastery through relevant
performance tasks and common
performance tasks and common
assessments.
assessments.
5.Continuous feedback isis
5.Continuous feedback
provided to students, teachers,
provided to students, teachers,
administrators and parents.
administrators and parents.
EAA Year Two
EAA Year Two
•Professional Development was
•Professional Development was
redesigned to follow aablended,
redesigned to follow blended,
personalized delivery model
personalized delivery model
•Schools began to look at new ways
•Schools began to look at new ways
to use time, space, talent and
to use time, space, talent and
resources.
resources.
•PASE, the SCL Village, the Open
•PASE, the SCL Village, the Open
Math Lab, and the Learning Pods
Math Lab, and the Learning Pods
were all created within EAA schools
were all created within EAA schools
to allow students to move completely
to allow students to move completely
at their own pace and to start
at their own pace and to start
leveraging time, space, and talent
leveraging time, space, and talent
more efficiently and effectively.
more efficiently and effectively.
•Students began transitioning freely
•Students began transitioning freely
between learning spaces and time.
between learning spaces and time.
Students are no longer tethered to
Students are no longer tethered to
one classroom and one subject.
one classroom and one subject.
EAA Year Three
EAA Year Three
Next Generation High School
Next Generation High School
•Staffing structures will be redesigned to truly
•Staffing structures will be redesigned to truly
leverage time, talent, and space with aa
leverage time, talent, and space with
distributed leadership model rather than aa
distributed leadership model rather than
traditional principal led school.
traditional principal led school.
•Students will demonstrate mastery of Next
•Students will demonstrate mastery of Next
Generation Competencies that truly prove
Generation Competencies that truly prove
they are Next Generation Ready.
they are Next Generation Ready.
•Staffing structures will allow for multiple
•Staffing structures will allow for multiple
adult including full time and part-time
adult including full time and part-time
teachers, tutors, paras, instructional
teachers, tutors, paras, instructional
assistants, mentors and interventions
assistants, mentors and interventions

18. SIA Tech Assessment

19. •• Achievement scores on state
Achievement scores on state
mandated exams penalize
mandated exams penalize
SIATech students, because the
SIATech students, because the
majority enter our schools
majority enter our schools
achieving at the 6th grade level
achieving at the 6th grade level
in reading & math.
in reading & math.
•• State tests do not work well
State tests do not work well
for us
for us
•• Prescribed time table
Prescribed time table
•• Limited administrations
Limited administrations
•• Too hard/too long for too
Too hard/too long for too
many
many
SIATech schools needed a different way to demonstrate
accountable for student learning!

20. Advocating for Alternative
Accountability
Reaching At Promise Students
Association
http://www.rapsa.org
Save the date: November 14-15 San
Diego
3rd Annual Alternative Accountability
Policy Forum

21. •• Adaptive
Adaptive
•• Multiple administration times
Multiple administration times
•• Short
Short
•• Correlated to state standards
Correlated to state standards
•• Reliable
Reliable
•• Measure & report growth and achievement
Measure & report growth and achievement
•• Can use results to improve learning and
Can use results to improve learning and
teaching
teaching
Renaissance STAR Math and Reading*

22. What is learning growth?
• Student learning growth is a statistical measure that estimates how
much each student progresses from one test to the next
compared to similar students
CST ELA SCALE SCORE
600
400
Advanced
349
Proficient
15
25
299
Below Basic
150
330
Basic
270
345
Far Below Basic
305
Grade 6
Grade 7
Grade 8

23. Benefits of measuring learning growth Benefit #1 – Holds schools accountable for the
things they have control over
Benefit #2 – Measuring learning growth fosters
student hope and engagement because gains are
based on individual progress – not unrealistic
targets
Benefit #3 – Measuring student learning growth sets
high expectations for every student, because every
student (high, middle, and low achieving) must
increase his or her learning every year

24. Students take ownership of personal
learning as schools make efficient use
of core processes
•• Interpretation of Individual
Interpretation of Individual
Student Growth Data
Student Growth Data
•• Individual Student Goal Setting
Individual Student Goal Setting
(weekly, biweekly, monthly)
(weekly, biweekly, monthly)
•• Student Accountability for
Student Accountability for
Tracking and Adjusting
Tracking and Adjusting
Academic and Learning
Academic and Learning
Progress
Progress
•• Evaluation of Course
Evaluation of Course
Completion and Academic
Completion and Academic
Growth
Growth

25. ••
••
••
••
Annual Development of SITE
Professional Development
Plans
Interpretation of Learning
Interpretation of Learning
Data to determine general
Data to determine general
success and gaps in school
success and gaps in school
outcomes.
outcomes.
Site Goal Setting
Site Goal Setting
Determination of a Site
Determination of a Site
Professional Development
Professional Development
Focus
Focus
Evaluation of Plan
Evaluation of Plan
Implementation and Results
Implementation and Results
(EOY)
(EOY)

26. Annual Development of Individual Staff Member Professional
Development Plans
•• Interpretation of Learning Data as
Interpretation of Learning Data as
related to specific content area(s).
related to specific content area(s).
•• Individual Staff Goal Setting
Individual Staff Goal Setting
•• Planning Pertinent Training
Planning Pertinent Training
(examples)
(examples)
•• Lesson Plan Development
Lesson Plan Development
•• Differentiation
Differentiation
•• Knowledge Acquisition
Knowledge Acquisition
•• Teaching Strategies
Teaching Strategies
•• Data Driven Instruction
Data Driven Instruction
•• Evaluation of Plan Implementation
Evaluation of Plan Implementation
and Outcomes (EOY)
and Outcomes (EOY)

27. • Monthly Content Teacher Conversations
Wednesdays 3:30-4:30 pm
Math, ELA, Science, Social Studies, CTSP
(4/year)
Sharing of best content practice with a
focus on:
- literacy (reading and writing across
content areas)
- small group lessons
- opening and closing whole group
lessons/activities

28. • Monthly distribution of active student
learning data
• Research into a computer based,
individual student reading program
(included into 2013-14 budget for all
sites)

29. Multiple Measures Include Gallup Poll
of Hope, Engagement and Well Being

30. Hope
the ideas and energy
we have for the future.
Hope drives:
attendance, credits
earned, and high school
GPA
Hope predicts:
college GPA and
retention
Hope scores are better
predictors of college
success than high
school GPA, SAT, and
ACT scores.

31. HOPE
Ways around a problem
Response Average
Average student response on a five point scale, where
1=strongly disagree and 5=strongly agree.
Question Text:
Question Text:
I Ican find lots of ways
can find lots of ways
around any problem.
around any problem.
(Five point scale)
(Five point scale)
Possible Student Activities to Impact this Area:
During direct instruction problem-solving, ask
students for other ways they would solve the
problem or issue.
Remind students of the variety of ways they can get
help and support for problem-solving
Help students learn that there are many ways to
success and learning (not just innate talent-there
are MANY talented but unsuccessful folks)
teach multiple problem solving techniques
encourage students to learn how they ‘best learn of
learn best’
provide students with specific strategies to deal with
their ‘blocks’ (e.g. memorizing, planning,
organizing, classifying, preparing, …)
Your Reflections / Actions for this Area:

32. Engagement
the involvement and
enthusiasm for school.
Engagement scores
separate
high-performing from
low-performing schools

33. ENGAGEMENT
Schoolwork important
Response Average
Response Average
Average student response on a five point
scale, where 1=strongly disagree and
5=strongly agree.
Possible Student Activities to Impact this
Area:
Question Text:
Question Text:
My teachers make me
My teachers make me
feel my schoolwork is
feel my schoolwork is
important.
important.
(Five point scale)
(Five point scale)
Teachers provide frequent, sincere, specific feedback on
schoolwork in a variety of ways
Teachers make relevant connections between schoolwork
and goal attainment;
Students are provided with opportunities to discuss with
teachers and/or mentors how their schoolwork is
helping them…
Your Reflections / Actions for this Area:

34. Well-Being
how we think about and experience
our lives.
Well-being is an indicator of how
students are currently doing.
Well-being also predicts future
success.
A thriving student earns 10% more
credits and a 2.9 GPA compared to a
struggling or suffering student who
completes fewer credits and earns a
2.4 GPA.

35. WELL-BEING
Treated with respect
Percent who said Yes
Possible Student Activities to Impact this
Area:
Question Text:
Question Text:
•
•
•
•
Were you treated with
Were you treated with
respect all day
respect all day
yesterday?
yesterday?
(Yes or no)
(Yes or no)
•
•
ALL staff knows and USES student names
Be and Do what you want to see from students
Explain the reasons/why or why not for critiques of student
work, not just X
Reward and/or acknowledge acts of respect to build a
culture
Demonstrate positive approach to classroom
management and student discipline (preserve dignity)
Find ways to help students answer their own
questions and then honor their success
Your Reflections / Actions for this Area:

36. EAA Assessment

37. Program Evaluation Questions
Question 1:
To what
extent is the
studentcentered
system being
executed as
planned?
Question 2:
To what extent is
there sufficient
support available
to staff, parents,
and students
who are
participating in a
student-centered
system of
education?
Question 3:
To what extent is
implementation of the
student-centered
system achieving
desired changes in
student performance
outcomes, socioemotional well-being
and teacher practice?
Question 4:
To what extent is
the program being
scaled in terms of
depth,
sustainability,
spread, shift, and
evolution?

38. 4-pronged Assessment System


Individual Growth (Performance Series)
Performance Tasks to assess Deeper Learning
(3 pieces of evidence for each Learning Target)
Learn (reflection)
 Practice (Understanding)
 Apply (real-world connections) Common Assessments



Achievement (State Assessment)
Next Generation Ready—Graduation by Defense

39. Integrated Teaching & Learning Platform
.
Real-time
Analytics
• Supports a
robust collection
of open sourced,
licensed, and
teacher created
resources.
• Learning objects
parsed in bite
sized chunks.
• Digital assets
mapped to
standards--local,
state, common core.
• Curriculum maps
directly linked to the
digital assets
• Students have
access to choicebased learning
pathways.
• Dashboards
provide data for
students,
parents and
teachers.
• Data is
provided as a
prescriptive
newsfeed for
immediate
application.
• Integrated
planner, profiles,
personalized
learning plans
with student,
teacher, parent,
communication
tools.
• Peer to Peer
support system.

40. Part 2: Access to Information—
monthly, weekly, real-time





Time & Topic
Pace/Progress
Badges for leadership, scholarship, character,
service personal milestones
Student Profile
Mastery:



School/Class/Individual by Level/Unit/Standard
Mastered/In progress/Not Yet Attempted
Digital Portfolio of Evidence

41. EDUCATION ACHIEVEMENT AUTHORITY OF MICHIGAN
EDUCATION ACHIEVEMENT AUTHORITY OF MICHIGAN
The teaching and learning platform:
The student experience…

42. Student Progress Page

43. Creating a personalized learning path

44. The Teacher Experience:
3-D Classroom Radar Report of Student
Productivity—who’s progressing and who is not vs. traditional 2-D, linear
gradebook.
Color-coding on students represents
Color-coding on students represents
self-assessment information
self-assessment information
Click a student
Click a student
to drill down to
to drill down to
view real-time
view real-time
personal
personal
performance
performance

45. Classroom Summary Report
Click a student to drill down to view
Click a student to drill down to view
real-time personal performance
real-time personal performance
Pie charts indicate mastery of
Pie charts indicate mastery of
individual learning targets.
individual learning targets.
Self-assessment results
Self-assessment results
Teachers can click to select
Teachers can click to select
individual or groups of
individual or groups of
students to assign
students to assign
supplemental resources
supplemental resources
and/or send notes.
and/or send notes.

46. Standards Report
Teachers, students and
Teachers, students and
parents can review
parents can review
progress against
progress against
individual standards as
individual standards as
well as access
well as access
supplemental study
supplemental study
resources.
resources.

47. Teacher Grading and Messaging Center
Item for grading are collected in
Item for grading are collected in
the to-do list making it easy for
the to-do list making it easy for
teachers to provide grades and
teachers to provide grades and
feedback.
feedback.

48. Teacher Personal Progress
Just as teachers can monitor real-time
Just as teachers can monitor real-time
student progress they also can monitor their
student progress they also can monitor their
own progress.
own progress.

49. Lessons Learned - EAA

50. Part 3: Other Assessments of
Progress/Implementation




End of Course for non-tested Subjects
GRIT
Value-added Growth Model for Pay for
Performance S ystem
Instructional Practice—scaffolded walkthroughs




Classroom Set-up/Visual Cues
Classroom Structures
Rituals/Routines
Student Ownership/Engagement
Note: We triangulate data across
reports/surveys/observations/focus groups

51. Lessons Learned – SIA Tech

52. 1. A Learning Growth model is imperative for School
Evaluation/Monitoring of reengaged students.
2. Graduation Rate Calculations must be appropriate
in terms of cohort.
3. Strong advocacy work at the local, state and
federal level needs to be a priority.
4. Competency-based education needs engaging,
specialized and customized curriculum to truly
accelerate learning
5. Staff development and training is crucial to success
with a dropout population
6. A robust data system needs to be in place in order
to show success in multiple, meaningful ways.

53. Contact Information
• Mary Esselman, EAA.
– MEsselman@eaaofmichigan.org
• Ernie Silva, SIATech
– Ernie.Silva@siatech.org
• Elizabeth Hessom, SIATech
– Elizabeth.Hessom@siatech.org