Some strategies on how edtech entrepreneurs can measure the impact their product or service has on learning.

1. Efficacy
how to prove that your edtech product works
@joeatmars
Impact8 2015

3. Source: GSV Advisors

6. Tradition: focus on inputs Efficacy: focus on outputs

7. …focused on teaching …focused on learning
…focused on inputs …focused on outputs
…tradition-driven …data-driven
…teacher as content authority …teacher as content guide
…teacher responsible for cause > effect …teacher iterates towards effect
…teacher is the sole proprietor of a
private practice
…teacher is a member of a Professional
Learning Community
…”best practices” …context sensitive
Educational researchEducational tradition

8. New methods of measuring impact
(for teachers and researchers)
1. Action-based research
2. Developmental evaluation
3. Design-based research
4. Control groups, A/B tests
5. SAMR

10. In medical terms, “efficacy” refers
to the ability of a product to
provide a beneficial effect.

13. Impact Frameworks
1. IRIS
2. Pearson
3. Learning Canvas

16. • The learning benefit of your product is not
self-evident
• Social metrics are often proxies for learning
outcomes
• Social metrics should be used to make
hypotheses about learning outcomes
From social metrics to learning metrics

17. Example: you have a product that teaches math online with a
parent-teacher bridge component. If you can increase parent-
teacher communication by 20 min/week you can hypothesize that
student achievement will increase based on “Parental Involvement
and Student Achievement: A Meta-Analysis” (Jeynes, 2005)
Your social metric + existing learning research = learning hypothesis
From social metrics to learning metrics

21. Impact Frameworks
1. IRIS
2. Pearson
3. Learning Canvas

26. From Good Intentions to Real Impact
5 Levels of Efficacy Method
Level 1: Why might the product have an impact on
learning?
Hypothesis
Level 2: Data shows some change amongst users Anecdotal evidence;
surveys
Level 3: Data shows the change is because of the
product
Control Groups
Level 4: Independent evaluation of
impact/explanation of learning impact
Independent researchers
Level 5: Ability to replicate/scale the learning
impact
Replication studies

27. Impact Frameworks
1. IRIS
2. Pearson
3. Learning Canvas

28. LEARNING GAINS
LEARNING
PROPOSITION
END USER
TECHNOLOGY
THEORY
METRICS
LEARNING OBSTACLES
Who are the
students who
use the
product?
What learning
gains do you
expect to see?
What hi-tech/low-
tech do the
students use?
What learning
theory predicts
these learning
gains?
What metrics will
you collect to prove
the learning gain?
1
4 2
3
8
6
5
7
9
SCHEDULE
How often do the
students interact
with the product?
PARTNERS
What partners
do you need
to execute?
- Age
- Grade
- Gender
- Special needs
- Socioeconomic
- Culture- Hand-outs
- iPad
- Lab equipment
- Sports equipment
- Once/week
- 30 min/day
- On demand
- Beginning of
each course
- Home use
- Increase in marks
- 21st C skills
- Complex thinking
- EQAO scores
- Scientific studies
- Theory search
- Existing products
- Benchmarks
- Marks
- Self-assessment
- Engagement
- Attendance
- Iterate on hypothesis, unanticipated metrics
- Teachers
- Researchers
- Lead students
- Principals
- Superintendents
(After implementation) What learning
gains resulted from the product’s use?
- Final metrics, confirmation of hypothesis
(After implementation) Did students get stuck
anywhere? Were there any unpredictable
negative side effects to the product’s use?

29. LEARNING GAINS
LEARNING
PROPOSITION
END USER
TECHNOLOGY
THEORY
METRICS
LEARNING OBSTACLES
SCHEDULEPARTNERS