On March 18 we were lucky to host Chaitu Ekanadham, a data scientist from Knewton. Knewton is an education technology company that uses adaptive learning techniques. It uses data about past learning to aid students in future learning experiences. For example, in traditional “text book” learning, students move through the content in a linear fashion, one chapter and then the next. While this is a tried and true method of teaching, it isn’t the best solution for every student in every situation. Some students may need only a review of some particular content, while others may need to spend a great deal of time on the same content. For that reason, it makes sense to direct students to content dynamically–show them only the content that would be most helpful for them at that time. When Knewton begins working with a publisher, they start by organizing the material in the textbook into a Knewton knowledge graph. This is used to represent the ways that content is related to each other conceptually. In this way students’ progression can be evaluated automatically. This gets the right content to the right student at the right time. So how does one even begin to apply models to this sort of information? Knewton primarily utilizes the data to create models describing the learners, and models describing the educational content. Modeling Learners The model of the learner is individualized to each student. After a student has completed some amount of graded content, the system has some idea of their capability. Knewton can then model the likelihood that a student will correctly answer some future question. Quiz questions can then be presented that are neither too hard nor too easy. Modeling Content When recommending new content for a student, the system avoids any content that is predicted to have a likelihood of either close to 0 (indicating the student will most likely not learn or retain this information) or 1 (which indicates that the student is familiar with the concepts and does not need any further instruction). All material is given a score for how difficult it is based on the number of students who historically found it difficult. This value is used in the above model of likelihood a student will get it correct. In addition, the content is judged for how it engages the students. If students generally move through the content with few breaks it is considered engaging – while content that is consistently associated with long breaks, is considered not engaging. Response time is summed up for all students to give a value for a typical response time for some given content. How These Models are Used Knewton uses this data in several different business purposes 1. Generate recommendations for students–what is the best content for that student to consume right now? 2. Analytics–are students on track? What do they need to do in order to pass the next assessment of their skills? 3. Content insights for the cre

1. NYC Open Data, March 18, 2015
Chaitanya Ekanadham
Representing
learning
experiences

2. personalize learning experiences
for students around the world
PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
Knewton’s mission
content inventory
student goals
interaction data
recommendations
analytics
content insights

3. provide high quality adaptive
education to everyone in the world
PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2014 KNEWTON, INC.
Knewton’s mission

4. leverage past learning experiences
to improve future learning
experiences
PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
data science team’s mission

5. when a learner is presented some content and
acquires knowledge as a result
PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2014 KNEWTON, INC.
what’s a “learning experience”?
learner
s
content
alice
sharo
n
..
..
.
video lecture on limits
derivatives definition
integrals quiz
limits exercise
related rates word problem
bob
past
future

6. leverage past learning
experiences to improve future
learning experiences
PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

7. past learning
experiences
models
learner
representation
content
representation

8. models
past learning
experiences
learner
representation
proficiency
learning speed
time investment
preferred mode
content
representation
similarities
difficulty
length
effectiveness

9. leverage past learning experiences
to improve future learning
experiences
PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

10. models
learner
representation
content
representation
hypothetical
future learning
experience
score

11. path
dependent
effects
difficult to
quantify
do not have
final grades
leverage past learning experiences to
improve future learning experiences
PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
unique challenges

12. content
representation

13. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
video lecture on
limits
limits problem set
derivatives
definition
common derivatives
differentiation
exercises
integrals as limits of
Riemann sums
fundamental
theorem of calculus
summation notation
reference
not scalable!

14. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
video lecture on
limits
limits problem set
derivatives
definition
common derivatives
differentiation
exercises
integrals as limits of
Riemann sums
fundamental
theorem of calculus
summation notation
reference

15. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
video lecture on limits
limits problem set
derivatives definition
common derivatives
differentiation exercises
integrals as limits of
Riemann sums
fundamental theorem of
calculus
summation notation
reference
instruction
assessment
both

16. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.

17. measuring
learner ability

18. models
learner
representation
content
representation
hypothetical
future
learning
experience
score

19. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
learner
proficiency
item
difficulty
item
discrimination

20. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
learner
s
assessment
content
alice
sharon
...
..
.
limits exercise
integrals quiz
limits challenge problem
related rates word problem
bob
differentiation problem

21. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
learner
s
assessment
content
alice
sharon
...
...
limits exercise
differentiation problem
integrals quiz
limits challenge problem
related rates word problem
bob

22. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
learner
s
alice
sharon
..
.
bob
assessment
content
limits exercise
differentiation problem
integrals quiz
limits challenge problem
related rates word problem...

23. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
learner
s
alice
sharon
..
.
bob
assessment
content
limits exercise
differentiation problem
integrals quiz
limits challenge problem
related rates word problem...

24. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
? ?
likelihood
prior
posterior
? ?

25. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
likelihood
prior
?
?
?
?
?
?? ?
?
t=3
? ? t=2
posterior
? ? t=1

26. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
differentiation
related rates
word problem
3D shape volume

27. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
assessing multiple concepts
compensatory likelihood: knowing 1 concept is good enough
differentiation
related rates
word problem
3D shape volume

28. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
assessing multiple concepts
non-compensatory likelihood: have to know both concepts
differentiation
related rates
word problem
3D shape volume

29. learner
timing patterns

30. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
time

31. timesession break
response times

32. histogram of log-taus
with highlighting!
48K students
9.6K modules
1M interactions

33. histogram of log-taus
with highlighting!
48K students
9.6K modules
1M interactions

34. histogram of log-taus
with highlighting!
session
breaks
response times
?
48K students
9.6K modules
1M interactions

35. log time
probability response times
session breaks

36. normalized response time
normalizedquittingrate

37. normalized response time
normalizedquittingrate
high
engagement

38. normalized response time
normalizedquittingrate low
engagement
high
engagement

39. normalized response time
normalizedquittingrate
non-sticky low
engagement
high
engagement

40. normalized response time
normalizedquittingrate
non-sticky
boredom?
low
engagement
high
engagement

41. normalized response time
normalizedquittingrate
non-sticky
boredom?
low
engagement
high
engagement
slow

42. normalized response time
normalizedquittingrate
non-sticky
boredom?
low
engagement
high
engagement
slow
frustration?
deep thought?

43. PROPRIETARY & CONFIDENTIAL - NOT FOR REDISTRIBUTION © 2015 KNEWTON, INC.
exciting challenges
● offline learning
● learner affinity for pedagogical strategies
● automating content graphing
● student agency
● controlled experiments
● “adaptivity-ready” content design

44. chaitanya ekanadham
managing data scientist
Knewton, Inc.
chaitu@knewton.com
Twitter: @knewton
thank you.

45. appendix