The CERI OECD/National Science Foundation International Conference took place in Paris, at the OECD Headquarters on 23-24 January 2012. Here the presentation of Session 6, Technology, Item 1.

1. Transformational Model of Translational
Research that Leverages Educational
Technology for Fast Data-Discovery
Feedback Loops
John Stamper
Pittsburgh Science of Learning Center
Human-Computer Interaction
Carnegie Mellon University
Connecting How we Learn to Educational
Practice and Policy: Research Evidence and
Implications International Conference
23-24 January 2012
1

2. rigorous, sustained scientific research
in education (NRC, 2002)
Vision for PSLC
• Why? Chasm between science & practice
– Low success rate (<10%) of randomized field trials
• LearnLab = a socio-technical bridge
between lab psychology & schools
– E-science of learning & education
– Social processes for research-practice
engagement
• Purpose: Leverage cognitive theory and
computational modeling to identify the
conditions that cause robust student learning
2

3. PSLC is multidisciplinary
170+ multidisciplinary researchers from California to
Germany
Ken Koedinger - Carnegie Mellon Co-Director
Charles Perfetti - University of Pittsburgh Co-Director
Executive Committee:
Vincent Aleven (HCI), Maxine Eskenazi (LTI; Diversity Director),
Julie Fiez (Psych), Geoff Gordon (ML), David Klahr (Psych; Education
Director), Marsha Lovett (Psych), Tim Nokes (Psych), Lauren Resnick
(Psych), Carolyn Rose (LTI), John Stamper (HCI)
3

4. The Setting & Inspiration
• Rich tradition of research on
Learning and Instruction at
CMU & University of
Pittsburgh
– Basic Cognitive Science
– Research in schools
– Intelligent tutors
• PSLC inspiration: Educational
technology as research
platform to launch new
learning science
Built in generalization to practice,
dissemination. 4

5. Translational Research
Feedback Loop
Design
Discover Deploy
Data
5

6. Real World Impact of
Cognitive Science
Algebra Cognitive Tutor
• Based on computational models of
student thinking & learning
• Course used nation wide
– Over 2600 schools, 500K students use
for ~80 minutes per week
• Spin-off:
Koedinger, Anderson, Hadley, & Mark (1997).
Intelligent tutoring goes to school in the big city.

7. Scaling Up
Learning &
Educational
Educational
Practice
Science
7

8. Translational Research 1:
Bringing Cognitive Science to School
Research base Practice base
Cognitive Psychology Math Educators
Artificial Intelligence Standards
Design
Cognitive Tutor courses:
Tech, Text, Training
Deploy
Address social context
8

9. Which kind of problem is most
difficult for Algebra students?
Story Problem
As a waiter, Ted gets $6 per hour. One night he made $66 in tips
and earned a total of $81.90. How many hours did Ted work?
Word Problem
Starting with some number, if I multiply it by 6 and then add 66, I get
81.90. What number did I start with?
Equation
x * 6 + 66 = 81.90
9

10. Data contradicts common beliefs
of researchers and teachers
Expert Blind Spot!
High School Algebra Students 100!
100% 90! % Correctly ranking equations as
80! hardest
!
70!
Percent Correct
80% 70%
61% 60!
60% 50!
42% 40!
40% 30!
20!
20% 10!
0!
0% Elementary! Middle! High School!
Story Word Equation
Teachers! School! Teachers!
Teachers!
Problem Representation
Koedinger & Nathan (2004). The real story behind story Nathan & Koedinger (2000). An investigation of
problems: Effects of representations on quantitative teachers beliefs of students algebra development.
reasoning. The Journal of the Learning Sciences.
Cognition and Instruction.

11. Cognitive Tutor Technology
Use ACT-R theory to individualize instruction
• Cognitive Model: A system that can solve problems in
the various ways students can
3(2x - 5) = 9
If goal is solve a(bx+c) = d
If goal is solve a(bx+c) = d
Then rewrite as abx + ac = d
Then rewrite as abx + c = d
If goal is solve a(bx+c) = d
Then rewrite as bx+c = d/a
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
• Model Tracing: Follows student through their individual
approach to a problem -> context-sensitive instruction
11

12. Cognitive Tutor Technology
Use ACT-R theory to individualize instruction
• Cognitive Model: A system that can solve problems in
the various ways students can
3(2x - 5) = 9
If goal is solve a(bx+c) = d
If goal is solve a(bx+c) = d
Then rewrite as abx + ac = d
Then rewrite as abx + c = d
Hint message: Distribute a
Bug message: You need to
across the parentheses.
multiply c by a also.
Known? = 85% chance Known? = 45%
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
• Model Tracing: Follows student through their individual
approach to a problem -> context-sensitive instruction
• Knowledge Tracing: Assesses student's knowledge
growth -> individualized activity selection and pacing
12

13. Translational Research 1:
Bringing Cognitive Science to School
Research base Practice base
Cognitive Psychology Math Educators
Artificial Intelligence Standards
Design
Cognitive Tutor courses:
Tech, Text, Training
Deploy
Address social context
13

14. Cognitive Tutor Algebra: Problems
that engage intuition & interest
Health Care
Extinction
Local Facts
Smoking Risks
Importance of Math Education
14

15. Cognitive Tutor Algebra: Rich Interactions
15

16. Cognitive Tutor Algebra: Rich Interactions
16

17. Cognitive Tutor Algebra course
yields significantly better learning
60
Course includes text, Traditional Algebra Course
tutor, teacher 50 Cognitive Tutor Algebra
professional
development 40
30
8 of 10 full-year
controlled studies 20
demonstrate
significantly better 10
student learning
0
Iowa SAT subset Problem Represent-
Solving ations
Koedinger, Anderson, Hadley, & Mark (1997).
Intelligent tutoring goes to school in the big city.

18. Scaling success? Yes
Done? No!
Why not?
• Final performance particularly in urban schools
is still far from desirable
• Weaknesses in field study results
– Not all studies are random assignment
– Two null results
• Many design decisions not guided by science
• We can use the deployed technology to collect
data, make discoveries, and continually
improve the instructional design
18

19. Scaling Down
Learning &
Educational
Educational
Practice
Science
19

20. Translational Research 2:
Fielded Systems Provide Data for New
Discoveries
Research base Practice base
Cognitive Psychology Math Educators
Artificial Intelligence Standards
Design
Cognitive Tutor courses:
Tech, Text, Training
Discover Deploy
Cognition, learning, Address social context
instruction, context
Data
Qual, quant;
process, product
20

21. How are cognitive models developed?
Cognitive Task Analysis
Traditional methods
• Structured interviews &
think alouds of experts & novices
=> Create symbolic model
Newer methods
• Data-Driven
• Educational Data Mining
=> Create statistical model => symbolic model
Meta-analysis: CTA produces 1.7 effect size (Lee, 2004)
21

22. Good Cognitive Model =>
Good Learning Curve
• An empirical basis for determining when
a cognitive model is good
• Accurate predictions of student task
performance & learning transfer
– Repeated practice on tasks involving the
same skill should reduce the error rate on
those tasks
=> A declining learning curve should emerge
22

23. A good cognitive
model produces a
learning curve
Without decomposition, using
just a single Geometry skill,
no smooth learning curve.
But with decomposition, (Rise in error rate because
12 skills for area, poorer students get
assigned more problems)
a smooth learning curve.
Is this the correct or best
cognitive model?
23

24. Inspect curves for individual
knowledge components (KCs)
Many curves show a Some do not =>
reasonable decline Opportunity to
improve model!

25. Can a data-driven process be
automated & brought to
scale?
Yes!
• Combine Cognitive Science,
Psychometrics, Machine Learning …
• Collect a rich body of data
• Develop new model discovery
algorithms, visualizations, & on-line
collaboration support
25

26. Automating the Cognitive
Model Discovery Process
Learning Factors Analysis
• Input
– Factors that may differentiate tasks
– Student performance across tasks & over time
• Output: Best cognitive model
Cen, H., Koedinger, K., Junker, B. (2006). Learning Factors Analysis:
A general method for cognitive model evaluation and improvement.
26

27. Discovery of new cognitive
models: Strategy & Results
• Mixed initiative human & machine discovery
– Visualizations to aid human discovery
– AI search for statistically better models
Stamper, J., Koedinger, K.R. (2011) Human-machine Student Model
Discovery and Improvement Using DataShop.
• Better models discovered in Geometry,
Statistics, English, Physics
27

28. LFA –Model Search Process
• Fully automated machine learning
guided search
• Input: Existing proposed models
• Output: Best cognitive model based on
splitting and merging existing models
Original
Model
BIC = 4328
Split by Embed Split by Backward Split by Initial
Automates the process of
50+
4301 4322 4312 4320
hypothesizing alternative cognitive
4320 4322 4313 4322 4325 4324
models & testing them against data
15 expansions later
4248
28

29. Summary
• Most ed field trials yield null results
– Need better data & cumulative theory
• Optimal instructional design requires
discoveries
– The student is not like me
• Scale up success: Cognitive Tutor
Algebra
• LearnLab: E-science infrastructure to
support science of learning
29

30. Opportunities Ahead
• Better models => better instruction
• Combine cog sci & machine learning
– Machine learning competitions
– PSLC s DataShop has 300+ datasets
– SimStudent learns new models
30

31. Thank you!
Acknowledgements
• Cognitive Tutors
John R. Anderson (Psych), Albert Corbett (HCI), Steve Ritter
(Carnegie Learning), …
• Cognitive Task Analysis
Mitchell Nathan (UW Ed Psych), Mimi McLaughlin (HCI), Neil
Heffernan (WPI CS), Marsha Lovett (Psych) …
• Cognitive Model Discovery
Brian Junker (Stats), Hao Cen (Machine Learning), Geoff
Gordon (ML) …
• Pittsburgh Science of Learning Center
– Kurt VanLehn (ASU CS) -- original PSLC co-director
– Ken Koedinger (HCI/Pysch), Charles Perfetti (Upitt Psych),
David Klahr (Psych), Lauren Resnick (Upitt Psych), Vincent
Aleven (HCI), Maxine Eskenazi (LTI), Carolyn Rose (LTI/HCI)
– All 200+ past & current members!
31