This document summarizes a presentation on an innovative online diagnostic assessment tool that provides dynamic assessment of language learners. The presentation included demonstrations of Chinese and French language tests and an explanation of test design, scoring procedures, and how learner profiles are generated. The tool aims to help teachers meet diverse learner needs through computerized dynamic assessment and mediation delivered via prompts within an online test format. Group data was presented showing learning gains from pre- to post-testing across various language skills and levels for different languages.

1. Helping Teachers Meet Learner Needs
Through Innovative Online Diagnostic
Assessment
Jim Lantolf, Matt Poehner, Xiaofei Lu,
Penn State University
RumiaAbleeva, University of Georgia
The project was funded by a grant from the U.S. Department of Education:
PO17A080071
ACTFL Annual Conference. Denver. November 19, 2011

2. Outline of Presentation
• 1. Introduction to Dynamic Assessment: Ableeva
• 2. Online Test Logic: Lu
• 3. Demonstration: Lu &Ableeva
– Chinese Listening Test
– French Reading Test
• 4. Scoring Procedures & Learner Profile: Lantolf
• 5. Audience Questions & Discussion

3. Computerized dynamic assessment (CDA)
• Interventionist (standardized, scripted menu of
mediating prompts)
• Mediation delivered via computer program rather
than human mediator
• Context of computer-based test of L2 listening and
reading comprehension in Chinese, French & Russian
3

4. Test design
• Each test item focuses on one or two specific
language constructs, e.g.,
– Phonetics, vocabulary, grammar, discourse, pragmatics
• Design of the mediating prompts guided by findings
from the the interactive DA sessions
4

5. Test Design
• Each item includes four increasingly explicit prompts
– Prompt 1: drawing learners’ attention to the most
relevant segments
– Prompt 2: drawing learner’s attention to the key language
constructs of the item resided in those segments
– Prompt 3: offering an explanation for the key language
constructs
– Prompt 4: offering the correct answer to the question
– Each item also contains an explanation for the answer
5

6. Sample item 1: Chinese listening
Script
男：您还有什么事？我马上要出去。
女：那好，王厂长，改天我再来拜访你。
What does the man want to do?
A. make a phone call
B. visit the woman
C. go out for dinner
D. cook at home
E. end the conversation
6

7. Areas of difficulty for sample item 1
Script
男：您还有什么事？我马上要出去。
女：那好，王厂长，改天我再来拜访你。
• Areas of difficulty revealed from DA sessions
– 出去 ‘go out’: students knew this word but failed to
recognize it in listening
– 改天 ‘another day’: students knew the characters but not
the meaning of the compound word
7

8. Sample item 1: Presentation in the test
8
8

9. Sample item 1: Prompt 1
9

10. Sample item 1: Prompt 2
10

11. Sample item 1: Prompt 3
11

12. Sample item 1: Prompt 4 and explanation
12

13. Sample item 2: French reading
• Language constructs
– Grammar (primary): Past aspect (imperfective), negation
– Discourse (Secondary): Relationship between a negative
clause and a mais plutot clause
13

14. Sample item 2: Presentation in the test
14

15. Sample item 2: Prompt 1
15

16. Sample item 2: Prompt 2
16

17. Sample item 2: Prompt 3
17

18. Sample item 2: Prompt 4
18

19. Sample item 2: Explanation
Although the author liked all aspects of the weekends at his grandparents'
house, his enjoyed the quiet evenings with his grandfather the most. The key
is the passage that starts with "Ce que j'aimais le plus" (what I liked the
most). The author states "ce n'était pas les les animaux" (it wasn't the
animals), "mais plutôt les soirées calmes avec mon grand-père" (but rather
the quiet evenings with my grandfather). He sets up an oppositions between
the negative "ce n'était pas les les animaux" with the "les soirées calmes avec
mon grand-père" by using the phrase "mais plutôt" (but rather).
19

20. Individual Score on French Listening Assessment
• Mediated (i.e., prompted) score and Actual (i.e.,
initial attempt to answer prior to prompting)
• NB: Maximum score possible on test is 60 (i.e., 4 x 15
items)

21. Performance of Individual Student on French
Listening Assessment
• X’s indicate level of prompts required by item. Highest score
on each item = 4 (i.e., initial attempt to respond before
prompts are provided)

22. Performance of Individual Student on French
Listening Assessment
• We report the time it takes for a test taker to initially respond
to each item and the time it takes for the individual to
respond following each required prompt.
– It took the individual here 23 seconds to provide an initial response
and 5 seconds to respond to the first prompt. The individual provided
the correct response (a) following the first prompt and therefore
earned a score of 3 for the item

23. Group Performance on French Listening
Assessment
• Group scores are reported for each item by the level of prompt required. Thus, of
a total of 18 students who took the test (several students did not complete specific
items and received a NA) 3 provided the correct response for item 1 on their initial
attempt. They therefore received a score of 4. Three students required one prompt
and received a score of 3; two required 2 prompts and received a score of 2; 1
student required three prompts and received a score of 1 and 1 students was
unable to provide the correct response after 4 prompts and received a score of 0.

24. Group Performance on French Listening Assessment
• Scores are also reported by rubric. Thus, the first 5 items included in the final row are
discourse relevant; the following 9 are relevant to grammar. The 5 items includes under
Transfer represent difficult items that are included at the final five items in the test and
are designed to determine the extent to which test takers may have learned during the
test. Notice that the first two Transfer items (11, 12) are grammatical and the final three
(13, 14, 15) are discourse.
– Notice that items 11, 12, and 14 were more difficult (as expected) given the low number of students that
provided the correct response and their first attempt. Items 13 and 15 were easier as 8 and 6 students
respectively provided a correct response on their initial try and only 2 and 4 students respectively required
only a single prompt to produce a correct answer.

25. French Reading: Low Intermediate
Group N Mean SD Min Max
FRA 23 47.478 11.712 20 60
FRM 23 54.304 6.123 36 60
t=4.487 p< .001 ES = 2.151 r = .732

26. French Reading: High Intermediate
Group N Mean SD Min Max
FRA 42 40.762 12.81 8 60
FRM 42 51.31 6.809 31 60
t = 9.826 p< .001 ES (d = 1.028;
r = .457)

27. French Listening: High Intermediate
Group N Mean SD Min Max
FLA 34 29.264 15.596 4 59
FLM 34 44.382 11.311 17 59
t = 12.529 p< .001 ES (d = 1.109;
r = .485)

28. Chinese Reading: High Intermediate
Group N Mean SD Min Max
CRA 81 47.654 23.3 8 96
CRM 81 70.802 18.02 10 96
t = 19.585 p< .001 ES (d = .1.111;
r = .480)

29. Chinese Listening: Low Intermediate
Group N Mean SD Min Max
CLA 71 42.592 25.977 0 92
CLM 71 63.535 18.904 7 92
t = 15.367 P < .001 ES (d = .922;
r = .418)

30. Learning Potential Scores
• “…the learning potential score (LPS) has to reflect
both gain made by the student from pre- to post-test
and an absolute achievement score at the post-test”
(Kozulin& Garb 2002: 121)
• LPS Sub-Groups
– High > 1.0
– Mid < .99; > .72
– Low < .71

31. Learning Potential Scores: Chinese Reading
Test
LPS
1.4000
1.2000
1.0000
0.8000
0.6000
0.4000
0.2000
0.0000
0 10 20 30 40 50 60 70 80 90

32. LPS for French Listening Chinese Reading
Assessment
ACTUAL SCORE MEDIATED SCORE LPS
20 45 .729
20 51 .854
20 59 1.020
24 49 .770
24 60 1.000
32 67 1.062
48 75 1.062

33. Interpretation of LPS
• In the previous slide three students had the same ACTUAL score on the Chinese
Reading Assessment: 20
– Notice that they each generated a different LPS
• One what low -- .729; one was moderate -- .854; and one was high –
1.020
• This indicates that they each reacted differently to mediation provided
by the prompt
– The first student had a mediated score of 45; the second had a
mediated score of 51 and third had a mediated score of 59
– LPS score then predicts how each student is likely to benefit from
instruction, with a low LPS pointing to the need for more
instruction and a high LPS indicating the need for less instruction in
order for the learner to develop.
– NB: These predictions must be supported by empirical research.
• Notice that two students with different actual scores (i.e., 32 & 48) have
identical LPSs even though their respective mediated scores also varied. LPS is
determined by relative difference between ACTUAL and MEDIATED score.

34. Tests Available Soon thru CALPER
• General information about dynamic
assessment, including working paper and
bibliography on CALPER’s webpage PROJECTS
• Second Edition of Dynamic Assessment:
Teacher’s Guide (available for purchase on
DVD for $20)
• http://calper.la.psu.edu/dyna_assess.php