1. a work in
progress

2. Motivation for study
Nothing can replace the teacher!
BUT…
0 Increasing desire among educational institutions to cut
costs:
0Infrastructure costs
0Personnel costs
0 Increasing pressure to embrace digital technologies:
0 Technology offers teachers and learners documented
benefits
0 Stand-alone programs are gaining popularity:
0 Among public in general
0 Among administrators
(if not among informed language teachers themselves!)

3. Digital software
language programs
0 Examples:
0 Rosetta Stone
0 Tell Me More (Auralog)
0 Learn Spanish Now (Transparent Language)
0 Pimsleur Approach
0 Claim to be as effective or more effective than classroom
learning
0 “fastest” or “only” way to learn a language??
0 “learn any language in just 10 days”??
0 Marketing heavily in recent years, especially in K-12 and
higher education arenas
Rosetta Stone is best seller and the
most complete (Godwin-Jones, 2007, 2009)

4. Rosetta Stone
0Informed by the Natural Approach to language
learning (Krashen & Terrell, 1983):
0 Works under the premise that adults learn a second
language in the same way that a child learns a first language
0However… research has shown that adult second
language learning is “fundamentally different”
0Bley-Vroman's [1988] Fundamental Difference
Hypothesis…
0Children learn in a highly social, affective, and rich setting
0Children lack the cognitive advantage of their linguistic
background that adults have

5. Reviews of these programs
0 Lafford, Lafford & Sykes (2010)
0 Evaluate if programs provide the tools necessary for effective
language learning, based on features that research has shown to be
important
0 Interaction, relevant contextualization of language, etc.
0 “… these products do not incorporate a number of the [necessary]
research-based insights (e.g., the need for culturally authentic, task-
based activities) that informed SLA scholars might have given
them.”
0 Santos (2011)
0 Lack of context
0 General inability to respond to spontaneous student speech
0 What Rosetta Stone calls interaction is “a rather poor and limited
version of what one would encounter in a real-life conversation”
0 DeWaard (forthcoming 2013)
0 “Not a viable replacement of current instruction at the
postsecondary level”

6. Studies on effectiveness
of Rosetta Stone and other programs
0 Few empirical studies
(Aside from Rosetta Stone’s own commissioned work, e.g.
Vesselinov, 2009)
0 Nielson (2011) – self-study programs in workplace; some
success but remarkable attrition; lack of community
(e.g., Rovai, 2002)?
0 Stevenson & Liu (2010) – lack of ability to engage learners
in true interaction; users do not take advantage of Web 2.0
tools to network.
Overall lack of evidence to support the notion that such
programs can replace the classroom or can
offer anything that a classroom program cannot

7. So….
The time has come to put the
claims made by Rosetta Stone
(and by extension other programs)
to the test!
This research was conducted
with Rosetta Stone’s
knowledge but they were not
involved in the design, data
collection or analysis.

8. Research Questions
1. How does (Spanish) learner achievement compare
across environments in terms of
0lexical items?
0structural/grammatical content?
0writing proficiency?
0listening proficiency?
0pronunciation?
0oral communicative proficiency?
0cultural awareness?
2. How do student attitudes toward the Spanish
learning experience differ across the learning
environments?

9. Research Questions
1. How does (Spanish) learner achievement compare
across environments in terms of
0lexical items?
0structural/grammatical content?
0writing proficiency?
0listening proficiency?
0pronunciation?
0oral communicative proficiency?
0cultural awareness?
2. How do student attitudes toward the Spanish
learning experience differ across the learning
environments?

10. Learning Environments
0Participants are University of Florida students
enrolled in Beginning Spanish 1 (avg. age = 20)
0L1 English
0No other L2s
0No prior Spanish instruction
0Participants belong to one of 3 environments:
0Classroom (C): (N=13)
0Rosetta Stone (RS): (N=20)
0Classroom+Rosetta Stone (RS+C): (N=14)

11. Participants
Control group (C)
0N=13
0In-tact section of Beginning Spanish section
0Follow traditional syllabus with standard
materials
0Carry out standard classroom assessment
materials
0Meet with researcher 3x during semester

12. Participants
Rosetta Stone group (RS)
0 N=20
0 Not required to attend any regular class
0 Rosetta Stone package (“Conversational Spanish”)
0 16-week course designed to cover material comparable to a face-
to-face beginning class
0 6 units of Rosetta Stone® Version 4 TOTALe® Spanish, each
contains 4 lessons [Level 1, half of Level 2]
0 At least 6 Rosetta StudioTM sessions
0 At least 8 hours in Rosetta WorldTM
0 Monitoring of program access and time on task
0 Follow predetermined deadlines in progressing through the
material
0 Meet with researcher 3x during semester

13. Participants
Classroom + Rosetta Stone group (RS+C)
0N=14
0In-tact section of Beginning Spanish class
0Same instructor as control group
0Using Rosetta Stone materials as the textbook
(including all features described for RS group)
0Meet with researcher 3x during semester

14. Rosetta Stone interface

15. Rosetta Stone interface
(vocabulary)

16. Rosetta Stone interface
(grammar)

17. Rosetta Stone interface
(pronunciation)

18. Rosetta Stone interface
(World – “play”)

19. Rosetta Stone interface
(World – “talk”)

20. Rosetta Stone interface
(World – “explore”)

21. Rosetta Stone interface
(Studio)

22. Data collected
0Assessment of attitudes
0January, May
0Discussion of experiences
0January, March, May
0General oral and written proficiency and skills
0January, March, May
0General language skills – partial CLEP test (30 items)
0 May

23. Sample CLEP test items

24. Data collected
0Assessment of attitudes
0January, May
0Discussion of experiences
0January, March, May
0General oral and written proficiency and skills
0January, March, May
0General language skills – partial CLEP test (30 items)
0 May
0 Oral proficiency – Versant test
0 May

25. Versant proficiency test

26. Versant proficiency test
0 Instrument has been tested for reliability and validity

27. Results
and preliminary observations

29. Results: CLEP test
Average scores (%)
10.00
25.00
40.00
55.00
70.00
CTRL RS RS_C
BoxPlot
Group
PercentageinCLEPtest
41.05 36.49
44.00

30. Results: CLEP test
Statistical analysis
Source Term DF Sum of
Squares
Mean
Square
F-
Ratio
p-value
Between groups 2 486.4109 243.2055 2.77 0.074158
Within groups 43 3781.544 87.94288
Total (Adjusted) 45 4267.955
0 F(2,43) = 2.77, p = 0.074
0 No significant interactions at alpha = 0.05
0 Groups performed statistically similarly on CLEP test.

31. Results: Versant test
Component scores (20-80)
Sentence
mastery
Vocabulary Fluency Pronunciation
OVERALL
PROFICIENCY
Control 35.55 23.73 35.73 42.00 34.64
Rosetta Stone 31.25 27.00 40.20 43.90 35.20
Rosetta Stone + class 29.42 24.83 37.25 42.58 32.75
20.00
30.00
40.00
50.00
60.00
70.00
80.00

32. Results: Versant test
Overall average scores (20-80)
20.00
31.25
42.50
53.75
65.00
CTRL RS RSC
BoxPlot
Group
PercentageinOverallVersanttest

33. Results: Versant test
Statistical analysis
Source Term DF Sum of
Squares
Mean
Square
F-
Ratio
p-value
Between groups 2 36.97765 18.48883 0.27 0.76296
5
Within groups 40 2646.802 67.86672
Total (Adjusted) 42 2683.78
0 F(2,40) = 0.27, p = 0.763
0 No significant interactions at alpha = 0.05
0 Groups performed statistically similarly on Versant test.

34. Does this mean Rosetta Stone
is just as good as the
classroom?
Not so fast…
There are other issues to consider!

35. Issue 1: Level tested
0Stages of SLA
0 Beginning levels marked
by lexical acquisition in
much greater
proportion to
morpho/syntactic
acquisition
0 Krashen and Terrell,
(e.g., the Natural
Approach, 1983)
recognized this as well
0RS does vocabulary well!
SOLUTION: Longer treatment period, encompassing greater acquisition;
also follow up beyond the treatment into next level(s)

36. Issue 2: (Unrelated) student attitudes
0Control group did not respond well to instructor’s
teaching style
0 Felt his expectations were too high
0 Emphasis on quizzes to demonstrate preparation
0 Perceived as disrespectful towards students
0 Affected control RS+class group also, but the negative
student attitude was much more contagious and
detrimental in the control group.
SOLUTION: ???

37. Issue 3: Assessment materials
0Goal was an objective test, not dependent
upon curriculum but sensitive to level and not
vulnerable to ceiling effects
0CLEP test
0May have picked too difficult an assessment!
0Versant test
0Although very good at low-level discrimination,
overall scores may not best reflect abilities
0 For example 

38. Sample VERSANT answer (ctrl)
0Muchos niños pasan horas cada día
practicando deportes. En su opinión, ¿es
bueno alentar a los niños a hacer deporte? Por
favor, aclare por qué si o por qué no.
(Many children spend hours each day playing
sports. Is it good to encourage children to play
sports? Why or why not?)
Sentence Vocab Fluency Pronunciation Overall
40 23 47 54 41

39. Sample VERSANT answer (RS)
0¿Cómo prefiere usted enterarse de las
noticias, por la radio o por el periódico? Por
favor explique su respuesta.
(When getting news information, would you rather
hear the information on a radio program, or do you
prefer to read the information in a newspaper?
Please explain why.)
Sentence Vocab Fluency Pronunciation Overall
20 27 31 32 24

40. Issue 3: Assessment materials
0Goal was an objective test, not dependent
upon curriculum but sensitive to level and not
vulnerable to ceiling effects
0CLEP test
0May have picked too difficult an assessment!
0Versant test
0Although very good at low-level discrimination,
overall scores may not best reflect abilities
0 For example 
SOLUTION: Investigate other objective means of assessment across
groups, suitable for lowest levels of acquisition.

41. Issue 4: Many more data!
0Next phase of analysis
1. Analysis of additional texts
0Written: patterns of errors, lexical use
0Audio: patterns of errors; lexical use; acoustic
analysis
2. Analysis of discussion of experiences, attitude data

42. Preliminary observations
from monthly meetings and discussions
0 Linguistic observations
0 All learners exhibit typical morphosyntactic errors common
to this level
0Number/gender agreement
0Verb conjugation
0Etc.
0 All learners evidence English-influenced phonology
0 Communicative skills are minimal across all groups
0But conversation is virtually impossible with RS-only group
0RS+class group comparable to control group
1

43. Preliminary observations
from monthly meetings and discussions
0 Attitudes and behavior
0 RS participants greatly enjoyed the program
0Flexibility, no need to attend class (RS only)
0 No problems with digital-only materials
0 Emerging concerns about continuing in Spanish next
semester (2nd half of two-semester sequence to fill the
language requirement)
0 RS students consistently do activities until they achieve
100% - this is not the pattern we see in our online workbook
in traditional classes. Why?
There are definitely lessons to be
learned as we develop digital
materials and activities…!2

44. Preliminary observations
from monthly meetings and discussions
0 RS+Class students enjoyed the materials far less than the
RS-only group:
0 Frustrated by lack of grammar explanations in RS
0 Sensed a lack of structure in materials
0 Instructor spent excessive class time on grammar
explanations, and trying to contextualize the RS material to
make it appropriate for conversational interaction…
In essence, they un-flipped the classroom!
3

45. Issue 4: Many more data!
0Next phase of analysis
1. Analysis of additional texts
0Written: patterns of errors, lexical use
0Audio: patterns of errors; lexical use; acoustic
analysis
2. Analysis of discussion of experiences, attitude data
3. How to assess cultural awareness?
SOLUTION 1: Keep analyzing! Follow up with these participants next
semester. Case studies as well as group trends.
SOLUTION 2: There is a need for a larger scale study –
more participants at different institutions, more levels of
Rosetta Stone program, longer duration, etc.

46. Thank you.
glord@ufl.edu
0Special thanks to:
0 UF College of Liberal
Arts and Sciences
0 Humanities
Scholarship
Enhancement fund
0 Carlos Enrique Ibarra
(instructor, statistician)
0 Caroline Reist and
Keegan Storrs (RAs)
0 Laura Bradley (Rosetta
Stone)

47. Works Cited
0 Bley-Vroman, R. (1988). “The fundamental character of foreign language learning.” In W. Rutherford & M.
Sharwood Smith (Eds.), Grammar and second language teaching (pp. 19-30). Rowley, MA: Newbury House.
0 DeWaard, L. (2013). “Is Rosetta Stone a viable option for L2 learning?” Forthcoming in ADFL Bulletin.
0 Godwin-Jones, R. (2007). “Emerging technologies; Tools and trends in self-paced language instruction.
Language Learning and Technology,” 11(2), 10-17. Retrieved 26 September 2012 from
http://llt.msu.edu/vol11num2/emerging/
0 Godwin-Jones, R. (2009). “Emerging technologies: Speech tools and technologies. Language Learning and
Technology,” 13(3), 4-11. Retrieved 26 September 2012 from http://llt.msu.edu/vol13num3/emerging.pdf
0 Krashen, S. D. & Terrell, T. D. (1983). The Natural Approach: Language acquisition in the classroom. Hayward,
CA: Alemany Press.
0 Lafford, B., Lafford, P. & Sykes, J. (2007). “Entre dicho y hecho …: An assessment of the application of
research from second language acquisition and related fields to the creation of Spanish CALL materials for
lexical acquisition.” CALICO Journal, 24(3), 427-529.
0 Nielson, K. B. (2011). “Self-study with language learning software in the workplace: What happens”
Language Learning and Technology, 15(3), 110-129. Retrieved 26 September 2012 from
http://llt.msu.edu/issues/october2011/nielson.pdf
0 Rovai, A. P. (2002). “Development of an instrument to measure classroom community.” The Internet and
Higher Education, 5, 197-211.
0 Santos, V. (2011). “Review of Rosetta Stone Portuguese (Brazil) levels 1, 2, & 3.”CALICO Journal, 29(1), 177-
194.
0 Stevenson, M. P. & Liu, M. (2010). “Learning a language with web 2.0: Exploring the use of social networking
features of foreign language learning websites.” CALICO Journal, 27(2), 233-259
0 Vesselinov, Roumen. Measuring the Effectiveness of Rosetta Stone.
http://resources.rosettastone.com/CDN/us/pdfs/Measuring_the_Effectiveness_RS-5.pdf