Presentation of an exploratory pilot project on using mobile devices for assessment of and for oral EFL skills at the PechaKucha session of the TISLID 2014 conference in Avila, Spain
1. Using mobile phones for
the assessment of and for
oral skills development in
secondary education.
Ton Koenraad
Hogeschool Utrecht, University of Applied
Sciences, Faculty of Education
TELLConsult
GLoCALL 2010
9. Why this Project?
Anglia: flexibility / assessment of
- sharing innovative speaking practice
materials
- delivery of formal language
assessments
English for Kids: mobile as infrastructure,
& schools assessment for:
washback effect of testing
10. Issues to be researched
general organisation
task and test design,
teacher competences
face & content validity aspects
system usability
Research design / instrument
development
11.
12. How it works
Teachers:
Personal online workspace to set spoken
questions/tasks.
Questions are easy to set online by using a
microphone or uploading audio files.
Audio player allows teachers to review spoken
work and leave feedback.
Over time teachers establish a digital portfolio
of student work.
12
13. How it works
Students:
Connect using:
Mobile phones
iPod Touch
Skype
Landline
Computer
Access spoken exercises & leave voice responses.
Connect with other students for role play.
Personal online workspace to store work, listen & receive
feedback.
Listen to exemplar and sample questions posted by teachers.13
14. Pilot
2 secondary ed. EFL teachers
2 Anglia member schools
Volunteer students (n= 20)
Assessment: asynchronous, interview
format
Oral presentation skills
15. Pre-Questionnaire:
Learner Profiles (2)
Aspect Group A Group B
Speaking Skills:
(Self reported)
Fairly good Good
Like speaking in
class
So, so Definitely
Actual speaking hardly Very frequently
Telecollaboration at
school
n/a Slightly more than
once
Tel. Experience
in projects n/a
Very occasionally
Tel. Experience
IRL
seldom seldom
16. Pre-Questionnaire:
Learner perceptions:
L2 in class & IRL
Aspect Group A Group B
L2 in lessons Once in 3
lessons
Practically every
lesson
Answers /Discussion Only now
and then
(very)
frequently
Pairwork Hardly ever sometimes
Use of English IRL 1. Chat in
games
2. Holidays
3. Skype
1. Holidays,
2. Chat in
games
3. Txt chat
17. Assessment of Oral skills
Aspect Group A Group B
Assessed Tasks in
2010
All: 1 All: 3.5
School reports: Oral
skills included?
35% ? 70% ?
Expectations:
Is tele-testing valid?
Yes: 35%
?: 55%
Yes: 60%
?: 40 %
Post: Valid Yes: 60% 40%
19. Post-Questionnaire (1)
Aspect Group A Group B
Technically OK? Yes So, so
Read
Instructions
Yes Sure
Different from
expectation
Yes Yes
Questions:
complexity,
speed,
loudness,
Hard to
remember,
Speed bit fast,
Not loud enough
Idem, but
Speed OK
20. Pupils’ Comments
Time constraint is unnatural
Was interrupted: new session needed
Retries: worries about costs
Questions could be louder
Questions: peer voice is more inviting
21. Teacher Perceptions
System usability
-System: fairly user-friendly
Topics
- More alignment with pupils’ interests might be
needed
Validity
-Content measured in time is less suitable as
criterion when no interaction is possible
- computer-based testing, as such, not perceived
as unusual or unfriendly.
- Retry option?
23. Teacher Perceptions
Teacher competencies:
- Knowledge of CEFR -> difficult, training
/ practice needed
- Evaluation categories (content,
accuracy, complexity, fluency.) useful;
scoring doable in one session.
But …would prefer a grading scale that
results in a CEF-level:
better match to Dutch current grade
system
24. Teacher perceptions
Implementation
- Use as practice material and preparation for
speaking test. Actual testing: rather face-to-
face
- Gives students the opportunity to practice
outside the classroom, extra practice
- Chances for providing individualised
feedback
- May help reduce anxiety of insecure & shy
students:
25. Conclusions / next steps
Improve briefing (demo, online tutorial) +
raise awareness implications of re-tries
Redesign questions (granularity)
Try-out alternatives:
- system access (landlines, computers)
- content aligned to syllabus / textbook
26. Conclusions
Pupils, teachers & management have
concerns about costs
Also found in other projects:
[…] cost to the end user is a major
consideration and can be a barrier to
successful uptake when using mobile
devices (Kukulska-Hulme & Shield, 2007)
27. Thank you for
your attention.
Comments, Questions?
www.koenraad.info
Ton.Koenraad@gmail.com
29. Literature selection
Collins, T. (2005). ‘English Class on the air: Mobile
Language Learning with CellPhones’, Proceedings of the
Fifth IEEE International Conference on Advanced Learning
Technologies (ICALT’05).
http://csdl2.computer.org/comp/proceedings/icalt/2005/23
38/00/23380402.pdf
Fallahkair, S., Pemberton, L. & Griffiths, R. 2007.
‘Development of a cross-platform ubiquitous language
learning service via mobile phone and interactive
television’. Journal of Computer Assisted Learning, 23 (4),
312-325.
Kukulska-Hulme, Agnes; Sharples, Mike; Milrad, Marcelo;
Arnedillo-Sanchez, Inmaculada and Vavoula, Giasemi
(2009). Innovation in Mobile Learning: A European
Perspective. International Journal of Mobile and Blended
Learning, 1(1), pp. 13–35.
30. Literature selection (2)
Naismith, L., Lonsdale, P., Vavoula, G. & Sharples, M.
(2004). ‘Literature Review in Mobile Technologies and
Learning’. FutureLab Report 11.
http://www.futurelab.org.uk/resources/documents/lit_revie
ws/Mobile_Review.pdf.
Sharples, M. (Ed.). (2006). Big issues in mobile learning.
Report of a workshop by the Kaleidoscope Network of
Excellence Mobile Learning Initiative, University of
Nottingham, UK.
Shield, Lesley and Kukulska-Hulme, Agnes (2008). Special
issue of ReCALL on Mobile Assisted Language Learning.
Cambridge University Press.
Thornton, P. & Houser, C. (2005). ‘Using mobile phones in
English education in Japan’. Journal of Computer Assisted
Learning, 21, (3): 217-228.
31. Pre-Questionnaire:
Learner Profiles (1)
Aspect Group A Group B
Group size 12 8
Girls 5 4
Boys 7 4
Years of English 3 3
Positive Attitude to
Learning English
2.42
STD: 0.51
3.11
STD: 0.33
Average score
at Secondary
6.37
STD: 0.9
7.6
STD: 1.7
Speaking Skills:
(Self reported)
6.58
STD: 0.9
7.55
STD 0.68
32. Pre-Questionnaire:
Learner Profiles (2)
Aspect Group A Group B
Like speaking in
class
2.67
STD: 0.78
3.78
0.44
Actual speaking 1.5
STD: 0.52
3.56
STD: 0.53
Telecollaboration at
school
n/a 2.0
0.0
Tel. Experience
in projects n/a
1.38
STD: 1.0
Tel. Experience
IRL
1.83
STD: 1.19
1.44
STD: 0.73
33. Pre-Questionnaire:
Learner perceptions: Oral L2 in
class
Aspect Group A Group B
L2 in lessons 2.1
STD: 0.50
5.0
0.0
Answers /Discussion 2.2 / 1.6 3.9 / 3.2
Pairwork 1.6 1.9
English IRL 1. Chat in
games
2. Holidays
3. Skype
1. Holidays,
2. Chat in
games
3. Txt chat
34. Assessment of Oral skills
Aspect Group A Group B
Assessed Tasks in
2010
All: 1 All: 3.5
Formal reports: Oral
skills included?
35% ? 70% ?
Is tele-testing valid? Yes: 35%
? : 55%
Yes: 60%
?: 40 %
Post: Valid Yes: 60% 40%
35. Post-Questionnaire (1)
Aspect Group A Group B
Likert scale Disagree 1 Agree 4
Technically OK? 2.43
STD: 1.13
1.8
0.84
Read
Instructions
2.8
STD: 0.7
3.0
0.7
Different from
expectation
2.7
STD: 0.76
2.8
1.3
Questions:
complex, speed,
loudness,
Not loud enough
Hard to
remember
idem
Speed OK
36. Post-Questionnaire (2)
Aspect Group A Group B
Likert scale Disagree 1 Agree 4
Answer time left 3.4
STD: 0.5
3.2
1.1
Expected Mark Just sufficient
STD: 1.4
O.K
1.1
Problem Topics Networks; 1M Euros Good School; Climate
Fun to do 2.8
STD: 0.4
2.4
0.9
More pleasant
without teacher
2.07
STD: 0.6
2.0
1.0