The document summarizes Ellen Head's presentation on investigating teachers' perceptions of fluency. It discusses definitions of fluency, comparison studies of fluency perceptions between Italian and UK teachers, and Head's own study of fluency perceptions among professors and ELT professionals in China. Key points included differences found between native and non-native speaker teachers, variables rated as important to fluency, whether untrained raters or robots can assess fluency, and implications for language assessment.

1. Investigating Teachers’ Perceptions of Fluency
11/25/2018 6.20pm Rm 906
Ellen Head, Miyazaki International College

2. • Why “perceptions of fluency”?
• Definitions of fluency: cognitive, utterance, acoustic
• Comparison study looking at ideas of fluency in Italy based teachers and UK
based teachers. (Dore, 2016)
• My study: Professors and ELT professionals in China. (2016)
• Putting these findings in context with other people’s
published research
• Can untrained raters rate?
• Can robots rate?
• Do big words make you fluent?
• Does fast-talking make you fluent?
• What is the glue?
• Implications?

3. Fluent: Latin “fluens” flowing
• Native speaker fluency (Fillmore 1979)
Speaking without unnecessary pauses
Coherent and semantically dense
Appropriate to social context
Creative

4. More about perceived fluency
• “the speaker’s ability to focus the listener’s attention on his/her
message by presenting a finished product, rather than inviting the
listener to focus on the working of production mechanisms.” (Lennon
1990)
• Includes many variables including non-temporal variables such as
intonation – and affective factors.

5. “The rapid, smooth, accurate, lucid and efficient
translation of thought or communicative intention
into language under the temporal constraints of
on-line processing.” (Lennon 2000)

6. Defining Fluency: Segalowitz (2010)
• Cognitive Fluency: speech is being produced effortlessly
by the brain
• Utterance Fluency: the physical aspects of speech
production function well, and provide evidence of
cognitive fluency
• Perceived Fluency: the listener’s impression and the
inferences they make about the speaker’s cognitive
fluency based on their perceptions of utterance fluency

7. Perceived fluency
Utterance fluency
Cognitive
fluency

8. More about cognitive fluency
Concepts
and ideas
Lexicalization
Phonological
encoding
Three Step Model of Speech Processing in the Brain (based on Levelt’s
1989 model)

9. Rating Fluency: What can be measured
objectively?
Temporal factors
(Performance)
Strategic
competence
(Dealing with
cognitive load)
Knowledge
factors
(Linguistic
Competence)
Head (2018), after Dore (2016)

10. Perceptions of Fluency (Dore, 2016)
• Research Questions
• 1. What aspects of fluency do teachers pay attention to
when rating NNS fluency?
• 2. Are there differences between NS and NNS teachers?
• 3. Are teachers’ perceptions correlated with training and
years of experience?
• Dore compared 24 UK-based teachers and 24 Italy-based
teachers
• Qualitative study with UK-based teachers
• Survey including quantitative study of aspects of fluency,
with participants scaring components of fluency derived
from focus group discussion and open-ended questions

11. Fluency variables (Dore. 2016)
• Fluency
• Pauses
(number/length)
• Hesitations
• Length of runs
• Speech rate
• Repetitions
• Fillers
• Automaticity
• Effortlessness
• Complexity
• Grammatical
complexity
• Variety of vocab
• Topic/content
complexity
• Structural
complexity
• Phonology
• Accent
• Native-like
rhythm
• Global
• Global
proficiency
• Communicative
competence
• Coherence
• Formulaic
sequences
• colloquialness

12. Fluency: non-temporal variables (Dore. 2016)
• Fluency
• Pauses
(number/len
gth)
• Hesitations
• Length of
runs
• Speech rate
• Repititions
• Fillers
• Automaticity
• Effortlessnes
s
• Complexity
• Grammatical
complexity
• Variety of
vocab
• Topic/conten
t complexity
• Structural
complexity
• Phonology
• Accent
• Native-like
rhythm
• Global
• Global
proficiency
• Communicati
ve
competence
• Coherence
and
cohesiveness
• Formulaic
sequences
• colloquialnes
s
www.britishcouncil.org

13. Dore (2015) UK and Italy-based teachers,
N=48
Minimum Maximum Mean Standard
deviation
Effortlessness
Coherence
No. and length of pauses
Automaticity
No. of hesitations
Length of runs
Rate of speech
Coping in soc. Situations
3
1
2
2
1
2
1
1
5
5
5
5
5
5
5
5
4.08
4.02
3.83
3.79
3.79
3.73
3.69
3.69
.710
.956
.753
.898
.898
.676
.748
1.133
www.britishcouncil.org

14. Perceptions of Fluency (Dore, 2016)
• Differences between Italy-based and UK-based teachers
• Italy-based teachers gave more importance to accuracy, accent,
colloquialness and communicative competence.
• UK-based teachers gave more significance to automaticity (large
effect size) and hesitations (slight effect).
• Negative correlation was found between length of training and
selecting the variables native-like rhythm and accent.

15. Head (2016)
- Based in China. Participants include NS and NNS working as
assessment consultants (n=7) and NS and NNS teachers teaching at
university (n=39)
- As in Dore’s study, participants were asked to listen to three speech
samples and rate them out of 7
- To answer open-ended questions about fluency
- To give scores for fluency variables
- Surveyed professors and examiners (N=11) in pilot study (2016) and
further 35 professors later in 2016

16. • Speaker 1: 120 words per minute: mean length of runs: 5
• Speaker 2: 75 words per minute: mean length of runs: 5
• Speaker 3: 59 to 77 words per minute: mean length of runs: 4
26/46 rater speaker 1 the most fluent.
13/46 rated speaker 3 the highest.
5/46 rated speaker 3 and 1 equal.
All were agreed speaker 2 was the weakest.

17. Fluency and Coherence in three
speech samples
• S1: At níght, round about 8 o’clock, he ^ arrived home/ waiting
for the news./ Wow/ He won the lottery prize he was SO
excited../ The following day ^ he went to the beach…
• S2: One day ^“takarakuji o atarimashita”^ so he get a lot of
money and he planning to have a one week holiday in Hawaii.^
an he did it.
• S3: And at 8 ^ 8..30pm he ^ was sitting on his couch ^ and and
watching TV /and was ^ hoping that his ticket / ^ that he wòn ^
and ^ which he did

18. Why did you choose speaker 1? (fluent lady)

19. Why did you choose speaker 3? (fluent kid)

20. Working with skewed speech samples
to probe judgements about fluency?
• Speaker 1: 120 words per minute: mean length of runs: 5
• Speaker 2: 75 words per minute: mean length of runs: 5
• Speaker 3: 59 to 77 words per minute: mean length of runs: 4
(Teacher
T1 etc) T1 T2 T3 T4 T5 T6 T7* T8* T9* T10*
speaker 1 5 best 5 5 4.5 5 6 7 7 6
speaker 2 4 least 3 3 4 3 2 6 3 4
speaker 3 6 mid 4 4 3.5 4 4 6 6 5

21. How professors rated the three samples:
(SILC professors N=35)
Speaker 1 Speaker 2 Speaker 3
Average 4.94 3.26 4.82
Range 2 - 6 2 – 4.5 2.5 – 6
SD 1.13 0.89 1.22

22. Pilot survey, teachers’ perceptions of fluency
(N=11)
Minimum Maximum Mean Standard
deviation
Effortlessness
Coherence
No. and length of pauses
Automaticity
No. of hesitations
Length of runs
Rate of speech
Coping in soc. Situations
3
1
2
2
1
2
1
1
5
5
5
5
5
5
5
5
4.08
4.02
3.83
3.79
3.79
3.73
3.69
3.69
.710
.956
.753
.898
.898
.676
.748
1.133

23. Follow-up study Teachers’ Perceptions of
fluency (N=35)
Minimum Maximum Mean Standard
deviation
Effortlessness
Coherence
No. and length of pauses
Automaticity
No. of hesitations
Length of runs
Rate of speech
Coping in soc. Situations
Accent
1
2
0
0
1
0
0
1
1
5
5
5
5
5
5
5
5
5
3.94
4.23
3.62
3.51
3.94
3.48
3.2
3.75
2.51
1.3
0.873
1.35
1.62
1.21
1.22
1.45
1.12
1.38

24. Rating Fluency
China-based respondents in pilot:
• Agreed on what fluency was NOT:
• Not “Colloquial” range 2/3 S.D. 0.4
• Not “Vocab” range 2/3 S.D. 0.5
• Not “Accent” range 1/3 S.D. 0.8
China-based respondents overall:
Agreed coherence was important range 1/5, mean 4, S.D. 0.9

25. Can untrained raters make good judgements
about fluency?

26. • Untrained raters’ judgments of fluency correlated with the number of
“pruned syllables” and with judgements of prosody by phoneticians.
• The samples were drawn from beginner-level students of English
(Mandarin speakers).
Derwing, Rossiter, Munro and Thompson (2004)

27. Can robots rate?
• Van Moere (2012), Berenstein, Van Moere & Cheng (2010)
“Psycholinguistic” approach to language testing: ability to repeat/ put
in order chunks of a sentence correlates quite well with measures of
fluency on other tasks and self-rating.
• Evanini et al. (2017) propose using hybrid system including automated
testing elements for K12 students in USA.

28. Does fast talking make you fluent?
• 12 syllables per run with 0.8 expected pause ratio is fluent enough
for highly proficient L2 speakers.
• (Expected pause ratio means number of expected pauses/number of
actual pauses, so NS will have expected pause ratio of 1, and
expected pause ratio of less than one means they pause more times
than expected, of more than one means they do not pause enough)
Park, Soohwan (2016)

29. Do big words make you fluent?

30. Iwashita & Vasquez (2013) An examination of discourse competence at
different proficiency levels in IELTS Speaking Part 2, IELTS Research
IELTS Speaking level
correlates with use of
Hyponyms for cohesion
but not with vocabulary
K-list level

31. Is vocabulary level related to perceived
fluency?
• Iwashita and Valasquez (2015)
• Discourse Competence in Part 2 at Different Proficiency in the IELTS
Speaking Test
- -Sample of 58 interviewees doing 2 minute speech, rated 5 to 7
- Found correlations between aspects of cohesion and coherence and the
rated level of fluency
- -Aspects included hyponymy (cohesion through using vocabulary
consisting of members of the same semantic group for example,
newspaper, headline, The Times, The Sports Section)
- NOT K- vocabulary level
- -Complexity of structuring content
www.britishcouncil.org

32. Is L2 fluency related to L1 personal speaking
style?
• Yes – research into duration of pauses between sense units found that
pause length in L2 was related to L1.
• (De Jong, Groenhout, Schoonen and Hulstijn (2015), Derwing, Munro,
Thomson and Rossiter (2009), Towell and Dewale (2005) and others.)
• Yes – research into articulation rate and speech rate showed L1 rate
was predictive of L2 (Bradlow, Kim and Blasingame (2017)

33. Fluency (Head, 2016)
• Listeners seem to make judgements in which the
presence of certain features compensates for the lack
of other features.
• Intonation
• Eye-contact compensate for
pausing etc
• Pause placement
• Cohesion
• Content complexity
• Interest
www.britishcouncil.org

34. Rating Fluency: Is perceived fluency subjective?
www.britishcouncil.org

35. Implications
• Creating communities of practice for language assessment
• Sharing criteria with students
• Transparency
• Openness about the divergence between formative and summative
assessment
Thank you for coming!