This document summarizes a presentation on how subtitles are read and processed. It discusses key differences between reading subtitles and printed text, including the fleeting and dynamic nature of subtitles. It also outlines major eye movements in reading, factors that influence word fixation and skipping, and research measuring these behaviors for subtitles of varying speeds and text segmentation. The document then discusses how deaf, hard-of-hearing and hearing viewers may process subtitles differently and the role of subtitler expertise, particularly in live subtitling using respeaking.

1. AGNIESZKA
SZARKOWSKA
Audiovisual Translation Lab (AVT Lab)
Institute of Applied Linguistics
University of Warsaw
a.szarkowska@uw.edu.pl
@ASzarkowska
https://avt.ils.uw.edu.pl/
https://www.facebook.com/AVTLab/

2. Agnieszka
Szarkowska
Łukasz
Dutka
Monika
Szczygielska
Agnieszka
Walczak
Wojciech
Figiel
Gabriela
Flis
Magdalena
Oziemblewska
Sonia
Szkriba
Tomas
Senda
Anna
Jankowska

4. HOW DO WE WATCH
SUBTITLED FILMS?

7. HOW IS READING SUBTITLES DIFFERENT
FROM READING PRINTED TEXT?
▪ ‘Fleeting text on a dynamic background’
▪ The viewer has no control over
the presentation speed of subtitled text
▪ Competing sources of information
▫ Visual
▫ Auditory
▫ Verbal
▫ Non-verbal

8. MAJOR TYPES OF EYE MOVEMENTS
▪ Saccades
▫ Rapid eye movements from 10 to 100 ms
▫ Forward and backward (regressions)
▪ Fixations
▫ Moments when our eyes are relatively still
▫ 200-300 ms (when reading English printed text)

9. READING PRINTED TEXT
https://2prowriting.files.wordpress.com/2013/06/eyetrackingmeasures.png

12. WORD FIXATION PROBABILITY
▪ Word frequency
▪ Predictability
▪ Length
▪ Previous knowledge
(Brysbaert & Vitu, 1998; Rayner & Schotter 2012;
Rayner & McConkie, 1976)

13. WORD SKIPPING
▪ Carpenter & Just (1983)
▫ Lexical words – 83%
▫ Grammatical words – 38%
▪ Rayner (1998)
▫ Lexical words – 85%
▫ Grammatical words – 35%

18. NEW EYE TRACKER –
NEW OPPORTUNITIES
SMI RED 250
EYELINK 1000

20. EYE-TRACKING STUDIES ON SUBTITLING
▪ Subtitle characteristics
▫ Speed
▫ Text segmentation
▪ Viewer characteristics
▫ Hearing status
▫ Language proficiency
▪ Subtitler characteristics
▫ Professionals vs. trainees
▫ Translation/interpreting background

22. By Stefan_Kögl CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=466937

23. http://www.tvprophire.co.uk/crt-tv-hire

25. SUBTITLE
CHARACTERISTICS:
SUBTITLE SPEED
AND TEXT SEGMENTATION

26. HOW DOES SUBTITLE SPEED
AFFECT THE READING PROCESS?
▪ Fast subtitles
→ lower comprehension
→ less enjoyment
→ higher cognitive load
→ higher mean fixation duration
▪ Slow subtitles
→ more regressions and re-reading

27. ▪ Subtitle speed
▫ Fast (20 cps)
▫ Medium (16 cps)
▫ Slow (12 cps)
▪ 77 participants
▫ 30 English
▫ 21 Polish
▫ 26 Spanish

28. WHAT DID WE MEASURE?
▪ Comprehension
▪ Self-reported cognitive load
▫ Difficulty
▫ Effort
▫ Frustration
▪ Scene and subtitle
recognition
▪ Eye tracking
▫ Absolute reading time
▫ Proportional reading time
▫ Fixation count
▫ Revisits

31. F(2,114) = 89.53, p < .001, 𝜂 𝑝
2
= .61

32. Fast subtitles
▪ No negative effect on:
▫ comprehension
▫ scene recognition
▫ subtitle recognition
▫ reading experience
Slow subtitles
▪ Higher frustration
▪ Lower enjoyment
▪ More re-reading
▪ Higher mean fixation duration
WHAT DID WE FIND?

35. TEXT SEGMENTATION IN SUBTITLING
Syntactically
segmented
Non-syntactically
segmented

37. ““Line breaks have their value,
yet when you are reading fast,
most of the time
it becomes less relevant.”

39. HOW DO PEOPLE
READ SUBTITLES
ON DIFFERENT DEVICES?

40. READING ACROSS DEVICES
EYE TRACKER

42. VIEWER CHARACTERISTICS:
DO DEAF, HARD OF HEARING
AND HEARING VIEWERS
PROCESS SUBTITLES DIFFERENTLY?

43. VIEWERS WITH HEARING LOSS
▪ Types of users
▫ The Deaf vs. the deaf
▫ The hard of hearing
▪ Degrees of hearing loss
▫ Mild, moderate, severe, profound
▪ Onset of hearing loss
▫ Pre-, peri- and post-lingual
Deafness
and poor
reading skills
often coincide

44. “
Our objective is to serve
as great a number
of the audience as possible
by seeking a more common
denominator: a lower reading
level and reading speed.
Downey, G. J. (2008). Closed captioning: subtitling, stenography,
and the digital convergence of text with television.
Baltimore: Johns Hopkins University Press.
Philip Collyer from WGBH,
The first director
of the Caption Centre

45. EDITING DOWN SUBTITLES
▪ Vocabulary simplification
▫ Modernising archaic words
▫ Removing idioms, word play, etc.
▪ Syntax simplification
▫ Complex to simple sentences
▫ Passive to active voice
▫ Subject-verb-object order

47. “We thought we were doing
the right thing.
Jensema, C., & Burch, R. (1999). Caption Speed and Viewer
Comprehension of Television Programs

50. Deaf participant Hard of hearing participant

52. VERBATIM OR EDITED SUBTITLES?
▪ Equal access
▪ Censorship
▪ Language learning
▪ Residual hearing
▪ Lip reading

54. SUBTITLER
CHARACTERISTICS:
THE EFFECTS OF EXPERTISE

58. SUBTITLER
CHARACTERISTICS
IN LIVE SUBTITLING
WITH RESPEAKING

61. RESPEAKING

64. INTRA- AND INTERLINGUAL RESPEAKING
▪ Respeaking in education
https://www.youtube.com/watch?v=yTc7XEF0r58&t=2s
▪ Interlingual respeaking at conferences
https://www.youtube.com/watch?v=rGL0kJKfYPo
▪ Live events - The Witcher
https://www.youtube.com/watch?v=Oo0QZSd1yIY

65. ARE INTERPRETERS
BETTER RESPEAKERS?

66. SMI RED 250 + Emotiv EPOC plug-in

68. RESPEAKING CRISIS POINTS

70. INTERLINGUAL LIVE SUBTITLING COURSE
▪ ILSA Project
▪ ILSA MOOC (edx) course on UVigo platform
https://mooc.campusdomar.es/courses/course-
v1:ILSA+001+2019/course/
▪ ILSA Project YouTube channel
https://www.youtube.com/channel/UCSMUzZxvSS0Tqe3iBvJGO_g/videos

71. HOW DO BLIND
AND PARTIALLY SIGHTED
PEOPLE WATCH FILMS?
Audio description (AD)

73. (TEXT-TO-SPEECH) AD IN EDUCATION

74. AUDIO DESCRIPTION AS AN AURAL GUIDE

79. HOW TO MAKE
MODERN ART
ACCESSIBLE?
MODERN ART FOR ALL

80. DOWNLOAD
THE OPEN ART APP
Use it from home
or in the museum

84. CINEMA ACCESSIBLE FOR SENIOR CITIZENS

85. DO VIEWERS AVOID LOOKING
AT ACTORS’ MOUTH IN DUBBING?

86. WANT TO KNOW MORE?
https://avt.ils.uw.edu.pl/
https://www.facebook.com/AVTLab/

87. WORD FREQUENCY
RESEARCH
AND SUBTITLING?

88. SUBTLEX
▪ A corpus of subtitles used to estimate word frequencies
▪ "Subtitle-based word frequencies
are the best estimate of reading behaviour"
▪ SUBTLEX-US
▪ SUBTLEX-PL
▪ http://crr.ugent.be/programs-data/subtitle-frequencies

90. Brysbaert, M., & New, B. (2009). Moving beyond Kučera and Francis: A critical evaluation of current word frequency
norms and the introduction of a new and improved word frequency measure for American English. Behavior Research
Methods, 41(4), 977-990.

91. WHY SUBTLEX MAY BE PROBLEMATIC?
▪ Text condensation in subtitling
▫ Removing elements of spoken language, repetitions,
false starts, vocatives
▫ Simplifying grammatical structures
▫ Using shorter synonyms for brevity
▪ Low-frequency words - underrepresented
▪ Written-to-be-spoken-as-if-not-written

92. AMATEUR SUBTITLES
▪ Using fansubs!
http://opensubtitles.org
▪ Relying on data generated
by unqualified teenagers in their free time
▪ Copyrights – relying on illegal material

93. Brysbaert, M., & New, B. (2009). Moving beyond Kučera and Francis: A critical evaluation of current word frequency
norms and the introduction of a new and improved word frequency measure for American English. Behavior Research
Methods, 41(4), 977-990.