Ishii, Y., & Sawaki, Y. (2018). A Mixed-Methods Investigation into Japanese EFL Learners’ Online Revision Activities during the Reading-into-Writing Task Completion Process. The 17th Symposium on Second Language Writing. Simon Fraser University.

1. A Mixed-Methods Investigation into Japanese
EFL Learners’ Online Revision Activities during
the Reading-into-Writing Task Completion
Process
Yutaka Ishii & Yasuyo Sawaki
Waseda University
yutakaishii@aoni.waseda.jp
Symposium on Second Language Writing
Vancouver, Canada
1

2. Acknowledgement
• This work was supported by Grant-in-Aid for
Scientific Research 16K02983.
• I would like to thank Kana Matsumura and Tatsuro
Tahara for their help with data coding and the
participants of this study.
2

3. Introduction
• Reading-to-write tasks (e.g., summarization) are
important task types in conducting academic work
in English (Rosenfeld, Leung & Oltman, 2001).
• In this project we examine characteristics of
Japanese university students’ performance on
summarization tasks. Our ultimate goal is to
develop materials for teaching summarization in
academic writing courses.
• This presentation focuses on the revision process
learners engage while completing a summary task
• Stimulated recall and keystroke logging are combined for
a fine-grained analysis of learners’ writing process
through data triangulation.
3

4. Previous Studies on L2 writing
task completion process
• Process features (Baaijen, Galbraith, & de Glopper,
2012; Barkaoui, 2016; Zhang, & Deane, 2015)
• Relationship between English language proficiency
and keyboarding skills (Barkaoui, 2015)
• Stimulated recall (Lindgren, & Sullivan, 2003;
Revesz, Kourtali, & Mazgutova, 2016)
• Revision process in Japanese (Fuse & Ishiguro;
Tanaka & Ishiguro, 2018)
• Integrated writing task completion process
(Barkaoui, 2015; Chan, 2017)
4

5. Research Question
• What task completion process do EFL learners in
Japan engage to respond to a summary task?
Specific focus:
Learners’ online revisions
5

6. Method (1)
• Participants
• Five undergraduate students majoring in English
language and literature at a private university in Tokyo
• All participants enrolled in a required academic writing
course for second-year students
• Materials
• Two summarization tasks (each based on a single source
text in English)
• Source texts were adopted from published Eiken test
forms for Grade Pre-1 (Eiken Foundation of Japan)
6

7. Method (2): Source texts
A. The War over
Antiquities
(3rd admin. 2014)
B. The Automated
Future
(1st admin. 2014)
Structure Argumentative
Length (words) 508 504
paragraphs/
sentences
4/23 4/24
Vocabulary
Level
6000 5000
Readability
(F-K Grade Level)
12.4 12.4
Summary length Approx. 80 words
7

8. Method (3)
• Procedure
• Each learner summarized one text in English (Approx. 80
words).
• Random assignment of participants to texts
• 40-minutes per task; dictionary use allowed
• The learner’s task completion process videotaped
• Keystroke logged via Writing MaetriX
• An observation sheet comprising six categories (Read text,
Annotate, Plan, Write, Review, & Revise) completed by observers
• Upon completion of the summary task, a stimulated recall
session was conducted in Japanese.
• Prompts:
• Video of the task completion process
• Replay of Keystroke logging data
• Completed summary response
• Test booklet with the participant’s notes (annotations and
outlines)
8

9. Method (4)
Video recording
Two observers took notes on
the observation sheet.
9
Test-taker
Online dictionary (Weblio)
Recording of keystroke logging

10. Method (5)
• WritingMaetriX (Kusanagi, Abe, Fukuta, &
Kawaguchi, 2014)
• A keystroke-logging program that can record, analyze,
and replay learners’ writing process
10

11. Output of process files
11
@

12. Output of product file
12

13. Replay mode of WritingMaetriX
13

14. Analysis mode of WritingMaetriX
14

15. Method (6): Online Revision Category
(based on Barkaoui, 2016)
Precontextual revisions
the changes made at the end of the
text produced so far
• Concept
• Form
• Typography
• Unclear
Contextual revisions
the revisions occurred when the writer
moves away from the point of
inscription to insert a new text or to
delete or substitute already written text
• Orientation :
• 1. Content
• 2. Balance
• 3. Organization
• 4. Language
• 5. Typography
• 6. Unclear orientation
• Linguistic Domain
• 1. Within-word
• 2. Within-sentence
• 3. Across sentences and above.
• 4. Unclear domain.
• Action
• 1. Addition
• 2. Deletion
• 3. Substitution
• 4. Reordering (words or clauses)
• 5. Unclear action 15

16. Example of precontextual revision
* In recent years, illegally imported antiques have been returned to their homeground.*
@ In recent years, illegally imported antiques have been returned to their homegroun.@
@ In recent years, illegally imported antiques have been returned to their homegrou.@
@ In recent years, illegally imported antiques have been returned to their homegro.@
@ In recent years, illegally imported antiques have been returned to their homegr.@
@ In recent years, illegally imported antiques have been returned to their homeg.@
@ In recent years, illegally imported antiques have been returned to their home.@
* In recent years, illegally imported antiques have been returned to their homet.*
* In recent years, illegally imported antiques have been returned to their hometo.*
* In recent years, illegally imported antiques have been returned to their hometow.*
* In recent years, illegally imported antiques have been returned to their hometown.*
* In recent years, illegally imported antiques have been returned to their hometown.*
* In recent years, illegally imported antiques have been returned to their hometown.*
* In recent years, illegally imported antiques have been returned to their hometown.*
16

17. Example of contextual revision
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. It is originally owned by Greece, but it was transfered to Britain for preservation. *
@ . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. It i originally owned by Greece, but it was transfered to Britain for preservation. @
@ . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. It originally owned by Greece, but it was transfered to Britain for preservation. @
@ . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. It originally owned by Greece, but it was transfered to Britain for preservation. @
@ . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. I originally owned by Greece, but it was transfered to Britain for preservation. @
@ . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. originally owned by Greece, but it was transfered to Britain for preservation. @
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. t originally owned by Greece, but it was transfered to Britain for preservation. *
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. th originally owned by Greece, but it was transfered to Britain for preservation. *
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. the originally owned by Greece, but it was transfered to Britain for preservation. *
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. they originally owned by Greece, but it was transfered to Britain for preservation. *
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. they originally owned by Greece, but it was transfered to Britain for preservation. *
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. they w originally owned by Greece, but it was transfered to Britain for preservation. *
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. they we originally owned by Greece, but it was transfered to Britain for preservation. *
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. they wer originally owned by Greece, but it was transfered to Britain for preservation. *
* . This is largly due to the treaty by UNESCO, but it is not perfect. One well-known example is the Elgin Marbles, a heritage of the Palthanon
in Athens, Greek. they were originally owned by Greece, but it was transfered to Britain for preservation. *
17

18. Method (7): Coding and inter-coder
agreement
• Authors and two doctoral students in the field of
applied linguistics coded the data based on the coding
scheme developed by Barkaoui (2016). All the data
were double-coded.
• First of all, coding manual was developed and authors
coded one essay individually. Through discussion, the
coding manual and coding categories were adjusted.
• On the basis of these criteria, the first author and two
doctoral students annotated the revision. The first
author annotated all the revisions and two doctoral
students coded two essays each independently.
• After first coding, inter-coder agreement was 52%. All
inconsistencies were resolved through discussion.
18

19. Results: Time Series Data of Learners'
Writing Process
Revision
First data input
Planning
19

20. Results: Descriptive Statistics of learners’
keystroke logging
Student1 student2 Student3 student4 Student5
Task A B B B A
Total
number of
words
84 75 80 75 94
First data
input (sec)
447.0 632.2 721.0 1390.6 893.9
Key input
(minute)
28.4 42.0 50.1 34.5 77.3
# Words
(minute)
5.57 3.41 3.64 4.61 4.57
Revision
(minute)
0.97 1.78 2.61 2.43 4.28
revision 0/7/22 0/24/53 0/45/89 0/16/96 0/79/151
20

21. Overall tendency of precontextual
and conceptual revisions
21
0
10
20
30
40
50
60
70
80
90
100
student 1 student 2 student 3 student 4 student 5
precontextual conceptual

22. Precontextual revisions
22
0
5
10
15
20
25
30
35
40
45
50
55
60
concept form typography
student 1 student 2 student 3 student 4 student 5
To paraphrase "to the
countries of origin", I
started to write
"homeground".
Because the meaning
of word was vague, I
searched for the
word
"kokyo(hometown)".
Then, I chose the
word "hometown".
(S1: 31)

23. Contextual revisions (Orientation)
23
Because I used the phrases
"negative impact" and
"negative influence" two times,
I substituted the phrase for
"one”. (S3: 125)
0
5
10
15
20
25
30
35
40
45
50
55
60
content balance organization language typography
student 1 student 2 student 3 student 4 student 5

24. Contextual revisions (Linguistic
Domain)
24
0
5
10
15
20
25
30
35
40
45
50
55
60
within-word within-sentence across sentences and above
student 1 student 2 student 3 student 4 student 5
Well, the word of my summary
approached 90 words. I thought
about what to delete. I used many
future tenses but I began thinking
that I don't have to use the future
tenses. Then, I started to delete "will".
(S3: 110)

25. Contextual revisions (Action)
25
I noticed that "marbles"
was referred to as "they".
Because I referred to
"marbles" as "it", I
corrected "it" to "they".
(S1: 57)
0
5
10
15
20
25
30
35
40
45
50
55
60
addition deletion substitution reordering
student 1 student 2 student 3 student 4 student 5

26. Discussion and Conclusion
• content and concept revisions are not frequently
observed.
• Characteristics of integrated writing task
• In the revision of linguistic domain, Barkaoui’s
(2016) research showed that within-sentence
revisions are more observed than within-word
regardless of task type, proficiency and keyboarding
skills. However, this study shows that within-word
revisions are more observed.
• Limitations of this study
• Small sample size
• Difficulty of coding revisions
26

27. References (1/2)
• Baaijen, V. M., Galbraith, D., & de Glopper, K. (2012). Keystroke analysis: Reflections on procedures and measures.
Written Communication, 29(3), 246-277.
• Barkaoui, K. (2016). What and When Second‐Language Learners Revise When Responding to Timed Writing Tasks on the
Computer: The Roles of Task Type, Second Language Proficiency, and Keyboarding Skills. The Modern Language Journal,
100(1), 320-340.
• Barkaoui, K. (2015). Test Takers' Writing Activities During the TOEFL iBT® Writing Tasks: A Stimulated Recall Study. ETS
Research Report Series, 2015(1), 1-42.
• Brown, A. L., & Day, J. D. (1983). Macrorules for summarizing texts: The development of expertise. Journal of Verbal
Learning and Verbal Behavior 22, 1-14.
• Chan, S. (2017). Using keystroke logging to understand writers’ processes on a reading-into-writing test. Language
Testing in Asia, 7(10), 1-23.
• De Larios, J. R., Manchon, R. M., & Murphy, L. (2006). Generating text in native and foreign language writing: A temporal
analysis of problemsolving formulation processes. The Modern Language Journal, 90(1), 100-114.
• Fuse, Y., & Ishiguro, K. (2018). The Intention of Self-Revision in JSL Learners' Writing Process: A Comparison between
Native Japanese Speakers and Advanced Chinese and Korean JSL Learners. National Institute for Japanese Language and
Linguistics Research Papers. 15, 17-42.
• Hijikata-Someya, Y., Ono, M., & Yamanishi, H. (2015). Evaluation y native and non-native English teacher-raters of
Japanese students’ summaries. English Language Teaching, 8(7), 1-12.
• Kusanagi, K., Abe, D, Fukuta, J, & Kawaguchi, Y. (2013). Visualizing writing process using a key-logging system: For
construct feedback to enhance autonomous learning. Paper presented at the 81st Spring Conference of the Chubu
Chapter, Japan Association for Language Education and Technology (LET), Tokai Gakuen University, Japan.
27

28. References (2/2)
• Lindgren, E., & Sullivan, K. P. (2003). Stimulated recall as a trigger for increasing noticing and language awareness in the
L2 writing classroom: A case study of two young female writers. Language Awareness, 12(3-4), 172-186.
• Plakans, L. (2015). Integrated Second Language Writing Assessment: Why? What? How?. Language and Linguistics
Compass, 9(4), 159-167.
• Plakans, L. (2009). The role of reading strategies in integrated L2 writing tasks, Journal of English for Academic Purposes,
8(4), 252-266.
• Plakans, L., & Gebril, A. (2012). A close investigation into source use in integrated second language writing tasks.
Assessing Writing, 17(1), 18-34.
• Révész, A., Kourtali, N. E., & Mazgutova, D. (2017). Effects of task complexity on L2 writing behaviors and linguistic
complexity. Language Learning, 67(1), 208-241.
• Rosenfeld, M., Leung, S., & Oltman, P. K. (2001). The reading, writing, speaking, and listening tasks important for
academic success at the undergraduate and graduate levels. Educational Testing Service.
• Sawaki, Y., Quinlan, T., & Lee, Y. W. (2013). Understanding learner strengths and weaknesses: Assessing performance on
an integrated writing task. Language Assessment Quarterly, 10(1), 73-95.
• Sawaki, Y. (2003). A comparison of summarization and free recall as reading comprehension tasks in web-based
assessment of Japanese as a foreign language (Unpublished PhD thesis). University of California, Los Angeles (UCLA).
• Tanaka, H., & Ishiguro, K. (2018). The Writing Revision Process by Learners of Japanese: An Analysis of Position and Type
of Revision by Chinese and Korean Learners. National Institute for Japanese Language and Linguistics Research Papers.
14, 255-274.
• Van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press.
• Zhang, M., & Deane, P. (2015). Process features in writing: Internal structure and incremental value over product
features. ETS Research Report Series, 2015(2), 1-12.
• Obunsha (Ed.) (2016). 2016 nendo eiken jun-1 kyu kako 6 kai zenmondaishuu (Test papers for the last six administrations
of the Eiken Grade Pre-1 test, 2016). Tokyo: Obunsha. 28