CWPA Presentation

1. Comparative Revision Writing:
Quantitative & Qualitative Research on Students’
Perspectives of Multimodal Instructor Feedback
Way Jeng
Tialitha Macklin
Mark Triana
Stacy Wittstock

2. 10:50-11:05 – Introductions & Ti Macklin
11:05-11:10 – Questions
11:10-11:30 – Mark Triana & Stacy
Wittstock
11:30 -11:35 – Questions
11:35-11:50 – Way Jeng
11:50-11:55 – Questions
11:55-12:05 – Overall Questions
Panel Overview

3. Study
Context

4. Response du Jour

6. Methods
o Fall 13 - Spring 14
o First-Year Composition
o 143 Student
Participants
o 3 Teacher Participants
o 9 Sections
o Written Control Groups
o Same Assignments
o Same Response Style
o Free Software for
Teachers & Students
o Adobe Reader
o Dropbox
o iAnnotate
o Pre- & Post-Study
Survey
o Collected One Set of
Essays

7. Student Response Preferences
11%
3%
55%
32%
0%
17%
2%
40%
26%
14%
10%
3%
64%
23%
0%0
0.175
0.35
0.525
0.7
Margins End Combo Conferences Audio
Pre-Study (N=120) Post-Study Audio (N=42) Post-Study Written (N=78)

8. Teacher Influence on Preferences
10
%
30
%
10
%
50
%
9%
63
%
23
%
5%
40
%
10
%
50
% Audio Group
21
%
79
%
21
%
65
%
14
%
4%
4%
60
%
32
%
Written Group
Teacher 2Teacher 1 Teacher 3

9. Student Narrative Feedback
Reasons for
Rejecting Audio
Response
o Unfamiliar
o Learning Styles
o Prevalence of Global
Commentary
o Time
o Technical Difficulties
Reasons for Selecting
Audio Response
o Clarity
o Learning Styles
o Prevalence of Global
Commentary

10. Overall Finding

11. But What About Revision?

12. Methods
o Fall 15 - Spring 16
o Same Data Set as Previous
Study (minus a few)
o 3 Comp Teachers (not part
of data set)
o 90 Essay Sets (redacted,
txt files)
o Removed Citation Pages
o Inter- Rater Reliability (4
essay sets)
o Training
o Google Docs
o Text Compare
o CWPA Grant
o Faigley, L., & Witte, S.
(1981). Analyzing Revision.
College Composition and
Communication, 32(4),
400–414.

13. Meaning Classifications
• Surface Change: No change to meaning at all. All Formal change types are
Surface changes.
• "You pay two dollars" → "You pay a dollar entrance fee.”
• "Because the horse lost his shoe the rider was lost." → "Because the
horse lost his shoe, the rider was lost."
• Local Change: These changes are fairly small. While they add, remove, or alter
information, the basic idea of the text remains the same. A summary
description of the text would not change, based on this revision.
• Global Change: These changes are substantial. They may have an effect on the
text globally. A summary of the text would change, based on this revision.

14. Type Classifications
- Addition: Make explicit what could earlier be inferred or was unknown
- Removal: Make implicit what was earlier explicit or make missing entirely
- Substitutions: Re-phrasing by using different words
- Permutations: Re-phrasing by using the same words
- Distributions: Material in one text unit is passed into more than one
- Consolidations: Material in multiple text units is compressed into fewer units
- Move: Material is moved whole
- Formal: Changes that affect only the form of text, and do not have the
potential to affect fundamental meaning. Often, these will be textual
features associated with a style guide or grammar rule
- Grammar: Grammatical elements (e.g., clause structure, punctuation) is
changed – syntactic structure

15. Coding Guide
Surface Local Global
Addition SA LA GA
Removal SR LR GR
Substitution SS LS GS
Permutation SP LP GP
Distribution SD LD GD
Consolidation SC LC GC
Move SM LM GM
Grammar SG LG GG
Formal SF

16. Rater Experience
&
Pedagogical
Implications

17. Coding Methodology
- Skimmed through each essay in the set
- Understanding the general gist of each essay helped with detecting
Global changes
- Worked in small, manageable samples, going paragraph by
paragraph in Text-Compare
- Worked consistently on the project as much as possible
- Long gaps between working on the project made it more likely that I
might change how I coded something
- Explained difficult coding decisions
- I felt that this might help the researchers understand coding
differences between raters

18. Text-Compare

19. Inter-Rater Reliability
- Similar institutional backgrounds
- All raters taught in the same program at WSU
- All raters were either current or former graduate students at WSU
- All raters experienced the same professional and teacher
development efforts at WSU
- Training also fostered uniformity
- Practiced coding sample essay sets together
- Encouraged discussion of disagreements over coding decisions

20. Impact on Assessment
- Feedback during the semester—increased focus on Global
revision through screencast feedback
- Less is more!
- Teaching revision—increased focus on helping students to
understand the difference between Global vs. Local vs. Surface
changes
- Quality vs. Quantity in revision work
- Assessment—looking for Global vs. Local vs. Surface changes
- Did the student actually revise? Or did he/she proofread?
- What did I give feedback on? Basing assessment on what I actually
commented on.

21. Cultivating Peer Review Literacies
- Establishing a complex yet manageable vocabulary
- What is the value (or potential) of asking students to itemize,
compartmentalize, and classify their individual revisions as well as
those made by their peers?

22. Cultivating Peer Review Literacies
- Putting distinctions between individual revisions and
personalized revision strategies into practice
- How might we go about designing more focused, formative, and
diverse writing and peer review activities that target particular kinds
of revision (global, local, and surface)?

23. Cultivating Peer Review Literacies
- Constructing peer review ecologies predicated on reflection and
meta-cognition
- To what extent can we frame peer review as a complex genre of
writing, governed as much by rigid conceptual frameworks as
subjective or holistic responses to writing as a whole?

24. Statistical Analysis
&
Overall Findings

25. A Brief Note About Inter-Rater
Reliability
This study used 3 different coders who looked at essays to produce
data
- What if they disagreed?
- Inter-rater reliability (agreement) was measured by having all 3
raters look at several essays, then comparing their coding to each
other
- Numerically, this is reported as Cronbach’s Alpha, .962
- Generally consider a very high value
- Intraclass correlation measured at .895
- Suggests that there were patterns to the coding of each paper, meaning that
it is like that we are observing a true construct
- Because Inter-rater reliability is so high, this study treats the coders
as effectively identical for purposes of statistical analysis

26. The Elephant in the Room
What about comparative
statistics?
- T-tests?
- Statistical significance?
The short version: There are
no statistically significant
differences between
conditions on any variable
we have tested so far

27. Case Study: Word Count
How much did word count change from first draft to second?
- Looking at all students, across all teachers
- Written condition: Mean=120.93, SD=277.64, N=40
- Verbal condition: Mean=219.42, SD =290.85, N=50
- t stat=1.672, t-critical (2-tail)=1.987
- p value (alpha)=0.098
- There is an almost 10% chance that the difference in the means is purely
the result of chance

28. But How Can that Be Chance?
The word count change was almost double in the verbal condition!
- But variance (standard deviation) was SUPER high
- Essay 3299: 2390 -> 3097 (+707 words added total)
- Unusual, but not unheard of for a final portfolio
- Essay 3329: 2971 -> 2235 (-736 words removed total)
- Probably an essay that had a lot of unnecessary parts
- Removing these words might have indeed been a good idea!
What does this mean?
- Sometimes an observed difference can be the product of random
chance
- Even a fair coin will have streaks
- There’s almost a 10% chance that the difference in these data is an
illusion

29. What is a Standard Deviation?
A measure of variance
- How much data vary
- “3, 3, 3, 3” vs “1, 1, 5, 5”
- 2/3s of all samples will fall within 1 standard deviation of the
mean
- Plus or minus
Why are they important?
- A high standard deviation indicates that students are revising in
very different ways

30. So, What Does This Mean?
Which form of feedback is better?
- Right now, there is no clear answer
- Student revisions are roughly equivalent
- With the high variance in the data, small effects are difficult to
detect
- Further research with larger data collection is advised
- Teachers can/should use whatever they like better
- Teachers with a preference aren’t clearly disadvantaging students
Do student preferences line up with improved performance?
- Numbers of students with a clear preference for verbal (audio)
feedback were simply too rare in this sample to say either way

31. So, What Does This Mean?
Why are revisions so varied?
- Probably because the reasons to revise are so varied
- No single kind of revision accounts for
- No single pattern of revision can (or should) be seen as “right”
This matches the writing process itself
- If the writing process is highly contextual and individualistic,
then the revision process will be similarly contextual and
individualistic

32. But Didn’t You Find Anything?
So far, only 1 statistically significant relationship has been found
- Instructor 3’s students exhibited lower word count change in
the audio condition, as compared to the written
- Mean audio: 6.17% (N=22, SD= 12.4)
- Mean written: 22.2% (N=20, SD= 8.6)
Were there any other effects for Instructor 3’s students?
- So far, no
- Analysis of Instructor 3’s student revision for type of revision has
not shown any other statistically significant differences
- The kinds of revision were similar, but the size of revisions was
smaller

33. What Does That Mean?
Possibly nothing
- Instructor 3’s students may have been outliers
- There’s still almost a 3% chance this is not a real effect
- A larger data collection would really help
Possibly, it means instructors shouldn’t be forced to use a certain
style
- Instructor 3 disliked audio feedback
- Maybe the students made smaller revisions as a result
- Maybe aversion has more of an effect on pedagogy than
preference

34. What Does This Mean for Future
Research?
Revision research requires massive data collection
- This study collected 90 essays from 3 instructors, yet samples
were still too small
- Probably 400-500 essays is the minimum for a study of this
scope
Institutional support is needed
- Coding this quantity of data is very labor intensive
- Getting enough instructors and students to participate is
difficult

35. Overall Finding

36. Comparative Revision Writing:
Quantitative & Qualitative Research on
Students’ Perspectives of Multimodal
Instructor Feedback
Ti Macklin – Tialitha.Macklin@csus.edu
Mark Triana – Mark.Triana@wsu.edu
Stacy Wittstock – Snwittstock@ucdavis.edu
Way Jeng – Wjeng@pierce.ctc.edu