Professor Chris Tisdell, Scientia Education Academy Fellow at the University of New South Wales (...and YouTube star, mathematician, former DJ...) kicked off the day by talking student word choice, feedback, and psychology, and wellbeing. Chris presented findings from a national study which used the feedback from students from more than 20 universities. Why? After every Studiosity session, students give feedback. That feedback from students needs to be analysed and used in practical ways (especially recalling Associate Professor Phill Dawson on Day One, who discussed the importance of feedback literacy and translating it into action.) Online, 24/7 support is needed as much to fulfil student expectations for their overall university service experience, as it is needed for delivering learning outcomes. This year's Studiosity 'Students First' Symposium was hosted at La Trobe University City Campus, 25 and 26 July 2019.

1. Online writing feedback:
A national study exploring the
service and learning experiences
of students

2. Acknowledgements
• Joint work between: Brodie, Tisdell & Sachs
• Research supported through an Industry Network Seed
Fund from UNSW & Studiosity
• Ethics approval (HC180014) obtained via UNSW’s Human
Research Ethics Committee
• Dr Lesley Halliday co-wrote the associated ethics paperwork
• The second and third authors are members of the Studiosity
Academic Advisory Board.

3. Our work in a nutshell
• This case study is designed to capture the lived
experience of university students receiving online,
formative pre-submission feedback on their academic
writing.
• We investigate three key research questions on the
provision of feedback to students and use of technology:

4. Research Questions
• What do students perceive to be the characteristics
of a positive online writing feedback interaction?
• What changes (if any) are there to students’
perceptions of themselves or their work as a result
of the interaction?
• What, if any, other factors influence the interaction?

5. Ask the Audience!
• In your opinion, what are the feelings identified
by students when seeking help / feedback?
https://tinyurl.com/Studiosity2019

6. What students feel when seeking help
Most commonly occurring words were:
• anxious,
• busy,
• disheartened,
• isolated,
• lazy,
• nervous,
• pressured, silly, stressed, struggling, tense, worried.

7. Data & Approach
• The data for the study was gathered from a 12 month
period across multiple universities in Australia, New
Zealand, Singapore and the UK.
• We employed a manifest content analysis approach to
analyse comments volunteered by students at the
conclusion of an online feedback interaction.

8. What is Writing Feedback?
• Online, asynchronous service offering formative, pre-
submission feedback on academic writing to students
within a 24 hour time period.
• Students can choose to submit work for feedback on
English expression and style, structure, spelling,
grammar and/or referencing.
• The service is generally available online to students from
within an institution’s LMS.

9. What is Writing Feedback?
• Studiosity reviewers have tertiary qualifications, come from a
wide range of discipline backgrounds and have a strength in
academic writing.
• They do not provide comments on subject content or rewrite
the piece of work.
• Students are provided with a summary of actionable
strengths and weaknesses, including suggestions/examples
for improvement.

10. Our Data Sample
• 12 month period with 50,758 Writing Feedback
transactions
• 11,264 comments explored in more detail
• Satisfaction ratings for 99% of these transactions
using a 5-point Likert scale

11. Our Data Sample
• Students from more than 20 universities across all
states and territories of Australia
• Comments and ratings were made by students from
every Year/Level, Study Type and Study Mode.

12. Representation
Table 2: Comments by Year/Level Compared to Australian University Student Population (2018 First Half Year)
Year/Level % of Comments (11,264) % of Population (1,213,125)
Undergraduate 1st Year 42% 29%
Undergraduate All Years 82% 71%
Postgraduate Coursework 18% 27%
On Campus 51% 77%
Online (includes mixed mode) 49% 23%
Full-time 65% 75%
Part-time 35% 25%

13. Documents for Feedback
• The most common Document Type submitted was an
Essay (45%).
The significant differences in proportions were:
• Undergraduate 1st Years submitted a higher proportion of
Essays and Text Analyses
• Undergraduate 2nd and 3rd Years submitted a higher
proportion of Scientific Reports.
• Postgraduate students submitted a higher proportion of
Case Studies.

14. Documents for Feedback
Year/Level Comments Case
Study
Essay Scientific
Report
Short
Story
Speech Text Analysis Other
Pathway
Program
4% 4% 4% 3% 3% 4% 5% 5%
UG 1st Year 42% 37% 45% 33% 54% 46% 52% 42%
UG 2nd
Year
20% 23% 18% 26% 22% 27% 17% 19%
UG 3rd Year 15% 15% 15% 19% 9% 12% 11% 14%
Honours
Year
1% 1% 1% 4% 0% 0% 1% 1%
PG 18% 20% 17% 16% 12% 12% 14% 18%
Total
Number
11,264 1,665 5,035 691 74 26 259 3,514
% 100% 15% 45% 6% 1% 0% 2% 31%
Table 3 Proportion of Document Types submitted by Year/Level of the Student

15. Analysis
• The full dataset was imported into NVIVO12 to conduct a
manually coded manifest content analysis.
• many comments contained single words or expressions that
reflected an overall response only - with no further
information on why this was the case.
• Just ten words accounted for 28.5% of the word frequency
count.

16. Emerging Perspectives
• The comments can be summarised by a “generic
comment” constructed from all of these words:
“Thanks very much for the feedback, it was much
appreciated. This is a really great and timely service”
• The overall satisfaction ratings left by students are
consistent with this common response.

17. Student Ratings
For the whole dataset, the combinations of the ratings
• “5 Extremely Satisfied” and
• “4 Somewhat Satisfied”
accounted for 92% of the transactions (both with or without
comments).

18. Emerging Themes from Comments
• Attitude: positive, negative.
• Student and reviewer characteristics: knowledge, feelings,
communication, expectations.
• Characteristics of the content sought and received: accuracy, precision,
comprehensiveness.
• Experience of the technology and feedback process: timeliness, ease of
use and service provided.
• Outcomes of the feedback process: including desired changes

19. Student Feedback (on their experiences)
• From the word frequency count we see that some students
do identify Writing Feedback as a service (this word was
used 1,751 times).
Category Sub-category Sub-category Sub-category
Assurance Knowledge (responder and/or
student)
Coverage of the feedback:
comprehensive/precise
Content of the feedback:
examples/explanation
Empathy Human kindness/patience Clear communication
Reliability Accuracy Consistency Technology that works
Responsiveness Timeliness Specificity
Website and Tools Design Ease of use
Table 4 Service Experience Coding Categories and Sub-categories

20. Expectations
• Gronroos (1988) also identified that perceived service quality
is affected by how well the outcomes and experience match
the initial expectations of the customer.
• Suggests a “customer” brings an active set of knowledge,
skills, prior experiences and perceptions to a transaction.
• The words “expected” and “surprised” occurred frequently
enough in the comments word count to suggest the inclusion
of a category for Service Expectations.

21. Themes in Learning Experiences
The most common words used by students to indicate a change
in their work or themselves in these interactions were related to:
• Confidence
• Improvement
• Understanding
• Reinforcement
These were grouped under the overall category of Learning
Experience.

22. Confidence
• The role that Confidence plays in the development of
competence (Chickering and Reisser, 1993) and motivation
(Schunk, 1991) is well documented.
• It is not surprising that this word occurred in the comments
more than 500 times.

23. Analysis of Comments
• Developing Confidence is not only important for commencing
or returning students, it also appears to be important as
students encounter different types of work at different stages
of their studies.
• A reported increase in Confidence is frequently linked in the
comments with Attitude: Positive Improvement, with students
indicating they will also apply what they have learned to both
current and future work

24. Analysis of Comments
• The reported changes most commonly link increasing
Confidence with Improvement to current and future work and
to greater Understanding of the academic writing process.
• The changes are also reported by students at different
stages of their studies and when undertaking new types of
written work.

25. Proportion of Comments
Table 5 Proportion of Comments coded at each Year/Level for each of the high level Categories
Year/
Level
Comments Positive
Attitude
Negative
Attitude
Positive
and
Negative
Attitude
Service
Expectations
Service
Improvements
Service
Experience
(SE)
Learning
Experience
(LE)
Pathway
Program
4% 4% 4% 3% 1% 2% 4% 5%
UG 1st
Year
42% 43% 37% 37% 41% 39% 39% 45%
UG 2nd
Year
20% 19% 23% 28% 20% 18% 21% 18%
UG 3rd
Year
15% 15% 16% 11% 15% 12% 16% 16%
Honours
Year
1% 1% 1% 0% 1% 1% 1% 1%
PG 18% 18% 20% 22% 19% 27% 19% 16%
Total
Number
11,264 9,740 1,767 243 327 367 3,401 1,825
Total % 86% 16% 2% 3% 3% 30% 16%

26. Analysis of Comments
• Proportion of Attitude: Positive codes was high, consistent
with the proportion of high satisfaction ratings found in the
quantitative analysis.
• The proportion of Service Experience codes assigned was
almost double the proportion of Learning Experience codes.
• Undergraduate 1st Year student comments accounted for a
higher proportion of Learning Experience codes and a lower
proportion of the Service Experience codes.

27. Coding of Comments
• Postgraduate students accounted for a higher proportion of
Service Improvements codes.
• Undergraduate 2nd Years accounted for a higher proportion
of codes that are both Attitude: Positive and Negative
• The codes assigned for Website Design and Tools (237
comments) were mostly associated with suggested Service
Improvements.
• The codes assigned to Ease of Use (120) were 80% Attitude:
Positive. The Attitude: Negative were mostly associated with
requests for Service Improvements.

28. Service Experience
Table 6 Proportion of Comments coded at each Year/Level for each of the Service Experience Sub-
categories
Year/Level Comments Assurance Empathy Reliability Responsiveness % of Comments
Pathway Program 4% 3% 5% 2% 5% 4%
UG 1st Year 42% 40% 41% 36% 38% 39%
UG 2nd Year 20% 21% 20% 21% 21% 21%
UG 3rd Year 15% 17% 16% 17% 16% 16%
Honours Year 1% 1% 1% 1% 2% 1%
PG 18% 18% 17% 23% 20% 19%
Total Number 11,264 1,850 338 317 896
% of SE Comments 54% 10% 9% 26%

29. Learning Experience
Table 7 Proportion of Comments coded at each Year/Level for each of the Learning Experience Sub-categories
Year/Level Comments Confidence Improvement Reinforcement Understanding % of LE
comments
Pathway
Program
4% 6% 4% 4% 4% 5%
UG 1st Year 42% 48% 41% 45% 45% 45%
UG 2nd Year 20% 17% 17% 21% 16% 18%
UG 3rd Year 15% 13% 18% 17% 15% 16%
Honours Year 1% 1% 0% 1% 0% 0%
PG 18% 15% 19% 12% 18% 16%
Total Number 11,264 547 492 385 401
% of LE
Comments
30% 27% 21% 22%

30. Analysis of Comments (Learning Experience)
• Proportion coded for Confidence was higher than for the other
sub-categories.
• Proportion of Confidence codes was highest for Pathway
Programs and Undergraduate 1st Years.

31. Analysis of Comments: Learning Experience
• Comments that mentioned “first time I have attempted a
piece of work of this type” provide an insight into why
Confidence appears in comments at all Year/Levels.

32. What students feel
Most commonly occurring words were:
• anxious,
• busy,
• disheartened,
• isolated,
• lazy,
• nervous,
• pressured, silly, stressed, struggling, tense, worried.

33. More on Feelings
• This is consistent with the work of Kahu and Nelson (2018)
and Chong (2018), who emphasise the importance of a
student’s emotional response as part of their experience.
• An Attitude: Positive code for Empathy was closely related to
fewer feelings of isolation and to increasing Confidence.

34. Anslysis of Comments (Service Experience)
• The number of comments coded to Service Experience
categories was almost double (1.86 times) the number of
comments coded to Learning Experience categories.
• One probable reason is that the Service Experience
categories are multifaceted
• Table 8 shows how some of these sub-categories help to
explain the differences in emphasis found in the coding
results by different types of students in relation to what
contributes to a positive feedback experience.

35. Analysis of Comments (Service Experience)
Table 8 Service Experience Sub-Categories and Emphasis by Student Type
Category Sub-category More Important for Student Type:
Assurance Knowledge (responder and/or student) Postgraduate and
Undergraduate 3rd Year
Coverage of the feedback: comprehensive/precise All
Content of the feedback: examples/explanation Undergraduate 1st Year
Non-native English speaker and/or lacking
knowledge of English grammar
Empathy Human kindness/patience Undergraduate 1st Year
Non-native English speaker and/or lacking
knowledge of English grammar
Clear communication Non-native English speaker or lacking knowledge of
English grammar
Reliability Accuracy All
Consistency Postgraduate and
Undergraduate 3rd Year
Technology that works All
Responsiveness Timeliness All
Specificity Undergraduate 1st Year
Website and Tools Design All
Ease of use All

36. Wrapping things up
• This case study supports the position that the characteristics of
a positive writing feedback interaction can be described using
the dual and complementary lenses of service and learning
• Student perceptions of a feedback interaction may also be
influenced by their own personal characteristics and feelings
and by factors in their institutional environment.
• Changes reported by students to their feelings and their
proposed application of what they have learnt support the view
that a positive feedback interaction is also an active one.

37. Addressing our Research Questions
• Students perceive online writing feedback
interaction as a service and learning experience
• Changes in self-confidence or confidence regarding
their work were a result of the interactions
• Other factors such as expectations and institutional
settings also influence an interaction.

38. Future Work
• Future comment analyses may benefit from access to data
from more universities and data containing more contextual
information for the students, such as gender.
• It would also be interesting to explore ways in which more
students could be encouraged to report why they found their
feedback interaction to be a positive or negative experience.
• This above would facilitate a deeper understanding of the
lived experience of students and assist in the provision of
more effective writing feedback, where applicable,
throughout their studies.