This document discusses reflective writing analytics (RWA), an approach to providing automated feedback on reflective writing assignments at scale. RWA was developed using a conceptual framework of reflection derived from literature. It analyzes student writing for rhetorical moves related to reflection using natural language processing. RWA was tested on pharmacy students, with most finding the feedback helpful and many revising drafts based on it. Limitations include the need for strong pedagogical framing and more discipline-specific feedback. Ongoing work aims to address these issues.

1. UTS CRICOS PROVIDER CODE: 00099F utscic.edu.au
REFLECTIVE WRITING ANALYTICS
FOR ACTIONABLE FEEDBACK
Andrew Gibson, Adam Aitken, Ágnes Sándor (XRCE),
Simon Buckingham Shum, Cherie Tsingos-Lucas, and Simon Knight
Andrew.Gibson@uts.edu.au • @andrewresearch

2. REFLECTIVE WRITING
What is reflective writing and why is it important?
Mental processing (Moon,1999)
Self referential bending back upon oneself
(Ryan,2013; Archer,2010)
A way of navigating, proceeding in the face
of uncertainty
(Reidsema et. al., 2010)
Also… Mezirow, Schön
Kat – Ponder (flikr)

3. EXAMPLE
It is hard to believe that I started placement almost three months ago. It has definitely been full of
experiences which have shaped my understanding of the role of a pharmacist. During my time at
Chester Square Pharmacy I was able to observe different sides of pharmacy including pharmacist-
patient interactions, retail, administration and the clinical aspects. To be honest my first thoughts
going into placement were negative. I dreaded the idea of having to interact with patients as well as
engaging with employees of the pharmacy. I felt that my lack of experience would cause an
inconvenience and I would leave a negative impression in front of the pharmacist and other
employees. However, I came to realise that my preceptor is an exceptional teacher and as the
weeks progressed I begun to look up to him as a mentor. Despite my lack of experience my
preceptor ensured I observed different counselling situations and even encouraged me to engage
directly with patients from the early days of placement. By allowing me to engage with patients I
have been able to build on important communication skills. My preceptor would also observe my
weaknesses and bring them to my attention so that each week we would work on overcoming my
weaknesses and turning them into strengths.

4. EXAMPLE

5. ACTIONABLE FEEDBACK
• Reflective writing requires a departure from objective analytical academic
writing for an external audience
• It is personal, subjective, and speaks primarily to self
• For students » adjusting to the style requires timely feedback
• For educators » feedback for large numbers of students is difficult
• How can students be given timely actionable feedback on their
reflective writing in a way that is scalable?

6. HUMAN != NLP FEATURES
• Computers can help with timely feedback
and address issues of scalability,
however…
• We need to avoid the assumption that
human understanding of reflective
writing can be directly represented by
computational features.
• We need a way of linking our
understandings of reflective writing within
learning to computational techniques for
the analysis of natural language.
Jim Cortez – Pondering (flikr)

7. REFLECTIVE WRITING ANALYTICS (RWA)
• Our approach to bridge learning features to computational text features:
1. Conceptual framework based on reflection and reflective writing
literature
2. Genre approach linking conceptual framework to high level rhetorical
(reflective) moves.
3. Construction of high level features from a combination of NLP
techniques (and low level tools).
4. Communication of high level constructs as feedback to writer
• More than a computational model… a complex of different ‘models’

8. CONCEPTUAL FRAMEWORK
Developed…
• From the reflection and reflective writing literature
• Through Psychological and Sociological lenses
• A framework from common dimensions of prevailing theory

9. Intention
What am I going to do
about it? In which
direction do I head next?
How do I want to
change? INTENTION
and PREDICTION:
change likely to lead to
certain future benefits.
MODALS : will can, hope
to
Integration
What impact will this
have on me and my
goals/aspirations?
CAUSALITY: Impact on
self in context
What other ideas could I
use to improve myself?
CITATION and
EVALUATION
(appraisal) of ideas
What do others suggest
is a way forward? How
do others address these
challenges? CITATIONS
and hedging (could,
might)
How can I learn from the
past? How can I learn
from other perspectives?
What perspectives are
best for me?
REFLECTING on
beginning of learning
experniece
Internalisation
What do these feeling
say about me?
CAUSALITY:
explanations for
impressions and feelings
How is this a problem?
DESCRIPTIVE
NARRATIVE. How does
this challenge me?
CAUSALITY: effect of
experience on self
What should I improve?
Why do I need to
improve? Causality:
explain changed belief or
approach, new
understanding,
competence,
empowerment
How can I improve?
CITATIONS used to
present a theory for self
improvement. PAST/
NOW contrast to show
change, learning,
Interpretation
What does it mean for
me? Why is it
significant?
COMPARATIVES
CAUSALITY,
REFLECTIVE
QUESTIONING,
DEFINITION (of issues)
Why do I feel this way?
FEELINGS as a
measure of expectations,
surprise, uncertainty,
lack of confidence etc.
Impression
What do I notice about
my situation? What is
happening to me and
around me? TEMPORAL
LINKS,
Thoughts Feelings Challenge Self critique Potential solution Learning opportunity
Martin et al
Luk 2008
Depth
Orientation Complication (or Problematisation AA) Resolution (or ACTION)
REFLECTIVE NARRATIVE (may be embedded in Subject Rubric Scaffold e.g Intro/conc; "Concrete Experience"/ "Abstract conc
Identify critical issues Analysis of Issue Recommendation and Solution of Issue

10. SIMPLIFIED FOR ACTIONABLE FEEDBACK
• The complexity of depth dimension
• The complexity of structural dimension
• Possible 30 indicators, or a least 12 possible feedback options
• Reduced to 3 + 1 primary indicators
• 3 Additional characteristics

12. REFLECTIVE MOVES

13. Conceptual framework
AWA categories

14. Tutor highlights* AWA highlights
*We thank Peter Jones from UTS to have provided us with his annotations of some student essays.

15. Rhetorical moves are conveyed by pre-defined
patterns of constituent concepts.
The constituent concepts are instantiated by words or
expressions in sentences which are linked by
syntactic dependencies.
Concept-matching framework
Ontology of concepts of reflection
CONTEXT
That early role-play felt distant and impersonal, as I made
a conscious effort not let my emotions interfere with the
job I had been given.

16. Concept-matching framework
Ontology of concepts of reflection
The constituent concepts are instantiated by words or
expressions in sentences which are linked by
syntactic dependencies.
Rhetorical moves are conveyed by pre-defined
patterns of constituent concepts.
CHALLENGE
I do not feel as though I have sufficiently developed my
practical knowledge, or phronesis (Schwandt 2007), to a
point where I can speak up and challenge my manager.

17. Concept-matching framework
Ontology of concepts of reflection
The constituent concepts are instantiated by words or
expressions in sentences which are linked by
syntactic dependencies.
Rhetorical moves are conveyed by pre-defined
patterns of constituent concepts.
CHANGE
In the future it would be beneficial for me to sit with them
and take notes on how they completed the task as this
would mean that I am still taking responsibility for my
role by learning how to face the challenge next time it
occurred.

18. Concept-matching framework
The constituent concepts are instantiated by words or
expressions in sentences which are linked by
syntactic dependencies.
Rhetorical moves are conveyed by pre-defined
patterns of constituent concepts.
Implementation in AWA: Xerox Incremental Parser
Transition to open source code (Stanford Parser)

19. RWA PLATFORM

20. AWA – UI CONSIDERATIONS
• Annotation levels, Icons, Textual
feedback
• Making the feedback actionable!

22. STUDENT USE – PHARMACY
• 59 Pharmacy students, 120 posts.
• Of the 52% of students that provided feedback on the software, 85%
considered it helpful
• 60% of students posted more than once
• 27% showed evidence of drafting
• Most draft modifications appeared to improve the quality of reflection
• General indications of actionability of the feedback

23. LIMITATIONS AND POTENTIALS
• Pedogogical framing was a significant factor in reception of the tool
• Difficulties with generalisation in a participatory design process
• Lack of specifics resulted in smaller than anticipated buy-in
• Ongoing work with specific subjects is addressing these issues
• Paragraph level feedback was problematic
• A redesign is likely to replace paragraph feedback with key discipline level
indicators
• A TAP redesign is making sectional division of text more flexible

24. THANK YOU AND QUESTIONS?
A big thank you to authors not present: Adam Aitken and Cherie Lucas
And also to other project partners:
Natalia Nikolova, Walter Jarvis, Alan Parr, Andy Leigh, Peter Jones, Jo
McKenzie, Rosalie Goldsmith, Susan Hoadley, Isabelle Benne , Sarab
Mansoor, Keenan Wilson, and Je Browi, Xiaolong (Shawn) Wang, and
Xerox Research Centre Europe (XRCE).
For more info on our work, see:
http://utscic.edu.au

25. STUDY WITH US @ CIC
utscic.edu.au/research/phd