This presentation reports the findings of an empirical investigation which set out to test a set of rainbow exercises. The rainbow diagrams are pictorial representations of formal graphs that are derived automatically from student essays. They were designed to allow students to discover how key concepts in a well written essay are connected together. The students would then be able to compare a rainbow diagram of their own essay with a good essay and make changes to it before submission to their tutor. A trial was undertaken with academics, teaching and learning staff, doctoral students at The Open University of Catalonia and the Open University UK, before implementation into the web application known as OpenEssayist.

1. Designing and Testing Visual
Representations of Draft Essays for
Higher Education Students
Denise Whitelock 
, Debora Field 
, Stephen Pulman
, John Richardson  ,
 Institute of Educational Technology – The
Open University
 Department of Computer Science –
University of Oxford

2. Learning Analytics and Student Feedback
• Assessment drives
learning (Rowntree,
1987)
• Communicating
assessment in a
meaningful way
• What, when and
How?
DMW LAK Workshop March 2014

3. SAFeSEA
Professor Denise Whitelock
Professor John Richardson
Professor Stephen Pulman
An automated
tool
supporting
online writing
and
assessment
of essays
providing
accurate
targeted
feedback
SAFeSEA: Supportive Automated Feedback for Short Essay
Answers http://www.open.ac.uk/researchprojects/safesea/
DMW LAK Workshop March 2014

4. DMW LAK Workshop March 2014
Grand Challenge representing analysis that
can be readily understood

5. Feedback to prompt Self Reflection
• Analysis must prompt
“Advice for Action”
• Self reflective discourse
with computer feedback
• Visual representation of
feedback can open a
discourse between tutor
and student
• Prompt peer to peer
discourse
DMW LAK Workshop March 2014

6. Don’t forget the Human Reaction
DMW LAK Workshop March 2014

7. DMW LAK Workshop March 2014
www.storiesabout.com
www.storiesabout.com/cr
eativepdp
c.mckillop@rgu.ac.uk

8. DMW LAK Workshop March 2014

9. Computer Feedback
• Non-
judgemental/objective
• Is not the tutor who
marks the final essay
• Feedback can be
accessed many times
• Can lead to contact with
tutor with more
questions
DMW LAK Workshop March 2014

10. Talk Back
• Checking
understanding by
‘talk back’
• Summaries in
OpenEssayist
• Key words = key
ideas
• http://www.open.ac.u
k/researchprojects/s
afesea/
DMW LAK Workshop March 2014

11. OpenEssayist: What it tells you
• The system’s focus is to present summaries of
students’ own work in different ways, to encourage
them to reflect constructively on what they have written.
• In other words Open Essayist tells them from its
analysis what are the most important or key points in
their essay. They can then think about whether that
was what they intended to emphasise in their essay.
If not then they can make the appropriate changes.
• A very important aspect of the OpenEssayist system is
that it will not tell students what to write, or how to
rewrite sections of their essay, or even what is correct
or incorrect in their essay.
DMW LAK Workshop March 2014

12. OpenEssayist: A tool for reflection
• Different views with distinct roles presented
• The purpose of this feedback is to encourage you
• to reflect on the draft text you submitted,
• to help you consider how your essay is
organised,
• how the key terms are being used across the
essay
• how they combine to form a cohesive discussion.
DMW LAK Workshop March 2014

13. OpenEssayist: How it gives feedback
• Three aspects of the students’ essays are analysed by
the system:
• the structure of the essay (which paragraphs
constitute the introduction, the conclusion, the
discussion sections, etc.),
• the key words and key phrases of their essay
(which are the most important words and phrases,
the ones that are most representative of the
essay's overall meaning)
• the key sentences of their essay (which are
whole sentences that are most representative of
the essay's overall meaning).
DMW LAK Workshop March 2014

14. ESSAY
GRAPHICS
ANALYSIS

15. Key word spring
DMW LAK Workshop March 2014

16. Chord diagram
DMW LAK Workshop March 2014

17. Key sentence adjacency matrix
DMW LAK Workshop March 2014

18. Middle Space
• Develop new
analytic methods
• Computational
• Representational
• Statistical
• Visualisation
DMW LAK Workshop March 2014

19. Visualisations as a Thinking Tool
• Visualising text (Bertin,
1981; Johnson et al,
1993)
• Free text visualisation
still problematic
• Do users need training
with visualisations?
• Convention vs.
Instruction?
DMW LAK Workshop March 2014

20. Short text for illustration of Rainbow
Diagrams
DMW LAK Workshop March 2014

21. Sentence graph of short text
DMW LAK Workshop March 2014

22. OU essay awarded high grade
DMW LAK Workshop March 2014

23. OU essay awarded low grade
DMW LAK Workshop March 2014

24. Stanford University Boothe Prize essay
DMW LAK Workshop March 2014

25. Pretend essay: 50 identical sentences
DMW LAK Workshop March 2014

26. Pretend essay: 10 identical paragraphs
DMW LAK Workshop March 2014

27. Participants for Empirical Study
Occupation University
Professor of Teaching & Learning OUC
Senior Lecturer Computer Science OUC
Administrator x 2 OUC
Professor OUC
Ph.D student x 6 OUC
Professor of Computer Science OUC
Vice Rector OUC
Senior Lecturer x 2 OU UK
Learning & Teaching Officer x 3 OU UK
Project Manager OU UK
OU student x 2 OU UK
Professor OU UK
Senior Administrator OU UK
Course Administrator OU UK
DMW LAK Workshop March 2014

28. Findings
DMW LAK Workshop March 2014
“One of the clues was talking about the way the
color nodes, that the red ones are at the end. If
they are a good connection with a good
explanation for each paragraph. They should be
connected together but the darker ones should
be in the middle. But you see the color groups
together so for me it automatically pulls my eye
to this page because all the colors are closer
together and more in the middle. So that would
be the student course assignment essay with
the highest mark.”
“You don’t show me
anybody’s text. You are
not revealing anyone
else’s essay. So
students cannot
plagiarize. But you are
saying ‘Look hang on,
this is the way this
essay connects
together.’ That’s what
telling a good story is
about this linking.”

29. Creating teaching and learning dialogues:
towards guided learning supported by
technology
• Learning to judge
• Providing
reassurance
• Providing a variety
of signposted routes
to achieve learning
goals
DMW LAK Workshop March 2014

30. Challenges
• Visualisation
• Social network analysis
• Communication and
collaboration,
• Discourse analytics
• Interpretation training
• What are the necessary
and sufficient conditions
for successful
implementation of
learning analytics?
DMW LAK Workshop March 2014

31. References
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J. (2013) ‘OpenEssayist: Extractive
Summarisation & Formative Assessment of Free-Text Essays’. Workshop on Discourse-Centric Learning
Analytics, 3rd Conference on Learning Analytics and Knowledge (LAK 2013), Leuven, Belgium
• Whitelock, D. (2011) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C.
McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global.
pp. 319–342.
• Field, D., Pulman, S., Van Labeke, N., Whitelock, D. and Richardson, J.T.E. 2013. Did I really mean that?
Applying automatic summarisation techniques to formative feedback. Proceedings of the 9th International
Conference Recent Advances in Natural Language Processing (Hissar, Bulgaria, Sep. 2013).
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013. What is my essay really
saying? Using extractive summarization to motivate reflection and redrafting. Proceedings of the AIED
Workshop on Formative Feedback in Interactive Learning Environments (Memphis, TN, Jul. 2013).
• Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J.T.E. and Whitelock, D. 2013. Using student
experience to inform the design of an automated feedback system for essay answers. Proceedings of the 2013
International Computer Assisted Assessment Conference (Southampton, UK, Jul. 2013).
• Field, D., Richardson, J.T.E., Pulman, S., Van Labeke, N. and Whitelock, D. 2013. Reflections on
characteristics of university students’ essays through experimentation with domain-independent natural
language processing techniques. Proceedings of the 2013 International Computer Assisted Assessment
Conference (Southampton, UK, Jul. 2013).
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013. OpenEssayist: extractive
summarisation and formative assessment of free-text essays. Proceedings of the 1st International Workshop
on Discourse-Centric Learning Analytics (Leuven, Belgium, Apr. 2013).
DMW LAK Workshop March 2014