This document summarizes a study on using learning analytics to foster metacognition and formative assessment in collaborative writing. The study developed a prototype called Uatu that surfaced real-time writing activity data from Google Docs. Researchers conducted a case study with 8 students over 15 weeks, collecting observation, interview, and artifact data. Key findings included how collocation, ephemeral contributions, and different types of "writing" impacted collaboration and assessment of contributions. The prototype was most useful for distributed/online contexts with ongoing multi-author work.

1. [ Learning Analytics for Collaborative Writing ]
[ A Prototype and Case Study ]
Brian McNely, Paul Gestwicki, J. Holden Hill, Phillip Parli-Horne, and Erika Johnson
Ball State University

2. [ Motivation ]
— Writing + Collaborative Knowledge
— Formative Assessment
∆ - Zone of Proximal Development in networked
environments

3. [ Analytics ]
Project questions:
— Do learning analytics related to collaborative writing
foster
greater metacognition among participants?
— Does such analytic data promote both instructor and
peer
opportunities for real time interventions as formative
assessment?

4. [ Analytics & Collaborative Writing ]
Affordances of Google Docs:
— Surfacing real-time collaborative writing activity
— Ample opportunities for interventions and meta-talk
— Suite of publicly available APIs
+ Docs Gdata API
+ Google Visualization API

5. [ Uatu Prototype | System Design ]

6. [ Uatu Prototype | Visualizations ]

7. [ Methods ]
Systematic qualitative case study over 15 weeks, using
ethnographic methods of fieldwork, including:
— classroom observations
— semi-structured and stimulated recall interviews
— out of class observations of collaborative activity
+ pair programming
+ collaborative writing
— documentary photography
— collection of participant artifacts
— triangulation across data types and instances

8. [ Data ]
Total data collected includes:
— 8 participants
— 20 classroom observations
— 24 semi-structured and stimulated recall interviews
— 14 out of class observations
— 70+ photographs
— 19 participant-produced written artifacts
+ including granular composition/revision history
captured by Uatu
— 42,000+ words of fieldnotes and analytic memos

9. [ Findings | Overview ]

10. [ Theme 1 | Collocation ]

11. [ Theme 2 | Ephemera ]

12. [ Theme 3 | What “Counts” as Writing? ]
— What kinds of practices meaningfully contribute to final
written deliverables?
— How do we measure who “writes” in collocated
collaborations?
— What role do other forms of writing work play in the
construction of final deliverables?
— How can such contributions be measured?

13. [ Implications & Opportunities ]
— Our prototype worked well, but it didn’t reveal much about
novice software developer’s writing-in-practice.
— The real utility of a system like Uatu is in:
+ Fully online learning environments
+ Distributed collaborative teams
+ Complex, ongoing, multi-contributor deliverables

14. [ Questions & Discussion ]
brian.mcnely@gmail.com | paul.gestwicki@gmail.com