My PhD presentation for the learn3 project

1. The feedback loop revisited
Israel Gutiérrez
Universidad Carlos III de Madrid

2. Context
photo: uc3mcom at flickr

3. Context
Lab session practices in computer rooms
photo: uc3mcom at flickr

4. Context
Lab session practices in computer rooms
Active learning activities: guided and open ended
photo: uc3mcom at flickr

5. Scenario
photo: candicejeanl at flickr

6. Scenario
Multimedia Applications (AMM)
photo: candicejeanl at flickr

7. Scenario
Multimedia Applications (AMM)
Lab sessions are 50% of the lectures
photo: candicejeanl at flickr

8. Scenario
Multimedia Applications (AMM)
Lab sessions are 50% of the lectures
Work in pairs and in a proposed practice
photo: candicejeanl at flickr

9. Problems
photo: berkeleylab at flickr

10. Problems
Feedback to students: effective but inefficient
photo: berkeleylab at flickr

11. Problems
Feedback to students: effective but inefficient
Issues in the orchestration of the session
photo: berkeleylab at flickr

12. Problems
photo: hackny at flickr

13. Problems
Feedback to the class: efficient but ineffective
photo: hackny at flickr

14. Problems
Feedback to the class: efficient but ineffective
Students engagement
photo: hackny at flickr

15. Possible solutions

16. Possible solutions
Use a backchannel chat to improve communication

17. Possible solutions
Use a backchannel chat to improve communication
Conole G., and Alevizou P (2010) “A literature review of the use of Web 2.0 tools in Higher
Education”. A report commissioned by the Higher Education Academy.

18. Possible solutions
Use a backchannel chat to improve communication
Conole G., and Alevizou P (2010) “A literature review of the use of Web 2.0 tools in Higher
Education”. A report commissioned by the Higher Education Academy.
Yardi, S. “Whispers in the Classroom." Digital Youth, Innovation, and the Unexpected. Edited by
Tara McPherson. The John D. and Catherine T. MacArthur Foundation Series on Digital Media and
Learning. Cambridge, MA: The MIT Press, 2008. 143–164.

19. Possible solutions

20. Possible solutions
Improve orchestration at enactment by means of
lanterns for distributed awareness

21. Possible solutions
Improve orchestration at enactment by means of
lanterns for distributed awareness
Alavi, H. S., Dillenbourg, P., & Kaplan, F. (2009). Distributed awareness for class orchestration.
Learning in the Synergy of Multiple Disciplines -- Proceedings of the 4th European Conference on
Technology Enhanced Learning (pp. 211-225). Springer.

22. !"
Possible solutions
Paulo Blikstein"
!
#$%&'(")*"+,("-./("012$%15.'6"7,$8,"1==.74"'(4(1'8,('4"5."%."<18?"19/"@.'5,"$9"5$;
2 1 ( 1 ;("5'18?$9%"

23. !"
Possible solutions
Paulo Blikstein"
Learning analytics for assessment !
#$%&'(")*"+,("-./("012$%15.'6"7,$8,"1==.74"'(4(1'8,('4"5."%."<18?"19/"@.'5,"$9"5$;
2 1 ( 1 ;("5'18?$9%"

24. !"
Possible solutions
Paulo Blikstein"
Learning analytics for assessment !
#$%&'(")* P. (2011). Using learning analytics to assess students ’ behavior in open ‐ ended programming
Blikstein, "+,("-./("012$%15.'6"7,$8,"1==.74"'(4(1'8,('4"5."%."<18?"19/"@.'5,"$9"5$;
2 1 ( 1 ;("5'18?$9%"
tasks. Proceedings of the 1st International Conference on Learning Analytics and Knowledge 2011

25. Our solution
Impact for Impact for
Measures
students teachers

26. Our solution
One-way websocket notifications
Impact for Impact for
Measures
students teachers

27. Our solution
One-way websocket notifications
Improve attention and effectiveness of general feedback
Impact for Impact for
Measures
students teachers

28. Our solution
One-way websocket notifications
Improve attention and effectiveness of general feedback
Record student interactions and use learning analytics
Impact for Impact for
Measures
students teachers

29. Our solution
One-way websocket notifications
Improve attention and effectiveness of general feedback
Record student interactions and use learning analytics
Improve orchestration in enactment and provision of
efficient feedback
Impact for Impact for
Measures
students teachers

30. Our solution
One-way websocket notifications
Improve attention and effectiveness of general feedback
Record student interactions and use learning analytics
Improve orchestration in enactment and provision of
efficient feedback
Improve orchestration design for the following session and
acquiring evidences for summative assessment
Impact for Impact for
Measures
students teachers

31. Our solution
One-way websocket notifications
Improve attention and effectiveness of general feedback
Record student interactions and use learning analytics
Improve orchestration in enactment and provision of
efficient feedback
Improve orchestration design for the following session and
acquiring evidences for summative assessment
Engagement in lab sessions (support of other artifacts)
Impact for Impact for
Measures
students teachers

32. Our solution
One-way websocket notifications
Improve attention and effectiveness of general feedback
Record student interactions and use learning analytics
Improve orchestration in enactment and provision of
efficient feedback
Improve orchestration design for the following session and
acquiring evidences for summative assessment
Engagement in lab sessions (support of other artifacts)
Awareness, reflection and reaction
Impact for Impact for
Measures
students teachers

33. Our solution
One-way websocket notifications
Improve attention and effectiveness of general feedback
classON
Record student interactions and use learning analytics
Improve orchestration in enactment and provision of
efficient feedback
Improve orchestration design for the following session and
acquiring evidences for summative assessment
Engagement in lab sessions (support of other artifacts)
Awareness, reflection and reaction
Impact for Impact for
Measures
students teachers

34. classON

35. classON
aCtive Learning ASSessment
Orchestration & Notifications

36. classON
aCtive Learning ASSessment
Orchestration & Notifications
Enactment phase: on class

37. classON
aCtive Learning ASSessment
Orchestration & Notifications
Enactment phase: on class
Activate the class: improve orchestration

38. classON
aCtive Learning ASSessment
Orchestration & Notifications
Enactment phase: on class
Activate the class: improve orchestration
Activate the class: students’ engagement

39. classON

40. classON
Formative
assessment
and quick
feedback

41. classON
Formative
HTML5 &
assessment
web
and quick
technologies
feedback

42. classON
Formative
HTML5 & Attention &
assessment
web notification
and quick
technologies system
feedback

43. classON
Formative
HTML5 & Attention &
assessment
web notification
and quick
technologies system
feedback
Learning
analytics

44. classON
Formative
HTML5 & Attention &
assessment
web notification
and quick
technologies system
feedback
Orchestrating
Learning learning at
analytics enactment

45. classON
Formative
HTML5 & Attention &
assessment
web notification
and quick
technologies system
feedback
Orchestrating
Learning Data
learning at
analytics visualization
enactment

46. Formative assessment and
feedback

47. Formative assessment and
feedback
Quick &
efficient

48. Formative assessment and
feedback
21st
Century
skills
Quick &
efficient

49. HTML5 & web technologies

50. HTML5 & web technologies
• Web notifications for forefront messages

51. HTML5 & web technologies
• Web notifications for forefront messages
• Web sockets for fast/real-time
communication (datavis in real time)

52. HTML5 & web technologies
• Web notifications for forefront messages
• Web sockets for fast/real-time
communication (datavis in real time)
• Key-value databases for a fast and efficient
backend: MongoDB

53. HTML5 & web technologies
• Web notifications for forefront messages
• Web sockets for fast/real-time
communication (datavis in real time)
• Key-value databases for a fast and efficient
backend: MongoDB
• JavaScript libraries for datavis: d3, protovis...

54. Attention & notification system

55. Attention & notification system
• Web notifications in order to make info
available without changing attention
(supported by websockets)

56. Attention & notification system
• Web notifications in order to make info
available without changing attention
(supported by websockets)
• One-click subscription to the notifications
channel from the wording of the session

57. Attention & notification system
• Web notifications in order to make info
available without changing attention
(supported by websockets)
• One-click subscription to the notifications
channel from the wording of the session
• Simple interface for the teacher to send
notifications to all the connected students

58. Experience

59. Experience
• AMM 2011

60. Experience
• AMM 2011
• 2 sessions

61. Experience
• AMM 2011
• 2 sessions
• Increasing
complexity

62. Experience
• AMM 2011
• 2 sessions
• Increasing
complexity
• Increasing
interest

63. Experience
• AMM 2011
• 2 sessions
• Increasing
complexity
• Increasing
interest

64. Future works

65. Future works
• Attention improvement delaying
notification delivery

66. Future works
• Attention improvement delaying
notification delivery
• Intelligent Tutoring System based on
common problems (notifications log)

67. Future works
• Attention improvement delaying
notification delivery
• Intelligent Tutoring System based on
common problems (notifications log)
• Collection of more student events and
create real time visualizations

68. Keywords of the ECTEL’11
short paper

69. Learning analytics

70. Learning analytics
Learning analytics is the measurement, collection,
analysis and reporting of data about learners and
their contexts, for purposes of understanding and
optimising learning and the environments in
which it occurs.
CfP of Conference on Learning Analytics and Knowledge 2011

71. What to
instrumentalize?

72. What to
instrumentalize?
• Lab session wording containing event
collection in JavaScript

73. What to
instrumentalize?
• Lab session wording containing event
collection in JavaScript
• Login info from LMS in order to associate
events to a student id

74. Event collection
interface
!

75. Event collection
interface
!
Buttons to start, finalize and
indicate a problem in an activity

76. Event collection
interface
!
Buttons to start, finalize and Todo list
indicate a problem in an activity of tasks

77. Recorded Events

78. Recorded Events
• Access to the wording

79. Recorded Events
• Access to the wording
• Start working on a section

80. Recorded Events
• Access to the wording
• Start working on a section
• Finish working in a section

81. Recorded Events
• Access to the wording
• Start working on a section
• Finish working in a section
• I have a problem / I solved the problem

82. Recorded Events
• Access to the wording
• Start working on a section
• Finish working in a section
• I have a problem / I solved the problem
• How the problem was solved:
individual effort, peer interaction or
teacher feedback

83. Recorded Events
• Access to the wording
• Start working on a section
• Finish working in a section
• I have a problem / I solved the problem
• How the problem was solved:
individual effort, peer interaction or
teacher feedback
How to translate that into learning outcomes achievement?

84. Contextual data
• Physical distribution of the labs

85. Contextual data
• Physical distribution of the labs
!

86. Orchestrating learning at
enactment

87. Orchestrating learning at
enactment
The metaphor of “orchestrating learning” has
been frequently used to address the challenges
of coordinating the complexity of conducting
learning activities.
Stellar D1.1

88. Teacher interface
!

89. Teacher information

90. Teacher information
• Progress

91. Teacher information
• Progress
• Stuck in a problem

92. Teacher information
• Progress
• Stuck in a problem
• Slow progress

93. Teacher information
• Progress
• Stuck in a problem
• Slow progress
• Consumed tutoring time

94. Teacher information
• Progress
• Stuck in a problem
• Slow progress
• Consumed tutoring time
• Teacher location

95. Teacher information
• Progress
• Stuck in a problem
• Slow progress
• Consumed tutoring time
• Teacher location
• Next step

96. Teacher information
• Progress 3
• Stuck in a problem
• Slow progress
• Consumed tutoring time
• Teacher location
• Next step

97. Teacher information
• Progress 3
• Stuck in a problem 3
• Slow progress
• Consumed tutoring time
• Teacher location
• Next step

98. Teacher information
• Progress 3
• Stuck in a problem 3
• Slow progress
• Consumed tutoring time
• Teacher location
• Next step

99. Teacher information
• Progress 3
• Stuck in a problem 3
• Slow progress
• Consumed tutoring time 2’
• Teacher location
• Next step

100. Teacher information
• Progress 3
• Stuck in a problem 3
• Slow progress
• Consumed tutoring time 2’
• Teacher location
• Next step

101. Teacher information
• Progress 3
• Stuck in a problem 3
• Slow progress
• Consumed tutoring time 2’
• Teacher location
• Next step

102. Teacher interface
!

103. Workflow

104. Workflow
• Use classON app to monitor student
problems and state in real time

105. Workflow
• Use classON app to monitor student
problems and state in real time
• In a normal situation teacher decides to

106. Workflow
• Use classON app to monitor student
problems and state in real time
• In a normal situation teacher decides to
• Directly help one of the teams

107. Workflow
• Use classON app to monitor student
problems and state in real time
• In a normal situation teacher decides to
• Directly help one of the teams
• Make one team help another one

108. Workflow
• Use classON app to monitor student
problems and state in real time
• In a normal situation teacher decides to
• Directly help one of the teams
• Make one team help another one
• Send a notification (since it’s a common
problem)

109. Data visualization

110. Data visualization
Information visualization is the interdisciplinary
study of the visual representation of large-
scale collections of information.
Michael Friendly (2008). "Milestones in the history of thematic
cartography, statistical graphics, and data visualization"

111. Intelligence
Amplification
(W. Ross Ashby, 1956)
Visualization
Amplifies Cognition
(@infosthetics, last Monday)

112. Intelligence
Amplification
(W. Ross Ashby, 1956)
rs !
h e
a c
r te
l fo Visualization
o o
t Amplifies Cognition
A
(@infosthetics, last Monday)

113. Critiques to datavis

114. Critiques to datavis
• Lack of usability and usefulness

115. Critiques to datavis
• Lack of usability and usefulness
• Not created for the average people

116. Critiques to datavis
• Lack of usability and usefulness
• Not created for the average people
• Noise

117. Datavis for students

118. Datavis for students
• Aesthetically attractive

119. Datavis for students
• Aesthetically attractive
• Fluent (straightforward and simple
interaction)

120. Datavis for students
• Aesthetically attractive
• Fluent (straightforward and simple
interaction)
• Make them reflect (in and out of class)

121. Datavis for students
• Aesthetically attractive
• Fluent (straightforward and simple
interaction)
• Make them reflect (in and out of class)
• Awareness of their knowledge and relate
it to the average

122. Datavis for students
• Aesthetically attractive
• Fluent (straightforward and simple
interaction)
• Make them reflect (in and out of class)
• Awareness of their knowledge and relate
it to the average
• React with an designed (re)action

123. Summing up: classON
One-way websocket notifications
Improve attention and effectiveness of general feedback
Record student interactions and use learning analytics
Improve orchestration in enactment and provision of
efficient feedback
Improve orchestration design for the following session and
acquiring evidences for summative assessment
Engagement in lab sessions (also other artifacts)
Awareness, reflection and reaction
Impact for Impact for
Measures
students teachers

124. Next steps

125. Next steps
• Define datavis for teachers’ and students’
interactions

126. Next steps
• Define datavis for teachers’ and students’
interactions
• Development of a classON prototype

127. Next steps
• Define datavis for teachers’ and students’
interactions
• Development of a classON prototype
• Save events for every interaction with the
system (even events manually annotated)

128. Next steps
• Define datavis for teachers’ and students’
interactions
• Development of a classON prototype
• Save events for every interaction with the
system (even events manually annotated)
• Use dataset to test the prototype

129. Evaluation

130. Evaluation
• AMM 2012

131. Evaluation
• AMM 2012
• Control and experiment groups OR
competition between groups

132. Evaluation
• AMM 2012
• Control and experiment groups OR
competition between groups
• How can we measure engagement

133. Evaluation
• AMM 2012
• Control and experiment groups OR
competition between groups
• How can we measure engagement
• Qualitative feedback

135. Thank you very much!
Any questions?