In this talk, I look quickly at several studies we've done over the years, which deal with the gap between research-driven educational technology proposals and what is actually used every day in our classrooms. It focuses especially on one of our latest studies at EPFL, in which we examine the effects, pros and cons of using a social media app (SpeakUp) in a face-to-face university lecture.

1. 11 ways of looking at
technology adoption in the
classroom
Luis P. Prieto
… including joint work with M. J. Rodríguez-Triana, A. Holzer and D. Gillet
… and many, many others
ITD-CNR, Genova – 5.10.2016

2. A few words about me
• B.Eng. Telecommunications
(Univ. Valladolid)
• Work in telecom industry
• Software developer
• Technical lead, EU projects
• Long-standing interest:
how people use technology
• … or not!

3. A growing concern:
educational technology adoption
Research
(classroom)
Practice
learning
learning
ANALYTICS
… etc.

4. Observational
study - routines
Orchestration
framework
LD deployment
& enactment
Teacher PD
workshops
Communities of
teacher LDers
Literature
reviews
Paper-based
technologies
Eyetracking &
orchestration load
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms

5. Part I
In-depth view of recent research on adoption of a particular
technology in the classroom

6. Examining the effects of social
media in co-located classrooms
A case study based on SpeakUp
María Jesús Rodríguez-Triana, A. Holzer, L. P. Prieto & D. Gillet
11th European Conference on Technology Enhanced Learning
September 14th, Lyon (France)

7. 7 /26
Motivation
Motivation SpeakUp Methodology ConclusionsStudy

8. Social media
8
Motivation SpeakUp Methodology ConclusionsStudy
/26

9. Social media
9
Motivation SpeakUp Methodology Conclusions
Increase participation
Increase interaction
Distraction
Multitasking
How to use social media
effectively in the classroom?
Study
/26

10. 10
SpeakUp
Motivation SpeakUp Methodology ConclusionsStudy
/26

11. speakup.info room name: SpeakUp @ ITDjoin room: 85524

12. 13
Research methodology
Motivation ConclusionsStudySpeakUp Methodology
/26

13. Research question
Does SpeakUp favor situations that lead
to learning?
‣ Topics:
‣Active participation (i.e., engagement)
‣Attention (i.e., remaining on-task)
‣Social interaction (on relevant content)
‣Teaching style
‣ Case study:
‣Authentic scenario (university)
14
Motivation SpeakUp Methodology ConclusionsStudy
/26

14. Informants & data sources
15
Motivation SpeakUp Methodology ConclusionsStudy
/26

15. Data analyses
16
Motivation SpeakUp Methodology ConclusionsStudy
‣ Mixed methods & triangulation
‣Quantitative:
‣descriptive statistics
‣exploratory computational analyses
‣Qualitative:
‣manual message coding (e.g., relevant/not relevant for
learning)
‣manual video coding (e.g., actor, action & target)
/26

16. 17
Case Study
Motivation ConclusionsSpeakUp Methodology Study
/26

17. Authentic scenario
18
Motivation SpeakUp Methodology Study Conclusions
/26
1st Lecture of a Communication course at
EPFL (Switzerland)
‣ 90 minutes
‣ 3 teachers
‣ 145
students
‣ SpeakUp

18. Topic 1: Engagement
‣ High proportion of
students joined the
room (147 users
out of 145
students!)
19
Motivation SpeakUp Methodology Study Conclusions
/26
SpeakUp was
engaging for
the students!
(μ = 5 in a 1-5 Likert scale)

19. Topic 1: Engagement
‣ Face-to-face vs SpeakUp-mediated participation
20
Motivation SpeakUp Methodology Study Conclusions
Teachers
(77 min)
Students
(11 min)
Teachers
(51
actions)
Students
(3481
actions)
/26

20. Topic 1: Engagement
‣ Participation profiles (K-means clustering)
21
Motivation SpeakUp Methodology Study Conclusions
77%
19%
4%
/26

21. Topic 2: Student attention
22
Motivation SpeakUp Methodology Study Conclusions
/26
SpeakUp had no
clear impact on
the student
attention
(μ = 3 in a 1-5 Likert scale)

22. Topic 2: Student attention
‣ Students perception vs usage
23
‣ What students said :) ‣ What students did :(
Motivation SpeakUp Methodology Study Conclusions
/26

23. Topic 3: Social interaction
24
Motivation SpeakUp Methodology Study Conclusions
/26
SpeakUp
promoted
interaction
between teachers
& students
(μ = 5 in a 1-5 Likert scale)
SpeakUp
promoted
interaction
among students
(μ = 4.5 in a 1-5 Likert scale)

24. Topic 3: Social interaction
‣ Face-to-face vs SpeakUp-mediated
interaction
25
Motivation SpeakUp Methodology Study Conclusions
/26

25. Topic 3: Social interaction
‣ SpeakUp-mediated interaction
‣15 isolated students
‣interaction degree:
‣Face-to-face interaction:
•Typical interaction degree < 8
26
Peer 1 Peer 2 Peer 3
Peer 4 Student Peer 5
Peer 6 Peer 7 Peer 8
Motivation SpeakUp Methodology Study Conclusions
/26

26. Topic 4: Teaching style
‣ High levels of relevant activity
correspond to activities guided by the
teachers
‣ Further and long term analysis are
needed
27
Motivation SpeakUp Methodology Study Conclusions
/26

27. Topic 4: Teaching style
28
Motivation SpeakUp Methodology Study Conclusions
/26
Managing two
simultaneous
channels is
demanding and
specially difficult
if teaching alone
SpeakUp
provided us with
awareness of a
students back
channel

28. 29
Conclusions
Motivation SpeakUp Methodology Study Conclusions
/26

29. Discussion - Results
Did SpeakUp favor situations that lead to
learning?
‣ T1: Students were engaged with SpeakUp but quite unevenly
‣ T2: Students were not always on task on SpeakUp and
inconsistent with their subjective perception of distraction
‣ T3: SpeakUp facilitated social interaction, especially between
peers (larger social network)
‣ T4: Positive effect of teacher prompts on message relevance
but SpeakUp usage can be too demanding for just one teacher
‣ Anonymity as a double-edged sword: participation vs distraction
30
Motivation SpeakUp Methodology Study Conclusions
/26

30. Discussion - Limitations
‣This is only one session in a longer (course-
long) analysis: novelty effects?
‣Link to learning still indirect (situations that
are known to lead to learning)
‣Rich analysis approach... but time
consuming! (how can we scale this up?)
31
Motivation SpeakUp Methodology Study Conclusions
/26

31. Future work
‣Finish the whole course analysis, incl. link
between relevancy and scores
‣Scaffold teachers in guiding SpeakUp
usage
‣Best patterns of usage for relevant engagement
‣Automated support: SpeakUp bots?
32
Motivation SpeakUp Methodology Study Conclusions
/26

32. Contact: maria.rodrigueztriana@epfl.ch , lprisan@hotmail.com
SpeakUp: www.speakup.info

33. Part II
Lightning view of previous research on adoption of edutech

34. Orchestration
framework
LD deployment
& enactment
Teacher PD
workshops
Communities of
teacher LDers
Literature
reviews
Paper-based
technologies
Eyetracking &
orchestration load
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms
Observational
study - routines

35. Observational classroom studies:
the discovery of ‘routines’
• Context: public primary school
• RQ: How do teachers adopt a
CSCL tool in their practice?
• Results: Importance of
improvised but recurrent
teaching actions (routines)
• So What?
• Can these routines be useful for
teacher PD?
• What is the impact of these
routines in the success of lesson?

36. Observational
study - routines
LD deployment
& enactment
Teacher PD
workshops
Communities of
teacher LDers
Literature
reviews
Paper-based
technologies
Eyetracking &
orchestration load
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms
Orchestration
framework

37. Conceptual framework for
orchestration of TEL: ‘5+3 Aspects’
• Context: Orchestration as a
worthy Ph.D. topic
• RQ: What do TEL researchers
mean when they talk about
‘orchestration’?
• Results: Literature review and
synthesis  ‘5+3 Aspects’
• So What?
• Orchestration is about making
edutech that plays well in
authentic classroom conditions
• Technologies for classroom should
support at all these aspects

38. Observational
study - routines
Orchestration
framework
Teacher PD
workshops
Communities of
teacher LDers
Literature
reviews
Paper-based
technologies
Eyetracking &
orchestration load
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms
LD deployment
& enactment

39. Technologies for orchestration of
learning designs in blended CSCL
• Context:
• A lot of LD authoring tools, but nobody
uses them
• … maybe because it’s difficult to deploy
in Moodle and other LEs?
• RQ: Can we support multi-LD, multi-LE
deployment and enactment?
• Results:
• Software architecture, prototype
• Evaluated in authentic settings,
workshops
• So What?
• Motivated teachers could deploy & enact
complex CSCL designs
• … but not many teachers want that level
of complexity!

40. Observational
study - routines
Orchestration
framework
LD deployment
& enactment
Communities of
teacher LDers
Literature
reviews
Paper-based
technologies
Eyetracking &
orchestration load
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms
Teacher PD
workshops

41. Workshop approach for teacher
professional development
• Context: Technology alone
does not drive usage, PD is
needed
• RQ: Can we design PD actions
that take into account this
idea of ‘orchestration’?
• Results: Approach based on
successful ‘routines’, GLUE!-
PS, conceptual+technological
• So What?
• Workshops were well-received!

42. Observational
study - routines
Orchestration
framework
LD deployment
& enactment
Teacher PD
workshops
Literature
reviews
Paper-based
technologies
Eyetracking &
orchestration load
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms
Communities of
teacher LDers

43. Supporting communities of
teachers as learning designers
• Context:
• A lot of LD authoring tools, not interoperable,
not deployable
• Communities of (LD) practice as key for its
adoption?
• RQ: Can we support teachers (conceptually
and technologically) to do LD with others,
using whatever tools they want?
• Results:
• ILDE: unified environment that integrates many
LD tools (powered by GLUE!-PS)
• PD package (workshops) for secondary, adult
and higher education
• So What?
• …………………………………………………. (?)

44. Observational
study - routines
Orchestration
framework
LD deployment
& enactment
Teacher PD
workshops
Communities of
teacher LDers
Paper-based
technologies
Eyetracking &
orchestration load
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms
Literature
reviews

45. Three TEL literature reviews
• Augmented paper in education:
• Mostly exploratory works, not many evaluated in
authentic conditions
• Development is still hard
• Generic UIs (e.g., tablets) have applicability
advantages
• Awareness and reflection in blended learning
• Learning dashboards:
• Still many proposals only use single platform logs
• Growing number of studies in authentic setting, but
few longitudinal
• Learning is seldom measured (!!)
• So What?
• Prototype syndrome as endemic in TEL
• Lack of evidence of learning advantages!

46. Observational
study - routines
Orchestration
framework
LD deployment
& enactment
Teacher PD
workshops
Communities of
teacher LDers
Literature
reviews
Eyetracking &
orchestration load
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms
Paper-based
technologies

47. Paper-based technologies for the
classroom
• Context: Primary schools, university
• RQ: Can we apply augmented paper
beyond spatial concepts/skills?
• Results:
• Several prototypes in Math, Chinese
• Software framework for easier
development of browser-based apps
• So What?
• Still “yet another prototype”
• Is the engagement and learning we see
sustainable after novelty wears off?

48. Observational
study - routines
Orchestration
framework
LD deployment
& enactment
Teacher PD
workshops
Communities of
teacher LDers
Literature
reviews
Paper-based
technologies
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms
Eyetracking &
orchestration load

49. Studies on orchestration load
using mobile eye-tracking
• Context: Face-to-face classrooms
• RQ: Can we measure teacher load
beyond just asking ‘how did it go’?
• Results:
• Method combining eye-tracking and
other sensors + video coding + machine
learning models
• Initial insights about orchestration load
patterns (e.g., whole-class activities are
higher load)
• So What?
• Initial validation gave encouraging
results… if you record multiple sessions
• Manual coding of videos still “expensive”!

50. Observational
study - routines
Orchestration
framework
LD deployment
& enactment
Teacher PD
workshops
Communities of
teacher LDers
Literature
reviews
Paper-based
technologies
Eyetracking &
orchestration load
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms
Multimodal
teaching analytics

51. Multimodal teaching analytics:
reversing learning design?
• Context: Face-to-face classrooms
• RQ: Can we get the enacted orchestration graph
(as opposed to the LD) from the sensors’ data?
• Results:
• Machine learning models to predict teaching
activity and social plane of interaction
• Reasonable accuracy (66-90%), with relatively
simple models
• Applicable for the teacher in the study only
• So What?
• Teacher can use this data for their PD  reflection
based on everyday evidence (~ fitness tracker)
• Researchers can look at why enactments of the
same LD get different results
• Could be used to ‘kickstart’ learning design
formalization?… at scale!

52. Observational
study - routines
Orchestration
framework
LD deployment
& enactment
Teacher PD
workshops
Communities of
teacher LDers
Literature
reviews
Paper-based
technologies
Eyetracking &
orchestration load
Multimodal
teaching analytics
Examining the effects of social
media in co-located classrooms
Tools for everyday
teacher reflection

53. Tools for evidence-based teacher
reflection from everyday data
• Context: Secondary schools
• RQ: Can we help teachers in reflecting for
PD, by gathering data from everyday
practice?
• Results:
• Two approaches:
1. High-tech (eye-tracking, beacons) 
visualizations (classroom mirror)
2. Low-tech (very simple questionnaire for
students + teacher predictions)  simple
dashboard (Prolearning)
• Low-tech shows improvement in student
experience and teacher behavior changes
• So What?
• Fancy tech does not necessarily mean better
adoption!

54. Observational
study - routines
Orchestration
framework
LD deployment
& enactment
Teacher PD
workshops
Communities of
teacher LDers
Literature
reviews
Paper-based
technologies
Eyetracking &
orchestration load
Multimodal
teaching analytics
Tools for everyday
teacher reflection
Examining the effects of social
media in co-located classrooms

55. Conclusion
Research
(classroom)
Practice

56. Conclusion: 5 tips for adoption
1. Get out of the building and inside the classroom
(for long periods, if possible)
2. Demonstrate advantages first (especially learning
gains!), before asking for adoption
3. Minimum Viable Prototypes (MVPs) over
specifications
4. If it cannot be used every day, it probably will not
be used after researchers go away
5. Think about scale from the outset

57. References (I)
• Social media in the classroom (SpeakUp)
• Rodríguez-Triana, M. J., Holzer, A., Prieto, L. P., & Gillet, D. (2016, September).
Examining the Effects of Social Media in Co-located Classrooms: A Case Study
Based on SpeakUp. In European Conference on Technology Enhanced Learning
(pp. 247-262). Springer International Publishing.
• Observational study & routines
• Prieto, L. P., Villagrá-Sobrino, S., Jorrín-Abellán, I. M., Martínez-Monés, A., &
Dimitriadis, Y. (2011). Recurrent routines: Analyzing and supporting
orchestration in technology-enhanced primary classrooms. Computers &
Education, 57(1), 1214-1227.
• Orchestration framework (‘5+3 Aspects’)
• Prieto, L. P., Holenko Dlab, M., Gutiérrez, I., Abdulwahed, M., & Balid, W. (2011).
Orchestrating technology enhanced learning: a literature review and a
conceptual framework. International Journal of Technology Enhanced Learning,
3(6), 583-598.
• Prieto, L. P., Dimitriadis, Y., Asensio-Pérez, J. I., & Looi, C. K. (2015).
Orchestration in learning technology research: evaluation of a conceptual
framework. Research in Learning Technology, 23.

58. References (II)
• LD deployment and enactment (GLUE!-PS)
• Prieto, L. P., Asensio-Pérez, J. I., Dimitriadis, Y., Gómez-Sánchez, E., & Muñoz-Cristóbal, J. A. (2011,
September). GLUE!-PS: a multi-language architecture and data model to deploy TEL designs to
multiple learning environments. In European Conference on Technology Enhanced Learning (pp.
285-298). Springer Berlin Heidelberg.
• Prieto, L. P., Asensio-Pérez, J. I., Muñoz-Cristóbal, J. A., Jorrín-Abellán, I. M., Dimitriadis, Y., &
Gómez-Sánchez, E. (2014). Supporting orchestration of CSCL scenarios in web-based Distributed
Learning Environments. Computers & education, 73, 9-25.
• Teacher PD workshop approach
• Prieto, L. P., Dimitriadis, Y., Asensio-Perez, J. I., Villagrá-Sobrino, S., & Jorrín-Abellan, I. (2013).
Fostering CSCL adoption: an approach to professional development focused on orchestration. In
Proceedings of the International Conference of Computer-Supported Collaborative Learning
(CSCL2013). volume 1.
• Communities of teacher learning designers (METIS – ILDE)
• Hernández-Leo, D., Chacón, J., Prieto, L. P., Asensio-Pérez, J. I., & Derntl, M. (2013, September).
Towards an integrated learning design environment. In European Conference on Technology
Enhanced Learning (pp. 448-453). Springer Berlin Heidelberg.
• Hernández-Leo, D., Asensio-Pérez, J. I., Derntl, M., Prieto, L. P., & Chacón, J. (2014, September).
ILDE: community environment for conceptualizing, authoring and deploying learning activities. In
European Conference on Technology Enhanced Learning (pp. 490-493). Springer International
Publishing.

59. References (III)
• TEL literature reviews
• Prieto, L. P., Wen, Y., Caballero, D., & Dillenbourg, P. (2014). Review of
Augmented Paper Systems in Education: An Orchestration Perspective.
Educational Technology & Society, 17(4), 169-185.
• Rodriguez Triana, M. J., Prieto Santos, L. P., Vozniuk, A., Shirvani Boroujeni, M.,
Schwendimann, B. A., Holzer, A. C., & Gillet, D. (in press). Monitoring,
Awareness and Reflection in Blended Technology Enhanced Learning: a
Systematic Review. Accepted at the International Journal of Technology-
Enhanced Learning.
• Schwendimann, B. A., Rodríguez-Triana, M. J., Vozniuk, A., Prieto, L. P.,
Boroujeni, M. S., Holzer, A., ... & Dillenbourg, P. (2016, April). Understanding
learning at a glance: An overview of learning dashboard studies. In Proceedings
of the Sixth International Conference on Learning Analytics & Knowledge (pp.
532-533). ACM.
• Paper-based technologies for the classroom
• Caballero, D., Wen, Y., Prieto, L. P., & Dillenbourg, P. (2014, November). Single
locus of control in a tangible paper-based tabletop application: an exploratory
study. In Proceedings of the Ninth ACM International Conference on Interactive
Tabletops and Surfaces (pp. 351-356). ACM.
• Web-based apps framework: http://chili-epfl.github.io/paprika/

60. References (IV)
• Eye-tracking studies and orchestration load
• Prieto, L. P., Sharma, K., Wen, Y., & Dillenbourg, P. (2015). The burden of
facilitating collaboration: towards estimation of teacher orchestration
load using eye-tracking measures. In Proceedings of the 11th
international conference on computer-supported collaborative learning
(CSCL 2015) (pp. 212-219). Sweden: Gothenburg.
• Prieto, L. P., Sharma, K., & Dillenbourg, P. (2015). Studying teacher
orchestration load in technology-enhanced classrooms. In Design for
Teaching and Learning in a Networked World (pp. 268-281). Springer
International Publishing.
• Multimodal teaching analytics and reversing LD
• Prieto, L. P., Sharma, K., Dillenbourg, P., & Jesús, M. (2016, April).
Teaching analytics: towards automatic extraction of orchestration
graphs using wearable sensors. In Proceedings of the Sixth
International Conference on Learning Analytics & Knowledge (pp. 148-
157). ACM.
• Tools for teacher reflection [Still unpublished]

61. Questions (again)?

62. Thank you! 
For further info, email me: lprisan@hotmail.com
(or just catch me this week in the corridor)