Apoyo al profesorado con analíticas de comunidad, diseño y aprendizaje
Supporting teachers with community, design and learning analytics
Seminario eMadrid, UAM 05/2019
http://www.emadridnet.org/index.php/es/eventos2/1100-seminario-emadrid-sobre-tecnologias-dentro-y-fuera-del-aula
http://www.emadridnet.org/index.php/es/28-eventos-y-seminarios/1102-apoyo-al-profesorado-con-analiticas-de-comunidad-diseno-y-aprendizaje
Abstract
I will present an overview of the educational technologies research conducted by the TIDE research group of the Information and Communication Technologies Department at Universitat Pompeu Fabra in Barcelona (http://www.upf.edu/web/tide @TIDE_UPF). The overview will be articulated around the perspective of supporting teachers and teacher communities (e.g., a school) in the design of the best possible (technology-enhanced) learning activities considering their students and their contexts. Main contributions that will be presented include a community platform for integrated learning design (ILDE), including multiple authoring tools (e.g. PyramidApp for collaborative learning, edCrumble for blended learning) and the use of data analytics at different levels (learning, design, community) to support community awareness and teacher reflection when designing for learning. The presentation will include results of several research projects (METIS, CoT, MdM-EDS, RESET, SmartLET, Illuminated).
En esta ponencia presentaré un resumen de la investigación en tecnologías educativas llevada a cabo por el grupo TIDE del Departamento de Tecnologías de la Información y las Comunicaciones en la Universidad Pompeu Fabra en Barcelona (http://www.upf.edu/web/tide @TIDE_UPF). El resumen se presenta desde la perspectiva del apoyo al profesorado y a comunidades de profesores (como, por ejemplo, una escuela) en el diseño de buenas actividades de aprendizaje considerando los estudiantes y sus contextos. Las contribuciones principales incluyen una plataforma de comunidad para el diseño integrado de actividades de aprendizaje (ILDE), incluidas herramientas de autoría (como PyramidApp para aprendizaje colaborativo apoyado por ordenador, edCrumble para aprendizaje híbrido) y el uso de analíticas de datos a diferentes niveles (aprendizaje, diseño, comunidad) ara facilitar la conciencia de comunidad y la reflexión por los profesores cuando diseñan para generar aprendizajes. La presentación incluirá resultados de varios proyectos de investigación (METIS, CoT, MdM-EDS, RESET, SmartLET, Illuminated).
http://www.upf.edu/web/tide
Mattingly "AI & Prompt Design: The Basics of Prompt Design"
Supporting teachers with community, design and learning analytics, Davinia Hernández-Leo, TIDE UPF, eMadrid
1. Apoyo al profesorado con analíticas de comunidad, diseño y aprendizaje
Supporting teachers with community, design and learning analytics
eMadrid, UAM, 10 Mayo 2019
Davinia Hernández-Leo
TIDE, Universitat Pompeu Fabra, Barcelona
twitter: @TIDE_UPF
website : http://www.upf.edu/web/tide
3. Support teachers in designing the “best possible”
conditions for their students to learn, documenting them,
making their design ideas explicit and sharable.
(Conole, 2012; Agostinho, 2011; Laurillard, 2013;
Mor, Craft & Hernández-Leo, 2013)
Learning Design
3
Teaching as a design science
5. The case of CSCL macro-scripting
Free collaboration does not necessarily lead to fruitful
learning. An option is to structure the collaborative learning
flow (activity sequence, roles, group formation, resource
allocation) to trigger potentially effective social interactions.
(Dillenbourg & Tchounikine, 2007; Kobbe et al., 2007)
Learning Design in Collaborative Learning
5
6. Learning design authoring and enactment
— Pyramid collaborative learning flow pattern. Scalable approach. Flow
creation and control mechanisms for elasticity and dynamism
6
Learning Design in CSCL: example of PyramidApp
https://www.upf.edu/web/tide/tools
Manathunga, K., & Hernández‐Leo, D. (2018). Authoring and enactment of mobile pyramid‐based
collaborative learning activities. British Journal of Educational Technology, 49(2), 262-275.
7. Design layers metaphor
(by spaces)
Systematically co-designed
with teachers
Emphasis in design for
blended learning
(PBL, flipped classroom, …)
7
Learning Design in blended learning: edCrumble
https://www.upf.edu/web/tide/tools
Albo L, Hernández-Leo D. edCrumble: designing for learning with data analytics. EC-TEL 2018. Best Demo Award.
8. 8
Learning design tools
Tools for conceptualization, authoring, implementation
Generic vs. specific tools wrt. pedagogical approach
Sharing, co-creating learning designs
9. 9
Integrated Learning Design Environment
Integrated solution to support the full lifecycle of (co-)designing,
sharing, and deploying ICT-based learning activities
Hernández-Leo, D., et al. (2018). An Integrated Environment for Learning Design. Frontiers in ICT, 5, 9
11. ILDE, previous results
Used in several different educational contexts
(higher education, vocational education, adult education)
— Well accepted in all its functions
— Fits the needs of different types of communities
— No difference in acceptance
— Flexible design processes
— Limited social awareness and inquiry process
Data used to evaluate and understand processes…
11
12. Learning design and data
Data used to evaluate and understand processes…
How can data be used to support learning design and learning
design communities?
12
13. Data Analytics in teaching & learning
Learning Analytics (LA): “the measurement, collection, analysis, reporting of
data about learners and their contexts, for porpuses of understanding and
optimizing learning and the environments in which it occurs” SoLAR (Ferguson,
2012)
Social Learning Analytics (Buckingham-Shum & Ferguson 2012) & Community
analytics and visualizations (Klamma, 2013; Vasileva and Sun, 2007), including Social
Network Analysis (SNA) (de Laat & Schreurs, 2013)
13
14. ANALYTICS LAYERS FOR LEARNING DESIGN FRAMEWORK
Supporting learning design with analytics
14
Community of practitioners
and related stakeholders
Learning Design (LD) tools
Learners experiencing
learning designs
Community Analytics
metrics and patterns of design activity
Design Analytics
metrics of pedagogical decisions
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
Hernández-Leo, D., et al. (2019) Analytics for learning design: A layered framework and tools, British Journal of Educational
Technology. https://doi.org/10.1111/bjet.12645
16. Available data
ILDE2, extension with data analytics features
— https://www.upf.edu/web/tide/tools/ilde2
CoT project, 2 schools
RESET, Smartlet, Illuminated:
— Master program for teacher training
— MOOC for teachers
— Teacher training workshops
Supporting learning design with analytics
16
17. Analytics for learning design
Community of practitioners
and related stakeholders
Community Analytics
metrics and patterns of design activity
Understanding community behaviour
Triggering orientation and inspiration
(e.g. awareness, provides a context for your own activity)
Data includes:
— what tools are being used,
— what types of designs are created, tags,
— who are the (co-)authors, edits, views,
— versioning of designs,
— social ratings …
Michos, K., Hernández-Leo, D., (2016) Understanding
Collective Behavior of Learning Design Communities.
EC-TEL 2016.
Garreta-Domingo, M., Sloep, P., Hernández-Leo, D., Mor, Y.,
(2018) Design for collective intelligence: Pop-up communities in
MOOCs. Journal AI & Society 33(1), 91-100.
18. Analytics for learning design
Community of practitioners
and related stakeholders
Community Analytics
metrics and patterns of design activity
Understanding community behaviour
Triggering orientation and inspiration
Data includes:
— what tools are being used,
— what types of designs are created, tags,
— who are the (co-)authors, edits, views,
— versioning of designs,
— social ratings …
Duplicates ….
19. Analytics for learning design
Community of practitioners
and related stakeholders
Community Analytics
metrics and patterns of design activity
Understanding community behaviour
Triggering orientation and inspiration
Data includes:
— what tools are being used,
— what types of designs are created, tags,
— who are the (co-)authors, edits, views,
— versioning of designs,
— social ratings …
Michos, K., Hernández-Leo, D., (2018) Supporting awareness in communities of
learning design practice, Computers in Human Behavior, 85, 255-270.
21. Blending the dashboard with learning design tasks
With/Without dashboard use
Differences in community members´ participation and perceptions
Community dashboard evaluation
21
22. Dashboard use increased social interactions by means of comments and profile views
Positive user experience (UMUX questionnaire) for understanding community participation
Alignment with tasks for learning design
Community dashboard evaluation
22
Variables Group N Ranks U Z p r
profiles views High dashboard use 25 31.26 138.50 -3.238 .001 -0.46
Low dashboard use 24 18.27
comments High dashboard use 25 30.70 157.50 -2.877 .004 -0.41
Low dashboard use 24 19.06
created designs High dashboard use 25 30.12 172.00 -2.570 .010 -0.36
Low dashboard use 24 19.67
re-used designs High dashboard use 25 30.08 173.00 -2.547 .011 -0.36
Low dashboard use 24 19.71
edits High dashboard use 25 29.60 185.00 -2.335 .020 -0.33
Low dashboard use 24 20.21
23. Learning Design (LD) tools
Design Analytics
metrics of pedagogical decisions
Categorise learning designs
Support awareness and reflection
about design decisions
Data includes:
— learning objectives,
— type of tasks tasks,
— the social planes (individual, collaborative),
— resources and tools associated with the task,
— expected length of time for tasks, …
Analytics for learning design
Albo L, Hernández-Leo D. edCrumble: designing for learning with data
analytics. EC-TEL 2018. Best Demo Award.
24. Learning Design (LD) tools
Design Analytics
metrics of pedagogical decisions
Categorise learning designs
Support awareness and reflection
about design decisions
Interaction with the
Community Analytics layer
Analytics for learning design
Albo L, Hernández-Leo D. edCrumble: designing for learning with
data analytics. EC-TEL 2018. Best Demo Award.
25. Learners experiencing
learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
Accumulated evidence of design’s impact
Support awareness and reflection
about effects of designs, redesign
Data includes:
— snapshot data capturing relevant milestones
(e.g., access to resources or completion of tasks)
— data about the process (e.g., participation in
discussions)
— performance (e.g., grades)
— students’ and teachers’ opinions…
Analytics for learning design
Michos, K., Manathunga, K., Hernández-Leo, D. (2016). Connecting pattern-based
learning designs with analytics: The case of the PyramidApp., CLAD - EC-TEL 2016
26. Learners experiencing
learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
Accumulated evidence of design’s impact
Support awareness and reflection
about effects of designs, redesign
Data includes:
— snapshot data capturing relevant milestones
(e.g., access to resources or completion of tasks)
— data about the process (e.g., participation in
discussions)
— performance (e.g., grades)
— students’ and teachers’ opinions…
Analytics for learning design
Michos, K., Fernández, A., Hernández-Leo, D., & Calvo, R.
(2017). Ld-Feedback App: Connecting Learning Designs with
Students’ and Teachers’ Perceived Experiences. EC-TEL.
27. Learners experiencing
learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
Accumulated evidence of design’s impact
Support awareness and reflection
about effects of designs, redesign
Interaction with Design Analytics
and Community Analytics
— Teacher inquiry, Community inquiry
Analytics for learning design
Michos, K., & Hernández-Leo, D., Albó, L. (2018). Teacher-led
inquiry in technology-supported school communities. British
Journal of Educational Technology 49(6), 1077-1095.
29. Teacher produced artifacts & Learning Analytics
29
Learning Analytics reports
Teacher produced artifacts
Reflections on the enactment
30. Some results
Awareness and regulation of design activity (before, during, and after enactment)
Connection between initial design and data-informed reflections
Documentation for later review
Sharing learning design problems with solutions, with contextual information
Teacher roles as designers is guided by practicality and time constraints
In-depth discussions about teaching and learning strategies with the use of student generated data
and TILE
30
31. Conclusions
31
Data driven learning design support in teacher communities
Support to teachers
Three analytics layers: Community, Design and Learning analytics
And interplays!
Awareness, inspiration, reuse, teacher (and community) inquiry, learning redesign
Future work:
— New communities (e.g., Maker activities)
— Motivations, incentives
— Socio-emotional aspects and learning design (Spotlighters project)
— Actionable dashboard for PyramidApp
Concerned about ethical aspects, e.g.
— Beardsley, M., Santos, P., Hernández-Leo, D., Michos, K. (2019). Ethics in educational technology research:
informing participants in data sharing risks. British Journal of Educational Technology. 10.1111/bjet.12781
5. Resumen de la ponencia
I will present an overview of the educational technologies research conducted by the TIDE research group of the Information and Communication Technologies Department at Universitat Pompeu Fabra in Barcelona (http://www.upf.edu/web/tide @TIDE_UPF). The overview will be articulated around the perspective of supporting teachers and teacher communities (e.g., a school) in the design of the best possible (technology-enhanced) learning activities considering their students and their contexts. Main contributions that will be presented include a community platform for integrated learning design (ILDE), including multiple authoring tools (e.g. PyramidApp for collaborative learning, edCrumble for blended learning) and the use of data analytics at different levels (learning, design, community) to support community awareness and teacher reflection when designing for learning. The presentation will include results of several research projects (METIS, CoT, MdM-EDS, RESET, SmartLET, Illuminated).
En esta ponencia presentaré un resumen de la investigación en tecnologías educativas llevada a cabo por el grupo TIDE del Departamento de Tecnologías de la Información y las Comunicaciones en la Universidad Pompeu Fabra en Barcelona (http://www.upf.edu/web/tide @TIDE_UPF). El resumen se presenta desde la perspectiva del apoyo al profesorado y a comunidades de profesores (como, por ejemplo, una escuela) en el diseño de buenas actividades de aprendizaje considerando los estudiantes y sus contextos. Las contribuciones principales incluyen una plataforma de comunidad para el diseño integrado de actividades de aprendizaje (ILDE), incluidas herramientas de autoría (como PyramidApp para aprendizaje colaborativo apoyado por ordenador, edCrumble para aprendizaje híbrido) y el uso de analíticas de datos a diferentes niveles (aprendizaje, diseño, comunidad) ara facilitar la conciencia de comunidad y la reflexión por los profesores cuando diseñan para generar aprendizajes. La presentación incluirá resultados de varios proyectos de investigación (METIS, CoT, MdM-EDS, RESET, SmartLET, Illuminated).
I will present an overview of the educational technologies research conducted by the TIDE research group of the Information and Communication Technologies Department at Universitat Pompeu Fabra in Barcelona (http://www.upf.edu/web/tide @TIDE_UPF). The overview will be articulated around the perspective, central to TIDE work, of supporting teachers and teacher communities (e.g. a school) in the design of the best possible (technology-enhanced) learning activities considering their students and their contexts. Main research contributions that will be presented include a community platform for integrated learning design (ILDE, including multiple authoring tools e.g. edCrumble), scalable and flexible orchestration of computer-supported collaborative learning scripts (PyramidApp), and the use of data analytics at different levels (learning, design, community) to support teachers in learning (re)design. The presentation will include results of European, Spanish and Catalan projects (METIS, RESET, CoT) and our initial work in recently started projects (SmartLET, Illuminated).
The overview will be articulated around the perspective, central to TIDE work, of supporting teachers and teacher communities (e.g. a school) in the design of the best possible (technology-enhanced) learning activities considering their students and their contexts.
What is designed
Activity sequence, roles, group formation, resource allocation, etc..
Mesh available learning design authoring tools, and to enable the sharing and enactment of learning designs (Ld), within an Integrated Learning Design Environment (ILDE)
We implemented the community analytics dashboard in the Graphical user interface of the ILDE platform
(Hernández-Leo, et al. 2018; Michos & Hernández-Leo, 2016)
(Hernández-Leo, et al. 2018; Michos & Hernández-Leo, 2016)
(Hernández-Leo, et al. 2018; Michos & Hernández-Leo, 2016)
Awareness, reflection and impact on how, what and with which effects practitioners design for learning
Awareness, reflection and impact on how, what and with which effects practitioners design for learning
(Hernández-Leo, et al. 2018; Michos & Hernández-Leo, 2016)
Questions:
Which designs are more popular in the community?
Which teachers are working on designs for particular pedagogies so they can be candidates for collaboration?
What has been the evolution of a (reused) design and how may I be inspired by this?
What tools I should think of exploring because they are better received by the other teachers in my community?
Questions:
Which designs are more popular in the community?
Which teachers are working on designs for particular pedagogies so they can be candidates for collaboration?
What has been the evolution of a (reused) design and how may I be inspired by this?
What tools I should think of exploring because they are better received by the other teachers in my community?
Questions:
Which designs are more popular in the community?
Which teachers are working on designs for particular pedagogies so they can be candidates for collaboration?
What has been the evolution of a (reused) design and how may I be inspired by this?
What tools I should think of exploring because they are better received by the other teachers in my community?
We designed three tabs: One about usage analytics of designs, another about usage analytics of learning design tools, and one about participation analytics of community members. The dashboard was iteratively tested in four educational communities. A final evaluation took place in a MOOC for learning design.
In the MOOC we evaluated community members participation and interactions with and without the use of the dashboard. Moreover, we evaluated participation groups of users who frequently used the dashboard versus groups of users who used it letss
We ran nonparametric t-tests and we found significant differences in high versus low dashboard use in the frequency of comments and profile views thus dashboard use increased social interactions in the platform. Moreover, we perfomed content analysis of the teachers online comments with without the dashboard and we found more comments who included community observations and comparisons..e.g. reference to design patterns for collaborative learning, reference to context. Last, we tested user experience of the dashboard with the UMUX and we received positive evaluation
Questions:
Is the accumulated percentage of individual vs. group tasks proposed balanced as desired?
Will the students be proposed to use a sufficient variety of resources and tools?
Is the time estimated so far for the learning tasks reasonable?
Can feed also community analytics layer
Questions:
Is the accumulated percentage of individual vs. group tasks proposed balanced as desired?
Will the students be proposed to use a sufficient variety of resources and tools?
Is the time estimated so far for the learning tasks reasonable?
Can feed also community analytics layer
Questions:
Is the average time length used to complete a task close to the expectations?
Is the design leading to unsatisfactory engagement or performance?
Is the design engaging sufficiently students with a particular profile?
Can feed also community and design layers
Questions:
Is the average time length used to complete a task close to the expectations?
Is the design leading to unsatisfactory engagement or performance?
Is the design engaging sufficiently students with a particular profile?
Can feed also community and design layers
The mediating role of teacher inquiry between LD and LA (Alhadad & Thompson, 2017; Persico & Pozzi, 2015)
We then involved teachers in cycles of design and enactment and through this process we iteratively designed the Teacer Inquiry tool for learning designs(TILE), to faciliated teachers in data-informed reflections. Based on other TI models we designed 4 steps visualized as tabs in the tool. In the first phases teacher document the identified problem and questions for evaluation, In the second step describe the intervention and evaluation design based on the problem. There we embedded tools for designing and collecting student data. In the 3rd step teachers analyze the collected student data and in the 4th step they reflect on the intervention based on the data and initial problem and questions by proposing future changes for enactment
Thus teachers were provided with learning analytics reports after classroom enactments. This includes participation analytics of students in the pyramid app and the content of their discussions. Moreover, in some cases teachers collected student feedback about the task.
To evaluate this process we relied on qualitative methods and analyzed interviews with teachers, online comments and open questions in the questionnaire. Moreover we analyzed all teacher artifacts who implemented learning activities. This resulted to tables like the following
The analysis of the technology supported teacher inquiry process shows that a guided process though the inquiry steps provides awareness and regulation of design activity (before, during and after the enactment of ld activities) which was previously consider more during the desing and enactment and not on the reflection.
Analysis of teacher artifacts shows that teachers did connect the initial design with the reflections based on the data.
The main added value of such technology is that documentation can help teacher review such reflection in later stage
Morevoer it helped to share and elaborate on learning design problems and solutions
Lasty, the investigation shows that teachers as designers role was driven by practical issues and time constraints