Slides from my presentation at EC-TEL 2012 in Saarbrucken, Germany, 18th September Abstract: This paper elaborates on the evaluation of a Metacognitive Scaffolding Service (MSS), which has been integrated into an already existing and mature medical training simulator. The MSS is envisioned to facilitate self- regulated learning (SRL) through thinking prompts and appropriate learning hints enhancing the use of metacognitive strategies. The MSS is developed in the European ImREAL (Immersive Reflective Experience-based Adaptive Learning) project that aims to augment simulated learning environments throughout services that are decoupled from the simulation itself. Results comparing a baseline evaluation of the ‘pure’ simulator (N=131) and a first user trial including the MSS (N=143) are presented. The findings indicate a positive effect on learning motivation and perceived performance with consistently good usability. The MSS and simulator are perceived as an entity by medical students involved in the study. Further steps of development are discussed and outlined.

1. An Initial Evaluation of
Metacognitive Scaffolding for
Experiential Training Simulators
Marcel Berthold, Adam Moore, Christina Steiner, Conor
Gaffney, Declan Dagger, Dietrich Albert, Fionn
Kelly, Gary Donohoe, Gordon Power, Owen Conlan

2. Overview
• Who?
• ImREAL Project
• Evaluating Self-Regulated Learning and Metacognitive Scaffolding in Experiential
Simulator
• How?
• Cross cohort comparison
• Baseline usage of sim vs scaffolded usage
• Initial – no cross correlation / matching
• Third year medical students
• What (did we find out)?
• Performance
• SRL behaviour
• Scaffolding encounters & effects
• Analysis of reflection text New! (Not in paper)

3. Rationale of ImREAL
• • ’
’
• ‘
’ ‘ ’ •

4. What do we want to know?
1. Is self-regulated learning supported?
2. Does the simulator augmentation through the service lead to better
learning performance?
3. Does the simulator augmentation through the service increase
motivation?
4. Is the service well integrated in the simulation and learning experience?

5. Self-Regulated Learning (SRL)
• SRL is composed of three cyclic learning phases: Forethought, Learning,
Reflection (Zimmermann, 2002)
• Good SR learners use appropriate learning strategies and techniques
• Good SR learners achieve
better learning results and
are more motivated to learn
(Zimmerman, 2002; Veenmann, 2011)
SRL@ET workshop

6. Pedagogical framework in
ImREAL
• Integrated SRL cycles + Peer experience
(Hetzner et al., 2011)

7. Simulator
• Dialogic
• Training for diagnostic interviews for psychiatric patients

8. Reflection Elicitation / Scaffolding
• Simulator has pre-existing
reflection tool
• Can be triggered by learner
• Prompts to reflect also added by
instructional designer
• In practice mode, cohort 2
students had ImREAL
Metacognitive Scaffolding Service
(MSS)

9. Scaffolding – Thinking Prompts
Phase 1 Trial

10. Cohorts
• Approximately 140 (cohort 1 = 131 / cohort 2 = 143) medical
students, on average 21 years old (40% male vs. 60% female, 80% Irish).
They performed the simulation as part of their third year medical
training at Trinity College, Dublin (cohort 1 in 2011, cohort 2 in 2012).
• Cohort 2 additionally has scaffolding prompts in practice mode only
• Experience
• No experience with ETU simulator
• Experienced with interviews (97 %)
• limited experience with interviewing psychiatric patients (15 %)

11. Methods Assessment
Log-Data
Have you set a
learning goal?
Questionnaire Reflection
Data postings

12. Methods: Evaluation approach in ImREAL
(First User trail)
• Project evaluation question
o Can SRL be enhanced through
Metacognitive Scaffolding Services (MSS)?
• Formative evaluation approach
• Research foci of ImREAL
MSS
 Integrated ImREAL services
• Investigate: Impact on SRL
reports, behaviour, qualitative
feedback
http://www.empowertheuser.ie

13. Results: SRL
• No change in SRL compared to baseline
• Highest usage of elaboration strategies
SRL strategies profil
100
90
80
70
60
50
%
40
30
20
10
0
Self- Time Achievement
Memorizing Elaboration Organisation Planning Internal Effort
monitoring management motivation
strategies strategies strategies strategies attribution strategies
strategies strategies strategies
First user trial 57.65 70.16 62.68 59.14 60.01 61.22 62.40 59.11 55.78
Baseline Evaluation 54.07 69.80 62.24 58.24 60.91 59.93 60.44 56.87 55.30
SRL strategy scales t20=3.34, p=.003

14. Results: Learning Experience
• Scores are above centre line of the rating scale
• Mentioned that prompts were not always presented at appropriate time
Learning Experience MSS
5
4
Ratings
3.42 3.33 3.39 3.38 3.45 3.42
3.12 3.27 3.26
3.09 2.97
3
2
1
Items

15. Results: Motivation
• Students were motivated
• To perform simulation
• Learn about interviews
• To apply skills learnt in simulation in real world
• State motivation after simulation higher in 1st user trial compared to
baseline
Motivation
4
3.35
3
Ratings
2.48
2
1
0
Trial
Baseline Evaluation First User Trial t118.47=-8.64, p<.001

16. Results: Scaffolding
• Comparison of expected and empirical MSS prompt distribution
• More scaffolds in Information Management phase
• Less scaffolds on reflection
Expected vs. Empirical Distribution of MCSs
800
700
Frequencies
600
500
400
300
200
100
0
Informantion
Planning Monitoring Debugging Reflection
Phases Management
Expected distribution 400.20 571.71 400.20 301.00 343.03
Empirical distribution 469.00 752.00 425.00 301.00 54.00
χ2(4,0.95)= 314.55, p<.001

17. Methods: Coding reflective text
• Positional – time, date, simulator, etc.
• “Rapport and Initial inquiry have been completed. Must now concentrate on moving
the interview forward”
• Technical – material addressing the system, questions or underlying
model
• Great concept. Has a lot of potential, will be much better when more points to
consider and points of information are added to the overview that accurately reflect
performance.
• Notes – observations of the patient
• “Px. unable to quantify length of period she has been like this for.
• Reflections – reflective text
• it is quite important to deal with the patient with empathy to make them comfortable
and also to try and illicit the cause of the bout of depression, in this case. I felt I did not
do too bad as far as expressing empathy is concerned, allowing the patient to open up
and try and formulate a management plan. More experience in history taking will be
the key

18. Results: Reflective Text
• Corpus 1 (from cohort 1): had 358 unique text entries from 81 unique
UserIDs giving an average of 4.41 entries per unique UserID
• Engaged percentage = 57.9%
• Corpus 2 (from cohort 2): had 107 unique text entries into the note-
taking tool from 50 unique UserIDs giving an average of 2.14 entries and
3418 views of the MST, resulting in 28 entries, 19 with text from 8 unique
UserIDs
• Engaged percentage = 35.7%

19. Results: Reflective Text
• percentage breakdown of each type of content within the note-taking
tool across the two cohorts:
Position Technical Notes Reflection
Cohort 1 15 33 25 69
Cohort 2 17 57 16 66

20. Discussion
• SRL
• No correlation of log-data and SRL reports were observed – needs to be investigated
why?
• No changes in SRL – long-term process
• MSS
• Learners seem to need more assistance in effectively processing information by hints
to use more organizational, elaborative, summarizing or selective learning strategies
• Are they more confident in the reflection phase and so pass the offer of scaffolds
• Time to engage
• Less availability of simulators in cohort 2
• Less lab time & shorter overall period
• Overall engagement and motivation increased
• Higher percentage of time being scaffolded

21. Lessons Learnt
• Provide MS at appropriate times
• Keep learners longer in simulation
• Provide additional services to promote SRL

22. What did we find out?
1. Is self-regulated learning supported?
• Yes
• Is it improved . . . . Maybe
2. Does the simulator augmentation through the service lead to better
learning performance?
• No! But . . . . Yes!
3. Does the simulator augmentation through the service increase
motivation?
• Yes
4. Is the service well integrated in the simulation and learning experience?
• Yes

23. Further Work
• Longitudinal study
• SRL changes slow
• Integration of changes into behaviour
• Fuller analysis of competency
• Metacognitive / SRL as well as that trained by the simulator
• Surfacing of reflection / competency – social?
• Affect support
• Encouragement / engagement / motivation
• Please take our survey!!!

24. Acknowledgements
• The research leading to these results has received funding from the
European Community's Seventh Framework Program (FP7/2007-2013)
under grant agreement no 257831 (ImREAL project) and could not be
realized without the close collaboration between all ImREAL partners.

25. Thank-you!
http://bit.ly/smILEY
Case sensitive!!!
Adam.moore@tcd.ie
@adam__moore
@ImREAL_project / www.imreal-project.eu