Slides form the keynote at the Simposio Internacional de Informática Educativa (SIIE 2018) http://siie2018.uca.es/index.php/en/keynotes-en/ Abstract: In the context of the 4th industrial revolution and a globalized world, there is a pressing need for continuous acquisition and update of skills to maintain efficiency and to ensure inclusion and participation of all citizens in the globalized workplace. At the highly automated and rapidly updated workplaces, the need for expertise and effective training is growing. In the EU-funded research-and-innovation project WEKIT, we address these challenges by developing a new approach to industrial training. This approach is based on the idea of using wearable sensors to capture expert performance and then making it available for trainees using Augmented Reality. The WEKIT training methodology and the technological platform allow creating effective educational experience efficiently using the time of the expert involved in content creation. The idea of capturing workplace experience finds another application area in the research project Virtual Internship, funded by the Norwegian welfare authority. In this project, we use augmented and virtual reality to increase awareness of schoolchildren about various professions and improve motivation of young unemployed to search for a new job. We aim to find out if immersive and interactive experiences of exploring workplaces and trying typical tasks can help in mitigating the youth unemployment.

1. Industrial Training and Workplace Experience
with Augmented and Virtual Reality
InternationalSymposiumonComputersinEducationSIIE2018
siie2018.adie.es
University of Cádiz, Jerez Campus
19-21 September 2018
Mikhail Fominykh
Researcher, Norwegian University of Science and Technology, Norway
Project manager, Europlan UK ltd, United Kingdom

4. # 427/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING

5. # 527/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING

6. # 627/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING

7. Can Technology improve
hands-on learning at the
workplace?
? ???

9. Computerization
Freya and Osborne (2016) The future of employment: How susceptible are jobs to computerisation?
https://doi.org/10.1016/j.techfore.2016.08.019

10. Computerization

11. Industry 4.0

12. More and more
complex workplaces
have to be managed by
less and less
people

13. Workplace training
will become
more and more
frequent

14. Augmented Reality

15. https://vimeo.com/133828847
Giuseppe Scavo, Fridolin Wild and Peter Scott (2015) The GhostHands UX: telementoring with hands-on
augmented reality instruction, DOI: 10.3233/978-1-61499-530-2-236
Enriching the ‘real’ world with virtual visual overlays

16. vtt.fi
Enriching the ‘real’ world with virtual visual overlays
Kaj Helin, Timo Kuula, Jaakko Karjalainen, Iina Aaltonen, Fridolin Wild, Juhani Viitaniemi, Antti Väätänen
(2015) Usability of the ARgh! Augmented Reality system for on-the-job learning
See video at:
https://www.youtube.com/watch?v=Snoyt5-pZRY

17. Augmented Reality for a
human in Industry 4.0

18. Error reduction in
the industry
The use of Augmented Reality when
performing a 46-step task ranging in
complexity from selecting the correct parts,
to properly aligning and fastening bolts
through multiple parts reduced time to task
completion by 30% and reduced errors (first
time quality results improved 90%).
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 18
RICHARDSON ET AL. (2014)
http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1090&context=imse_conf

19. AR and wearables
in Society
Page 19WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING27/10/2018
o Wearables
o Games
o Tablets
o AR glasses

20. AR and wearables
in the Industry
o Operations
o Maintenance
o Error prevention
o Training
Date Page 20

21. Wearable Experience for
Knowledge Intensive Training -
WEKIT
Disclaimer
This project has received funding from the European Union’s
Horizon 2020 research and innovation programme under
grant agreement No 687669. http://wekit.eu/

22. WEKIT partners

23. Training with AR –
knowledge conversion

24. Experience and knowledge
Learning = converting
experience to knowledge
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 24
Mikhail Fominykh, Fridolin Wild, Carl Smith, Victor Alvarez and Mikhail Morozov: "An Overview of Capturing
Live Experience with Virtual and Augmented Reality”, DOI: 10.3233/978-1-61499-530-2-298.

25. Experience and knowledge
Learning =
= converting experience to knowledge
immediate experience
(aka ‘practice’)
information for a master
level
(aka ‘theory’)
separated
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 25

26. Experience and knowledge
Learning =
= converting experience to knowledge
immediate experience
(aka ‘practice’)
information for a master
level
(aka ‘theory’)
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 26

27. Experience and knowledge
Learning =
= converting experience to knowledge
immediate experience
(aka ‘practice’)
information for a master
level
(aka ‘theory’)
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 27

28. Experience and knowledge
Learning =
= converting experience to knowledge
immediate experience
(aka ‘practice’)
information for a master
level
(aka ‘theory’)
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 28

29. Experience and knowledge
Learning =
= converting experience to knowledge
immediate experience
(aka ‘practice’)
information for a master
level
(aka ‘theory’)
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 29

30. Experience and knowledge
Learning =
= converting experience to knowledge
immediate experience
(aka ‘practice’)
information for a master
level
(aka ‘theory’)
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 30

31. Experience and knowledge
Learning =
= converting experience to knowledge
immediate experience
(aka ‘practice’)
information for a master
level
(aka ‘theory’)
Experiencedlearner
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 31

32. Why WEKIT.one
There are 26 million active enterprises with some 144
million persons employed in Europe alone. One third of
industrial enterprises in Europe offer continuing
vocational training.
Industries need high-quality specialized workplace
training if they want to stay competitive. Such training is
either ineffective or expensive. This makes expertise
transfer very difficult.
Holographic training and experience capturing enabled
by Wearables and Augmented Reality can address these
challenges by offering an effective training while
reducing the trainer’s workload.

33. Why WEKIT.one
There are 26 million active enterprises with some 144
million persons employed in Europe alone. One third of
industrial enterprises in Europe offer continuing
vocational training.
Industries need high-quality specialized workplace
training if they want to stay competitive. Such training is
either ineffective or expensive. This makes expertise
transfer very difficult.
Holographic training and experience capturing enabled
by Wearables and Augmented Reality can address these
challenges by offering an effective training while
reducing the trainer’s workload.

34. Why WEKIT.one
There are 26 million active enterprises with some 144
million persons employed in Europe alone. One third of
industrial enterprises in Europe offer continuing
vocational training.
Industries need high-quality specialized workplace
training if they want to stay competitive. Such training is
either ineffective or expensive. This makes expertise
transfer very difficult.
Holographic training and experience capturing enabled
by Wearables and Augmented Reality can address these
challenges by offering an effective training while
reducing the trainer’s workload.

35. WEKIT.one
Industrial Training Platform
WEKIT.one platform allows to create training scenarios in three steps
Capture
Experience
Select appropriate
Transfer Mechanisms
from the library and
demonstrate the tasks
and procedures by
performing them.
Re-enact
Experience
Perform the tasks and
procedures guided by
the captured
demonstrations and
assisted by formative
feedback.
Review and
Analyze
Review the recordings
in post-analysis,
compare performances,
discover parts where
improvement is
needed.

36. WEKIT methodology
Capturing

37. Capture Experience

38. Capture Experience
See video at:
https://www.youtube.com/watch?v=AiKt9wJ3chg

39. WEKIT methodology
Re-enactment

41. Re-enact Experience

42. WEKIT methodology
Review

43. Review and Analyze
Experience

44. WEKIT methodology
Learning
Experience
Modeling

45. Compatible with the draft IEEE
ARLEM Standard
Augmented Reality Learning Experience Model
(ARLEM) is an integrated conceptual model and the
according data model specifications for representing
activities, learning context and environment.

46. 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 46
Augmented
Reality
Learning
Experience
Model
P15891 working group of the
IEEE standards association
http://arlem.cct.brookes.ac.uk

47. WEKIT Software
Trainer features
• Virtually annotating objects in the physical space (text, image, video, audio, 3d objects)
• Capturing expert performance using a multimodal sensor recording
• Transforming recording into ARLEM

48. WEKIT Software
Trainee features
• Importing ARLEM recording
• Automatically generated task list and guide functions to all annotations
• Contextualized multimodalguidance and re-enactment of experience
• Capturing trainee performance

49. WEKIT Software
Analysis features
• Mapping data to ARLEM
• Visualization of performance and biological data
• Comparison of multiple performances

50. WEKIT Hardware

51. 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 51
Participatory design
See video at:
https://www.youtube.com/watch?v=ND913dF8iyc

52. Wearable
prototype 1
Wearable
prototype 2

54. Wearable
prototype 3

56. Wearable Sensors and
Augmented Reality
Smart glasses
Smart glasses are worn on
the head, currently
Microsoft Hololens is used.
It functions both as one of
the sensors and as the
main augmenting device.
Posture
A combination of
accelerometers and
gyroscopes capture
posture movements of
the user in the
environment for real time
feedback and warnings.
Bio sensors
A number of sensors
collect biological data to
provide necessary
feedback and warnings in
real time or in post-
analysis, detecting such
states as stress, fatigue or
lack of focus.

57. Wearable Sensors and
Augmented Reality
Gestures
Hand gestures are
captured if they are
necessary to understand
the performance. Gestures
are also used to interact
with the system.
Force feedback
By applying silent
vibrational feedback on the
arms, the users can be
given guided feedback on
their actions.
Wearability
The hardware and sensors
are designed with
wearability and fashion in
mind.

58. Training Methodology
Research-based
Specifically for industrial training at the workplace
Designed for Augmented Reality and Wearables

59. Training Methodology
Step 1. Prepare
Break down complex tasks to subtasks, identify properties
of subtasks and
Select Transfer Mechanisms

60. Training Methodology
Step 2. Capture
Demonstrate each subtask while wearing the WEKIT
wearable solution
Step 3. Re-enact
Perform the tasks and procedures guided by the captured
demonstrations and assisted by formative feedback.
Step 4. Review and Analyze
Review the recordings in post-analysis, compare
performances, discover parts where improvement is
needed.

61. Aircraft Maintenance
Use Case

62. Medical Training
Use Case

63. Astronaut Training
Use Case

64. Evaluation results
from trials 1 in 2017
have to be managed by Trials 2 are running
now Summer-Autumn 2018

65. TAMARA
Technology Acceptance Model for AR/WT (TAMARA) questionnaire
contains some questions to check the technology acceptance of the
user toward the proposed prototype. The questionnaire has been
developed in Wild et al. (2017).
The first 19 items use a 7-point Likert scale ranging from strongly
disagree (1) to strongly agree (7), while the final item, usage frequency,
is rated on a 6-point scale. They are presented to the experiment
participants in the questionnaire in the following manner:
ATU4 I look forward to those aspects of my job that require me to use AR & WT.
65
strongly
disagree (1)
disagree
(2)
somewhat
disagree (3)
neither agree
or disagree
(4)
somewhat
agree (5)
Agree
(6)
strongly
agree (7)

66. 66

67. SUS
System Usability Scale (SUS) is a tool for measuring both usability and
learnability. The SUS scores calculated from individual questionnaires represent
the system usability. SUS yields a single number representing a composite
measure of the overall usability of the system being studied. Scores for
individual items are not meaningful on their own. SUS scores have a range of 0
to 100 (Brooke, 1996; 2013). According to validation studies, the acceptable SUS
score is about 70 (Brooke, 2013; Bangor et al., 2009).
SUS is based on John Brooke (2013): SUS: A Retrospective, In: Journal of
Usability Studies, 8(2):29-40
67

68. SUS
68

69. SSQ
The Simulator Sickness questionnaire (SSQ) was filled in by the
participants before and after the test. The questionnaire before the test
is used to make sure that the user does not have medical issues since
the test has to be done only with healthy subjects. The questionnaire
after the test is used to check the symptoms of the participants for
three main areas: Nausea, Oculomotor and Disorientation.
According to the NASA Johnson Space Centre Table, the score of the
SSQ questionnaire are interpreted as: no symptoms if score < 0,
negligible symptoms if score <5, minimal symptoms if score is between
5-10, significant symptoms if scores is between 10-15, concerning if
score is between 15-20 and problem simulator if >20.
69

70. SSQ
70

71. SGUS
Smart Glasses User Satisfaction (SGUS) questionnaire was created for the
WEKIT trials. SGUS measures subjective satisfaction focusing especially on test
participants’ experiences on the features that support learning. SGUS is based
on the evaluation criteria for web-based learning by Ssemugabi & de Villiers
(2007) and statements taken from Olsson, T. (2013) ‘Concepts and Subjective
Measures for Evaluating User Experience of Mobile Augmented Reality
Services’. SGUS consists of 11 items (statements) with a seven point Likert scale
(1-7).
For the analysis, the overall average from all items was calculated. Then this
overall average was compared with averages from individual items on a scale of
1-7.
71

72. SGUS
72

73. QUIS
The Questionnaire for User Interaction Satisfaction (QUIS) measures subjective
satisfaction with specific aspects of the interface, including usability and user
experience (Chin et al., 1988). It also measures the feeling about the AR glasses.
QUIS was modified, using only the relevant items from the viewpoint of this
study. Altogether 15 QUIS items with a scale mapped to numeric values of 1 to 7
were used.
For the analysis, the overall average from all items was calculated. Then this
overall average was compared with averages from individual items.
73

74. QUIS
74

75. TM
The Transfer mechanism (TM) Questionnaire was designed to evaluate
the adherence of the prototype to the Wekit framework.
Each Transfer Mechanism was formulated into one or several self
reflective statements
Participants had no pre-knowledge of Transfer Mechanisms and were
unaware that they were following the framework
Both experts and students were asked to rate statements that were
oriented to the successful implementation of each Transfer Mechanism
after being exposed to the prototype.
75

76. TM
76
LT
EB
IT
AL
TE
C

77. General conclusions
• The organization and execution of the 3 trials was very successful
• The use of questionnaires and interviews to quickly collect feedback from the
users were a very good idea
• The second iteration will implement the results and feedback collected during
this first campaign of test
• Particular emphasis in enhancing the design and the usability of the WEKIT
applications
• For both evaluation cycles we promised 600 people in total
• We have achieved around 50 per Industrial case during iteration 1
• We had almost 150 people for Iteration 1; that leaves 450 people for Iteration 2,
•Options (to be discussed):
• 1) additional smaller trials
• 2) more formal feedback gathered at conferences, workshops and fairs
• 3) online surveys with videos and explanations (TAMARA, House of Quality)
WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 7705/07/2017

78. 27/10/2018 78

80. Object-
centric
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 80
Action-
centric

81. Information
communication
27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 81
Experience-
communication
Images: http://www.unmuseum.org/, http://en.wikipedia.org/, http://digitalxtrememedia.com/;
https://www.youtube.com/watch?v=7d59O6cfaM0, https://www.microsoft.com/en-us/research/project/holoportation/

82. Virtual Internship in VR/AR
with game elements
Mikhail Fominykh
Innovative Immersive Technologies for Learning, NTNU
mikhail.fominykh@ntnu.no

83. Virtual Internship
Project summary
►Simulation of workplaces with VR/AR
►Job taste: give an insight to professions
►Mastery/coping and security feeling about the
typical job tasks
▼Mapping the user needs
▼Development of prototypes with Industry partners
▼Evaluation

84. Healthcare

85. Healthcare

86. Healthcare

87. FiskeoppdrettFish farming

88. Fish farming

89. Fish farming

90. Office work / startup

92. Main results of Evaluation in cycle 1
• Both councelors and young unemployed replied positively
towards the concept of virtual internship.
• The apps must be improved to reach the full potential.
• The participants did not agree if gaming elements should
be central or not, but collaboration and storytelling were
rated positively
• Many participants wished to have more workplaces.
• There was no agreement among the participants if 360
videos or graphical simulations were best.

93. Figur 11: Jeg kan tenke meg å bruke appene ofte gjennom NAV. 1) Svært uenig, 2) Litt
uenig, 3) Verken enig eller uenig, 4) Ganske enig, 5) Svært enig
I would use such apps often at welfare centers
Fully disagree Fully agree

94. Phase 2 Interview

96. Phase 2. Fishery VR
See video at: https://youtu.be/Ry--5VdjTUA

97. Q & A
Slides: https://www.slideshare.net/mfominykh