Viewbrics: mirroring and mastering complex generic skills with video enhanced rubrics through a technology-enhanced formative assessment methodology by Ellen Rusman (OUNL).

1. Viewbrics: mirroring and mastering complex generic skills
with video enhanced rubrics
through a technology-enhanced
formative assessment methodology
Dr. Ellen Rusman
Welten Institute, Open University of the Netherlands
& Viewbrics project team
Webinar during EMPOWER e-Assesssment week,
28th of June 2018, online:
https://empower.eadtu.eu/events/100-empower-events
‘Like master, like man’

2. Complex generic (transversal) skills
Complex skills consist of constituent subskills which concertation
require high cognitive effort and concentration [1][2], and
prolonged repetitive practice in order to master them.
Complex generic (also ‘transversal’ or ‘21st century’) skills are not
specific for a domain, occupation or type of task, but important
for all kinds of work, education and life in general. These skills are
applicable in a broad range of situations and many subject
domains [3].
=> many schools are struggling with how to teach and evaluate
students’ performance of complex generic/transversal skills in
their daily educational practice [4].

3. Who’s interested in this webinar?
Are you a:
A. Teacher in Higher Education
B. Manager in Higher Education
C. Researcher of Higher Education
D. Advisor/trainer in Higher Education
E. Otherwise, namely: ………………………

4. Your opinion:
Agree/disagree/neutral?
It is important for students to learn complex generic skills
I know how I can support students’ learning complex generic skills in my lessons
I have enough educational methods and material available to support students
while learning complex generic skills

6. https://commons.wikimedia.org/wiki/File:Rubric.jpg

7. New Zealand

8. However….
• many aspects of complex skills mastery refer to motoric activities, time-consecutive
operations and processes that are hardly captured in text (e.g. body posture or use
of voice during a presentation)
• context in which the skill is practiced is important for its learning, as it implies and
generates implicit knowledge (tacit knowledge, ‘knowing how/’knowing why’),
which is interwoven with practical activities, operations and behavior in the
physical world [11].
• text supposedly leaves more space for personal interpretation of the performance
indicators of a complex skill than video
=> text-based rubrics only have a restricted capacity to clarify the targeted mastery
level of a skill and to assess shown behaviour [12].

9. Towards Video-Enhanced Rubrics (VER)
A Video-Enhanced Rubric (VER) is the synthesis of video modelling
examples and a text-based analytic rubric in a digital formative
assessment format.
We expect that:
• video-enhanced rubrics can foster learning from observation of
(good/bad) video modelling examples [11] [12][13], thus
supporting (more concrete and consistent) mental model
formation
• improved quality of feedback (concreteness, consistency &
perceived quality) given by teachers or peers during skills
training,
=> more effective skill mastery

12. Financed by:

13. Two main ‘research phases’:
1. Design research (ended)
2. Semi-experimental research (now in data collection phase)
Design research for development of:
- Ecologically valid rubrics for 3 skills
- Video-modelling examples
- Feedback-and reflection methodology

15. Design-based research approach

16. Presenting Information literacyCollaborating

17. Evaluation of rubrics with teachers in Viewbrics
project team

18. Evaluation of language &
feasibility of rubrics with pupils

19. Skills hierarchy ‘Collaborating’
Complex skill
Skill cluster
Sub-skills

20. Skills hierarchy and rubric ‘Collaborating’
Mastery level descriptions

21. Skills hierarchies for
‘Information literacy’ & ‘Presenting’ skills

22. Design based research

23. Online feedback & reflection tool
• Make rubrics and video modeling examples for complex generic skills available to
support practicing skills
• Prevent piles of ‘paper work’
• Convenient, regular and structured feedback in a ‘set’ format while practicing skills:
• Video-enhanced rubrics
• Dashboard with automatically summarized, immediate and timely personal
feedback
• Insight in what goes well and what may still be improved by means of
‘reflection wheel’ visualization and (essential) tips and tops
• Individual reflection support on performances and mastery of skills
• Automatic and well-organized registration and archiving of performances and
growth in mastery of skills

24. Evaluation of Viewbrics methodology
with pupils (Sint Jan)

25. Pilot evaluation of Viewbrics with pupils - Grotius

26. Pilot evaluation of Viewbrics with pupils - Grotius

27. Answering questionnaires - Grotius

28. Assessing a ‘dummy student’s’ presentation
performance - Porta Mosana

29. Pilot evaluation of Viewbrics with teachers
and pupils – Porta Mosana

30. Describing experiences with Viewbrics tool –
pupils Porta Mosana

31. A positive evaluation of the Viewbrics tool –
pupils Porta Mosana

32. The result: feedback- and reflection process in online tool
1. Watch video-enhanced rubrics
2. Practice skills
3. Assess own performance
4. Look at and analyze
feedback of teacher and peers
5. Determine (next) learning objectives

33. c
Messages with tasks

34. Looking at the video-modeling examples with processing support and ‘linking’ questions
(linking video modeling examples to sub-skills in rubric)

35. Processing support and ‘linking’ question ‘Linking’ video fragments
to sub-skills

36. Answering the ‘linking’ questionAnswering the ‘linking’ question

37. Providing rubric
description of the
highest mastery
level as answer model to
pupils

38. • Watch complete video
without questions
• Reviewing video fragments organized
by sub-skill

39. Practicing a complex generic skill: ‘Presenting’

40. Self-assessment on
performance

41. Skill-cluster
Sub-skill

47. Providing feedback to other pupils

50. Research is still ongoing, first results on effectiveness expected October 2018

51. Your opinion:
Agree/disagree/neutral?
Viewbrics can help students to provide insight in their growth of complex
generic skills.
I think Viewbrics would also be applicable in the context of higher
education
I would like to know more about Viewbrics

52. Video: https://www.youtube.com/watch?v=nnQb7zSPkmg

53. Also interested? Contact us!
Ellen.Rusman@ou.nl
Project leader/principal investigator
@EllenRusman
More info on Viewbrics:
website: www.viewbrics.nl
Do you want to stay updated?
Register for updates via:
Mieke.Haemers@ou.nl

54. Thanks for your attention!
We would like to gratefully acknowledge the funding of our
project by the practice-oriented research programme of the
Netherlands Initiative for Education Research (NRO), part of
The Netherlands Organisation for Scientific Research (NWO).

55. References
[1] P. Kirschner and J. Van Merriënboer, “Ten Steps to Complex Learning A New Approach to Instruction and
Instructional Design,” in In T. L. Good (Ed.), 21st century education: A reference handbook, Thousand Oaks, CA: Sage.,
2008, pp. 244–253.
[2] D. Galligan, Frank; Maskery, Colin; Spence, Jon; Howe, David; Barry, Tim; Ruston, Andy; Crawford, Advanced PE for
Edexcel. Oxford: Heinemann Educational Publishers, 2000.
[3] K. Bowman, “Background paper for the AQF Council on generic skills,” Canberra, 2010
[4] E. Rusman, A. Martínez-monés, J. Boon, M. J. Rodríguez-triana, and S. Villagrá-sobrino, “Gauging Teachers ’ Needs
with Regard to Technology- Enhanced Formative Assessment ( TEFA ) of 21 st Century Skills in the Classroom,” in
Computer Assisted Assessment. Research into E-Assessment. Proceedings of the International Computer Assisted
Assessment (CAA) Conference, 2014, pp. 1–14.
[6] C. Sluijsmans, Dominique; Joosten-ten Brinke, Désirée; Van der Vleuten, “Toetsen met leerwaarde,” Heerlen,
Maastricht, 2013.
[7] J. J. G. Van Merriënboer, Training Complex Cognitive Skills. Educational Technology, 1997.
[11] W. Westera, “On the Changing Nature of Learning Context: Anticipating the Virtual Extensions of the World.,” Educ.
Technol. Soc., vol. 14, no. 2, pp. 201–212, 2011.
[12] O. Berry, M. Price, C. Rust, B. O. Donovan, M. Price, and C. Rust, “Teaching in Higher Education: Know what I
mean ? Enhancing student understanding of assessment standards and criteria,” Teach. High. Educ., no. March 2015, pp.
37–41, 2007.
[13] H. Rohbanfard and L. Proteau, “Live versus video presentation techniques in the observational learning of motor
skills,” Trends Neurosci. Educ., vol. 2, no. 1, pp. 27–32, 2013.