Presentation in the research group seminar, Institute of Informatics, Tallinn University, 7 March 2012. Based on the following publication: Põldoja, H., Väljataga, T., Tammets, K., & Laanpere, M. (2011). Web-based Self- and Peer- assessment of Teachers’ Educational Technology Competencies. In H. Leung, E. Popescu, Y. Cao, R. Lau, & W. Nejdl (Eds.), Advances in Web-Based Learning – ICWL 2011: 10th International Conference, Hong Kong, China, December 2011. Proceedings (pp. 122–131). Berlin / Heidelberg: Springer. http://www.springerlink.com/content/e3t2042568271213/

1. Web-based Self- and Peer Assessment
of Teachers Digital Competences
Mart Laanpere, Hans Põldoja

2. cba
This work is licensed under the Creative Commons
Attribution-ShareAlike 3.0 Unported License. To view a copy
of this license, visit http://creativecommons.org/licenses/by-sa/
3.0/ or send a letter to Creative Commons, 444 Castro
Street, Suite 900, Mountain View, California, 94041, USA.

3. Research context
• Importance of educational technology
competencies
• Generic ICT competency frameworks (e.g. ICDL)
do not cover all the competencies needed for
educational use of ICT
• Educational Technology Competency Model
(ETCM) for Estonian teachers
• DigiMina project for assessing teachers’
educational technology competencies

4. Research problem
To what extent and how could be
teachers’ educational technology
competencies assessed using a Web-
based tool?

5. Existing competency
frameworks
• International Computer Driving License
• UNESCO ICT Competency Framework for
Teachers
• ISTE National Educational Technology Standards
for Teachers (NETS-T)

6. • ECDL / ICDL
• Used in 148
countries
• Focused on ICT
usage
• Standardized
testing

7. • Launched 2008,
revised 2011
• Guidelines for
creating national
competency models
• 6 subdomains

8. • ISTE NETS-T 2008
• 20 competencies in
5 competency
groups
• Used in Norway,
Netherlands,
Finland, etc.

9. Design challenges

10. Design challenges
• How to define performance indicators of all
competencies in ETCM?
• How to select appropriate methods and
instruments for assessing competencies?
• How to implement selected assessment methods
in a Web-based tool?

11. Educational Technology Competency
Model for Estonian Teachers
• Based on ISTE NETS-T 2008
• 5-level assessment rubric developed by local
expert group

12. Measuring Educational
Technology Competencies
• Assessment methodology and instruments must be
reliable, valid, flexible, but also affordable with
respect to time and costs.
• Four levels of measuring competencies (Miller, 1990):
• knows
• knows how
• shows how
• does

13. Web-based Assessment of
Competencies
• Five-dimensional framework for authentic assessment (Gulikers et
al, 2004):
• tasks: meaningful, relevant, typical, complex, ownership of problem
and solution space;
• physical context: similar to professional work space and time
frame, professional tools;
• social context: similar to social context of professional practice
(incl. decision making);
• form: demonstration and presentation of professionally relevant
results, multiple indicators;
• criteria: used in professional practice, related to realistic process/
product, explicit

14. Design and
development of
DigiMina

15. http://trac.htk.tlu.ee/digimina/

16. Research-based design methodology
Contextual Inquiry Participatory Design Product Design Software Prototype
As Hypothesis
Concept mapping User stories
Information
architecture
Participatory design
sessions
Paper prototyping
Scenario-based High-fidelity
Personas Agile sprints
design prototyping
Adapted from (Leinonen et al, 2008)

17. Personas
• Teacher training master student
• Novice teacher
• Experienced teacher
• Educational technologist of a school
• Trainings manager (in a national organization)
(Cooper et al, 2007)

18. Scenarios
• Master student is evaluating her educational
technology competencies
• Peer assessment of problem solving tasks
• Educational technologist of a school is getting an
overview of teachers’ educational technology
competencies
• Training manager is compiling a training group with
sufficient level of competencies
(Carroll, 2000)

19. Stsenaarium 2: Pädevuste partnerhindamine
Kaisa on noor matemaatika õpetaja, kes on esimest aastat koolis. Õpetaja kutseaasta
käigus tuleb muuhulgas tähelepanu pöörata ka oma pädevuste kaardistamisele. Kaisa on
korra varem ülikooli ajal juba oma haridustehnoloogilisi pädevusi hinnanud, kuid
nüüd koolis on ta oma teadmisi ja oskusi praktikas saanud rakendada.
DigiMina keskkonda sisenedes näeb ta oma eelmisel aastal tehtud pädevuste hindamise
testi. Ta saab valida, kas alustab pädevuste testi täitmist eelmise aasta tasemelt
või päris algusest. Talle meenub, et eelmisel korral võttis kõigi ülesannete
lahendamine suhteliselt palju aega ning ta otsustab sel korral alustada testi
täitmist eelmise aasta tasemelt. Sel korral on lisaks valikvastustega küsimustele ka
probleemülesandeid, mille puhul tuleb vabas vormis omapoolne lahendus kirjutada.
Vastamise lõpus kuvatakse pädevusprofiili diagramm, kuid osad pädevused on sealt
puudu. Kaisa loeb kõrvalolevalt märkuselt, et vabas vormis küsimusi peavad hindama
teised DigiMina kasutajad ning täielikke tulemusi kuvatakse alles siis kui tema
vastused on hinnatud.
Uuesti avalehele jõudes kuvatakse Kaisale teadet, et ta võib osaleda teiste
kasutajate poolt sisestatud probleemülesannete lahenduste hindamises. Kaisale pakub
huvi, kuidas teised etteantud probleeme lahendasid, ning ta otsustab ühte
pädevustesti hinnata. Ülesande hindamisel peab Kaisa lisama omapoolse kommentaari
pakutud lahendusele ning valima menüüst hindepunktid. Hindamise lihtsustamiseks
kuvatakse talle ka hindamiskriteeriume. Kui esimene vastus on hinnatud otsustab ta
hinnata veel paari vastust. Kaisa ei näe, kelle vastuseid ta hindas, kuid ta oletab,
et tegemist võis olla teiste õpetaja kutseaasta üliõpilastega.
Paari päeva pärast saab Kaisa e-maili, et tema vastused on kahe DigiMina kasutaja
poolt hinnatud. Sisselogides näeb ta oma vastuste kohta kirjutatud lühikesi
kommentaare ning pääseb kokkuvõtva diagrammi juurde. Kokkuvõttes saab ta võrrelda
oma pädevuste taset ka õpetaja kutseaasta üliõpilaste keskmise pädevustasemega ning
kõigi kasutajate keskmise pädevustasemega.

20. Participatory design sessions
• 2 sessions
• Discussing the
scenarios
• Drawing the
sketches

23. Main concepts

24. Competency Test
• Competency test can be taken several times to
measure the advancement
• Usability issue: large number of tasks (20
competencies, 5 levels)
• Solutions:
• Can be saved and continued later
• Setting the starting level with self-evaluation

25. Tasks
• Task types:
• automatically assessed self-test items
• peer assessment task
• self reflection task
• Need to increase the number of competencies that
can be assessed with a self-test
• Peer assessment requires blind review from a user
in a same or higher competency level

26. Competency Profile
• Level of competencies is displayed as a diagram
• User can compare her competency level with the
average level of various groups
• Privacy settings (private, group members, public)
• Can be linked or embedded to external web pages

27. Group
• Typically created for a school or a group of
teacher training students
• Group owner can see competency profiles of all
members
• Anybody can create a group
• Groups can be set up as private or public

28. Competency Requirements
• Large number of competency profiles would make
DigiMina a valuable planning tool
• Competency requirements can be created by the
training manager, teacher trainer and group owner
• Will be implemented in a later phase

29. Current prototype

34. Task development

35. Assessment rubrics example
3.1. Demonstrate fluency in technology systems and the transfer of current knowledge to new
technologies and situations
1 2 3 4 5
Creates a user Manages access Solves Transfers working Chooses
account in a web- rights to the independently the methods from (compares,
based system and resources problems that known web evaluates) the most
creates/uploads published in the occur during the environment/ suitable tool for a
resources; uses web. use of ICT tools software to an given task.
common software/ (using help, manual, unknown
web environments/ FAQ or forums environment.
hardware with the when needed);
help of a user combines different
manual; uses tools; changes the
presentation tools settings of a web-
and a printer; based system.
saves/copies files to
external drive.

37. Example self-test task
• Screen recording of a teacher joining national
educational portal Koolielu and making several
errors during the process
• Multiple response question about the errors made

38. Example peer review task
• Teacher must adapt a given study guide to her own
working context (age range, subject area, software)

39. Example self-reflection task
• 4.1 level 1 — participates as a partner in a
multicultural project

40. Development of test items
• Test items are authored using IMS QTI compatible
test authoring tool TATS (Tomberg & Laanpere,
2011)
• 3 IMS QTI question types currently supported:
• choiceInteraction (multi-choice)
• choiceInteraction (multi-response)
• extendedTextInteraction

41. Future work

42. Development of DigiMina
• Visualizations and analytics
• Support for additional QTI question types:
orderInteraction and associateInteraction
• Competency requirements
• Integration with Koolielu portal

43. References
• Gulikers, J. T. M., Bastiaens, T. J., & Kirschner, P. A. (2004). A Five-Dimensional Framework for
Authentic Assessment. Educational Technology Research & Development, 52(3), 67–86.
• Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine,
65(9), S63–S67.
• Leinonen, T., Toikkanen, T., & Silvfast, K. (2008). Software as Hypothesis: Research-Based Design
Methodology. In Proceedings of the Tenth Anniversary Conference on Participatory Design 2008 (pp.
61–70). Indianapolis, IN: Indiana University.
• Cooper, A., Reimann, R., & Cronin, D. (2007). About Face 3: The Essentials of Interaction Design.
Indianapolis, IN: Wiley Publishing, Inc.
• Carroll, J. M. (2000). Making Use: Scenario-Based Design of Human-Computer Interactions.
Cambridge, MA: The MIT Press.
• Tomberg,V., & Laanpere, M. (2011). Implementing Distributed Architecture of Online Assessment
Tools Based on IMS QTI ver.2. In F. Lazarinis, S. Green, & E. Pearson (Eds.), Handbook of Research
on E-Learning Standards and Interoperability: Frameworks and Issues (pp. 41–58). IGI Global.

44. Thank you!
• mart.laanpere@tlu.ee
• hans.poldoja@tlu.ee
• http://trac.htk.tlu.ee/digimina/