A talk given at the conference Examining Excellence, London191206.

1. Some experiences from evaluating
digital examination systems
Per Olof Arnäs
Chalmers University of Technology

2. I do not like exams.
really

3. Why do we give exams?

4. Why do we give exams?
To teach?
To pass
verdict?

5. Why do we give exams?
To pass
verdict?
Yes, in a sense.
We are obligated
to evaluate
knowledge.

6. Why do we give exams?
To teach?
Yes!
We are also
obligated to
transfer
knowledge.

7. Me
I love the 21st
century
Senior lecturer inlogistics at ChalmersUniversity of
Technology
Co-produced two
MOOCs in Logistics
Research on thedigitalization of thetransportation industry
Per Olof Arnäs
Vice Dean of Education
Dep. of Technology Management
and Economics
Chalmers University of Technology

8. My beliefs
Society does not
want test-takers
Understanding is muchmore important thanknowledge of facts
Written exams is often a badway to ensure knowledge
and understanding
My job is to make mystudents understandthe subject better andfaster than I did as astudent myself Digital development is
key to human evolution

9. Active learning Blended learning
• Students are involved more than listening
• Less emphasis is placed on transmitting information
and more on developing students’ skills
• Students are involved in higher-order thinking
(analysis, synthesis, evaluation)
• Students are engaged in activities (e.g. reading,
discussing, writing)
• Greater emphasis is placed on students’ exploration
of their own attitudes and values
(Bonwell and Eison 1991)
Bonwell, C. C. and J. A. Eison (1991). Active Learning: Creating Excitement in the Classroom. 1991 ASHE-ERIC Higher Education Reports, ERIC.
Garrison, D. R. and H. Kanuka (2004). "Blended learning: Uncovering its transformative potential in higher education." The Internet and Higher Education 7(2): 95-105.
Blended learning uses a combination of
face-to-face learning with asynchronous
content (on the internet) and has a large
transformative potential (Garrison and
Kanuka 2004).

10. Why do we give exams?
To teach?
This means that
the exam should
be part of the
learning process
of the student!

11. The short version: It’s a definite upgrade from analog

12. The ultra short version:
The Devil is in the details

13. The cost of things

16. Economy-of-Scale

17. Learning curve for the teacher

18. Fixed costs are going down

19. Administration
cost is lowered
(a lot)

20. Economy-of-Scale incl. administration

21. Evaluated systems
Ping pong
DigiExam
Inspera

22. Three processes
Student Teacher Admin

23. Three phases
Before During After

24. Students before exam
Preparations
Download
software
Do test
exam
HardwareOK?
Borrow
computer?

25. Students before exam
+
Ability to test
system
-
Learning curve
Some planning
needed

26. Students during exam
User
interfaceOnboardingInternet
access
Login
issues
Math
Images

27. Students during exam
+
Faster than analog
No handwriting
Copy/paste
Higher quality of
answers
-
Some prefer analog
Connectivity issues
Compatibility issues
Math
Images

28. Students after exam
FeedbackLearning!
Results
accessible

29. Students after exam
+
Rich feedback
Swift feedback
Learning
opportunity
-
?

30. Teachers before exam
Create examQuestions
Feedback texts
Solutions
Prepare
students
Hardware?
Software?
Question
types

31. Teachers before exam
+
Re-use questions
Multiple teachers
-
Prepare students
Some question
types still missing
Enter solution when
entering question

32. Teachers during exam
Stand-by
Answer
questions
Assist in
onboarding

33. Teachers during exam
+
Message to all
students via system
-
?

34. Teachers after exam
Grading
Multiple
graders
Rich
feedback
Pre-written
feedback

35. Teachers after exam
+
Multiple graders
Automatic grading
Fast grading of essays
Rich feedback!
-
Need for mouse
Want to add same
feedback texts to all
students
PDF-comments
Sorry, but this sucks!

36. Admin before exam
Room(s) People
Extra
hardware
Outlets
Wifi
Training
Support
Management

37. Admin before exam
+
Centralised
management
-
Special rooms
Trained personnel
Computer lending

38. Admin during exam
Security Support
Wifi security
Visible
screens Onboarding
Lending of
computers
Offline
support

39. Admin during exam
+
Digital overview
Secure
-
New type of security
needed
Connectvity issues
Power failures

40. Admin after exam
Register
results
LMS
integration
Archive?

41. Admin after exam
+
Fraction of
workload
compared to
analog
-
Manual transfer of
results to LMS
Infinite archive

42. Grading - some ideas
Word cloud
from essays
“Flagged”
words
Highlight
in answers
Canned
feedback
“Learn” during
grading
Verbal/videocommenting
Secondopinion
Send question
to colleague

43. Grading - some ideas
See statistics
Per
question
Diagrams
See totals for
anonymous
students

44. Grading -
some
ideas
Example of a
very difficult
question

45. Insights - digital exams…
+
Save money
Free up time
Are scalable
Can be used to
increase learning
-
Need physical
infrastructure
Need new
competences
Difficult to go back…

46. 100%
Trad. exams
Dig. exams
The transition

47. 100%
Scenario 1: Big bang
Scenario 1: Big bang
Trad. exams
Dig. exams
The transition

48. 100%
Scenario 1: Big bang
Scenario 1: Big bang
Scenario 2: Messy hybrid
Scenario 2: Messy hybrid
Trad. exams
Dig. exams
The transition

49. 100%
2019
Scenario 1: Big bang
Scenario 1: Big bang
Scenario 2: Messy hybrid
Scenario 2: Messy hybrid
Trad. exams
Dig. exams
The transition

50. 100%
2019
Scenario 1: Big bang
Scenario 1: Big bang
Scenario 2: Messy hybrid
Scenario 2: Messy hybrid
Danger
Zone
Trad. exams
Dig. exams
The transition

51. 100%
2019
Scenario 1: Big bang
Scenario 1: Big bang
Scenario 2: Messy hybrid
Scenario 2: Messy hybrid
Danger
Zone
Trad. exams
Dig. exams
The transition
Do not
underestimate
this!

52. Why do we give exams?
To teach!
This means that
the exam must be
part of the
learning process
of the student!

53. Some experiences from evaluating
digital examination systems
Per Olof Arnäs
Chalmers University of Technology
per-olof.arnas@chalmers.se