1. Computer based testing vs. paper based testing
Nynke R.Bos,Martijn van der Weerd, Roel Sijstermans
Academic Medical Center – University of Amsterdam
The Netherlands
Background
The ‘Klinische Lijn’ (KL) exam is the final exam students have to pass
before they start their internship. The KL is a paper based exam
consisting of 130 case based questions. Developing a computer based
version of the KL offers the possibility of adding multimedia to the KL
exam. Also, the organization of the KL is a complex logistical process,
since eight different medical specialties are involved in providing and
grading the questions. Digitalizing the KL would therefore reduce
logistical problems.
Methods
The first step in digitalizing the KL is to establish whether there are differences between setting a paper based test
and a computer based test. 187 students were assigned to do half of the test on computer and the other half on
paper. Group A (n=99) did the first part on the computer and the second half on paper. Group B (n=88) did the first
part on paper and the second part on the computer. The other 189 students did the whole test on paper at a
different location, due to a shortage of available computers.
A comparison was made on word count, but only with those questions that needed an elaborative answer. Also, the
p-values for each version and each question were calculated to establish whether there was a version of the test
that proved easier.
Results
During the exam, students had the choice to start with either the
digital or the paper part of the test. Nearly all students chose to start
with the digital form. 69.8% of the total number of students (n=376)
passed the KL exam.
The word count proved that students used more words in the digital
part of the exam than in the paper based version (36.2 ± 15.5 vs. 22.3
± 4.9) (P<.05)
An increased word usage would suggest a higher p-value for the
digital test. The average p-values are more or less the same suggesting
that a higher word usage on the computer does not affect the
difficulty level of the computer based test. (0.60 ± 0.18 vs. 0.59 ± 0.17)
Some p-values of the paper version made by group B were higher
compared to those of group A and the control group. Students in
group B started with the second part of the test, which might explain
this difference.
Conclusion
Students use more words on a computer based test than on a paper based test. No significant differences were
found between the p-values, making each test evenly difficult.
Students seemed to prefer the digital settings over the paper based version. It is important that the sequence of the
questions is the same for all students, in order to have equal p-values for all questions.