1. Gyöngyvér Molnár
University of Szeged
http://www.staff.u-szeged.hu/~gymolnar
Technology-based Assessment:
Challenges and Promises
6th EDEN Research Workshop
Budapest, Hungary, 24-27 October, 2010
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged
„Developing Diagnostic Assessments” Project

2. Outline
• Need for change
• Possibilities and promises of TBA
• Implementing TBA in international projects
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

3. The need for change
• Technology abounds
• Change - Live, learn,
communicate
• New insight into the
learning process
• Potential to transform
education
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

4. The need for change
• New expectation,
lifestyle - the Net
Generation “digital
natives” => calls for
different type of teaching
• Schifting from generation
gap to generation lap
• New demands on the
education
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

5. The need for change
• Existing models of
assessment fail
• New assessments are
required
• Multi-sensory feature
• Retention rates
• Technology alone will not
result in higher achievement
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged
• Technology is not the issue - > catalyst

6. The need for change
• Data about education
quality and its impact on
economic output
• After 4 cycles of PISA
• Economic modeling
• E.g. how the GDP would
change according to the
change in educational
output
• Result: relatively small
improvements – large
impact on future
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

7. Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged
0
200
400
600
800
1000
1200
Mexico
Turkey
Greece
Portugal
Italy
Luxembourg
Spain
UnitedStates
Poland
Norway
SlovakRepublic
OECD
Hungary
Denmark
Germany
Iceland
France
Ireland
Sweden
Austria
Switzerland
Belgium
CzechRepublic
UnitedKingdom
Australia
NewZealand
Canada
Netherlands
Japan
Korea
Finland
Catching up with Finland
(in percent of GDP)
% current
GDP

8. Why is assessment so important?
„You can not improve what you can not measure”
(Schleicher, 2010)
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

9. The feedback in education
• ICT – growing in every segment of education
• International tendencies (OECD, ETS, NCES) in
educational assessment
• PPT reached its barrier
• Human resources, cost of money and time
• TBA solve this problem
• NEED to use technology to improve the efficiency of
assessment
• Improved quality of data and increased speed of
assessment
• Better conditions to improve the educational system
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

10. Possibilities and promises of TBA
(1)
• Reach stimulus material (multimedia tools)
• New item formats
• New construct (PS in technology reach environment)
• New areas (dynamic assessment) – assessing dynamic
– MicroDYN approach
– example
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

12. Possibilities and promises of TBA
(2)
• TBA rises new issues in assessment:
– background data – testing prozess - e.g. educational
data mining – log file analyses - issues of reaction
time, sequence
– annotations regarding eye and face tracking
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

13. Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

14. Possibilities and promises of TBA
(3)
• offers new assessment methods (adaptive testing)
– the difficulty of the test tailors dynamically to the
student’s ability level
– items are selected from an item bank dependent at a
difficulty appropriate for the student
– provides more time-efficient and accurate
assessments.
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

15. -4
-3
-2
-1
0
1
2
3
4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
logitscale
item
Adaptive testing
Person ability level
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

16. (4)
• increases motivation - students with high and low ability
level - flow experience
(5)
• changes the whole assessment process (item generation,
scoring, data-processing, information flow, feedback and
the speed of assessment)
• rapid and precise feedback
(6)
• poses new questions and problems – if TBA – replace PP
∑ improved quality and increased quantity of data and
increased speed of assessment
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged
Possibilities and promises of TBA

17. Implementing TBA in international
projects
• Europe: growing attention in several European countries
and the European Union also facilitates these attempts
• At national level Luxemburg leads the way:
– able to assess an entire cohort simultaneously
– automatic generation of the feedback to the teachers
– University of Luxembourg and the Public Research
Center Henri Tudor
– TAO (Testing Assisté par Ordinateur)
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

18. Implementing TBA in international
projects
• Germany – DIPF – TBA project :
– adapts and further develops the TAO platform
– establish a national standard for technology-
assisted testing
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

19. Implementing TBA in international
projects
• In Hungary – 2008 – first – media effect study:
– Inductive reasoning
– 5th grader (PP + CB)
– TAO platform
– Result: global achievement highly correlate – items,
persons – behave differently
• 2009 - a large-scale project was launched by the Center
for Research on Learning and Instruction - University of
Szeged - develop an online diagnostic assessment
system for the first six grades of primary school in
reading, mathematics and science
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

20. „Developing Diagnostic
Assessments” Project
• the development of assessment frameworks
• devising a large number of items both on paper and on
computer - building item banks
• research on comparing the achievements on the tests
using different media.
• long-term project aims: an online formative assessment
system for the first six grades of primary school
• First phase: pilot the system in 200 schools
• study the related technological and methodological
issues in detail
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

21. Implementing TBA in international
projects
• US - many instances
– CBA and online assessment so wide spread - testing
industry - engaged in writing and developing task and
different testing software
– Despite of this – in 2009 – Microsoft, Cisco and Intel -
transforming global educational assessment and
improving learning outcomes
– Teaching of 21st Century Skills project
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

22. Teaching of 21st Century Skills
project
• Aim: resolve the gap - exists between what goes on
in schools and what goes on in the real world
• better measure the skills - needed for the 21st-
century
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

23. Teaching of 21st Century Skills
project
• Specify high-priority skills, competencies, - 21st century -
turn these into measurable standards and assessment
framework
• Examine innovative learning environments and formative
assessments - draw implications for ICT-based
summative assessments
• Identify methodological and technological barriers –
support solutions
• implementation of these standards and methodologies,
pilot test, make recommendations for broader
educational assessment reform.
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

24. OECD PISA
• Started to implement CBA
• 2006 – CBAS
• 2009 - ERA
• 2012 - problem solving
• 2015 - completely to CBT
• still a lot of open questions
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

25. • Despite of the advantages - in large-scale national and
international assessment - faces several technological
challenges
• complexity of issues - no simple and clear solution
• Several technological means available in schools
• diversity, compatibility, connectivity require further
considerations
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

26. But the future is more promising…
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
Institute of Education,
University of Szeged

27. Thank you for your kind attention!
http://www.staff.u-szeged.hu/~gymolnar
Research Group on the Development of Competencies,
Hungarian Academy of Sciences
… and remember:
„Without data, you are just another person
with an Opinion” (Schleicher, 2010)