The Review looks at the various components of assessment and evaluation frameworks. These include: Student assessment; Teacher appraisal; School evaluation. Education system evaluation; Other types of evaluation (programme evaluation, evaluation of school leadership etc)

1. OECD Reviews of
Evaluation and Assessment in Education: Czech Republic
By Paulo Santiago, Alison Gilmore, Deborah Nusche and Pamela Sammons
www.oecd.org/edu/evaluationpolicy
Conference, Prague
The main conclusions and policy
recommendations
Presentation by
Paulo Santiago
Directorate for Education, OECD
15 March 2012

2. Outline of presentation
 Part 1
The OECD Review on Evaluation and Assessment Frameworks for
Improving School Outcomes and the Review of Evaluation and
Assessment in the Czech Republic
 Part 2
Conclusions and Recommendations
• The Evaluation and Assessment Framework
• Student assessment
• Teacher appraisal
• School evaluation
• Education system evaluation

3. PART 1
(1) The OECD Review on Evaluation and
Assessment Frameworks for Improving School
Outcomes and;
(2) the Review of Evaluation and Assessment in
the Czech Republic

4. OECD REVIEW - OBJECTIVES
●Purpose: To explore how systems of evaluation and assessment can be used to
improve the quality, equity and efficiency of school education.
A Review of national approaches to evaluation and assessment in school education
●The Review:
 Synthesises research-based evidence on the impact of evaluation and
assessment strategies and disseminate this knowledge among countries.
 Identifies innovative and successful policy initiatives and practices.
 Facilitates exchanges of lessons and dialogue.
 Identifies policy options for governments to consider.

5. OECD REVIEW - SCOPE
• The Review looks at the various components of assessment and evaluation
frameworks. These include:
• Student assessment;
• Teacher appraisal;
• School evaluation;
• Education system evaluation;
• Other types of evaluation (programme evaluation, evaluation of school
leadership etc)
• Comprehensive approach: Investigation of each component individually, as
well as the coherence of the framework as a whole (including the links between
the different components).

6. OECD REVIEW - KEY ISSUES
Key policy issues for analysis
– Governance: Designing a systemic framework for evaluation and assessment
– Procedures: Ensuring the effectiveness of evaluation and assessment procedures
– Competencies: Developing competencies for evaluation and for using feedback
– Use of results: Making the best use of evaluation results
– Implementation: Implementing evaluation and assessment policies

7. OECD REVIEW - BROAD FEATURES
●Oversight by the Group of National Experts (GNE) on Evaluation and Assessment
https://community.oecd.org/community/evaluationpolicyinschools
●Countries working collaboratively with the Secretariat
Countries do part of the work of the Review, a national co-ordinator is nominated, dialogue within
countries is encouraged
●Countries exchanging lessons and experiences, in particular through GNE
meetings
●Collection of views and perspectives from a range of stakeholders
Through their participation in GNE meetings and their interaction with Country Review expert
teams
●A diverse set of external experts involved in the Country Reviews
Involvement of top academics and policy makers to bring expertise into the Review
●Ensuring a range of principles are respected
Independence of analysis; inclusiveness in terms of the views and perspectives which are collected;
extensive reviews of the literature; evidence-based policy advice; site visits to interact with
school realities; contextualisation of policy advice.

8. OECD REVIEW - METHODOLOGY
 Analytical strand
• Identifying the key questions for analysis and the background information needed
from countries
• Reviewing the literature and evidence on the impact of evaluation and
assessment procedures
• Gathering data on countries’ policies and practices (Country Background Reports)
 Country Review strand
• Country Reviews provide specific advice to individual countries.
• OECD-led Review Team with external experts
• The scope and focus is determined by the country in consultation with the
Secretariat
 Synthesis report
• Comparative report to analyse policy options and highlight good practices across
countries.
 Collaboration with other OECD activities and international agencies
• INES (in particular NESLI), PISA, TALIS, CERI’s ILE , Longitudinal Information Systems and
Governing Complex Education Systems project, CELE’s on evaluating quality in
educational facilities, GOV’s work on Public Sector Evaluation.
• EC, Eurydice, the World Bank, Standing International Conference of Inspectorates(SICI),
UNESCO, BIAC, TUAC.

9. OECD REVIEW - PARTICIPATION
●Twenty four systems are preparing a Country Background
Report:
Australia, Austria, Belgium (Flemish Community), Belgium (French Community), Canada, Chile,
Czech Republic, Denmark, Finland, France, Hungary, Iceland, Ireland, Korea, Luxembourg, Mexico,
Netherlands, New Zealand, Norway, Poland, Portugal, Slovak Republic, Slovenia and Sweden.
●Twelve countries also opted for a Country Review:
Australia, Belgium (Flemish Community), Chile, Czech Republic, Denmark, Luxembourg, Mexico,
New Zealand, Norway, Portugal, Slovak Republic and Sweden.
●All OECD Member countries and Observers to the Education Policy
Committee are members of the Group of National Experts on
Evaluation and Assessment

10. REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC
●Objectives:
1. To provide insights and suggestions for policy development in the Czech
Republic;
2. To inform the wider international community about (a) the main features of the
Czech Republic’s evaluation and assessment policies; and (b) innovative and
effective approaches to evaluation and assessment in the Czech Republic; and
3. To inform the final synthesis report from the project.
●Structure of visit (29 March – 5 April, 2011):
• Review team with 4 members: 2 OECD Secretariat members; and researchers
from New Zealand and the United Kingdom;
• Interacted with about 200 individuals; had about 45 meetings amounting to
about 50 hours of discussions; visited 6 schools in Prague, Ostrava and Liberec
• Discussions were held with a wide range of national, regional and local
authorities; education officials; the Schools Inspectorate; parents’ organisations;
representatives of schools and school directors; teacher educators; and
researchers with an interest in evaluation and assessment issues.

11. REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC
●Relevant features:
1. External perspective: looking from a distance;
2. Independent perspective: no review team member has a vested interest in
the system;
3. Not an examination: qualitative analysis that seeks to provide an input of a
specific nature to the internal debate;
4. Bring together research evidence, data and information available in a broad
comprehensive comparative framework.
●Country Review report released on 30 January, 2012 and
available at: www.oecd.org/edu/evaluationpolicy

12. PART 2
Conclusions and policy
recommendations

13. Conclusions and policy recommendations
The evaluation and assessment
framework

14. The Evaluation and Assessment Framework
 Strengths
 There are common references at the national level to provide the basis for
evaluation and assessment
4-year Long-term policy objectives with indicators; Framework Education
Programmes
 There are good conditions for adapting learning to local needs
School Education Programmes: curriculum innovation; collaborative work
Regional 4-year Long-term policy objectives
 Responsibilities across the evaluation and assessment framework are well
articulated
MEYS, CSI, Regions and municipalities, schools

15. The Evaluation and Assessment Framework
 Strengths (continued)
 There is a range of initiatives to strengthen evaluation and assessment in
the school system
National standardised tests, common part of the school-leaving examination,
external school evaluation, school self-evaluation mandatory, national
indicators on education
 There is an “open door” climate among teachers
 There is a good principle of supporting policy work with specific expertise
Institutes with specialised expertise, data collections, OECD projects
 There are some reporting requirements:
Education database, inspection reports

16. The Evaluation and Assessment Framework
 Challenges
 The evaluation and assessment (E&A) framework needs to be completed
and made coherent
o There is no integrated E&A framework
o The E&A framework is incomplete (formative assessment of students,
moderation of marks, systematic teacher appraisal, school self-evaluation
incipient, no framework for school leadership appraisal)
o Some articulations within the E&A framework are not sufficiently developed
 There is little attention to equity and inclusion in the evaluation and
assessment framework
 The Framework Education Programmes are not perceived as specific
enough to guide teaching and assessment
Little shared understanding about what constitutes adequate, good and excellent
performance
 It is unclear that the students are at the centre of the evaluation and
assessment framework

17. The Evaluation and Assessment Framework
 Challenges (continued)
 There is a narrow understanding of the purposes of evaluation and
assessment
Perceived as instrument to hold stakeholders accountable, to “control” and assess
compliance with regulations
 There is a need to strengthen competencies for evaluation and assessment
across the system
 The articulation between levels of government and the support from the
centre are limited
o Concerns about the lack of systematic application of national directions,
inconsistency of practices and little capacity or commitment to developing
quality frameworks.
o Weak articulations between the different decision-making levels.
o Limited provision from the centre of tools and guidelines to assist evaluation and
assessment activities
 There are challenges in the implementation of some evaluation and
assessment initiatives

18. The Evaluation and Assessment Framework
 Policy recommendations
 Better articulate learning goals
o Establish clear goals for education and make equity and inclusion more prominent
o Clarify reference points and criteria for quality in evaluation and assessment
 Integrate the evaluation and assessment framework
Develop a strategic plan or framework document that conceptualises a
complete evaluation and assessment framework and articulates ways to
achieve the coherence between its different components
 Strengthen some of the components of the evaluation and assessment
framework
Teacher appraisal, appraisal of school leaders, formative student assessment, school
self-evaluation
 Further develop some articulations within the evaluation and assessment
framework

19. The Evaluation and Assessment Framework
 Policy recommendations (continued)
 Build on some key principles to effectively implement evaluation and
assessment
o Place the students at the centre of the evaluation and assessment framework
o Ensure a good emphasis on the improvement function of evaluation and
assessment
o Communicate the rationale for evaluation and assessment
o Recognise the importance of school leadership
o Establish an implementation strategy
 Develop evaluation and assessment capacity across the school system
 Improve the articulation between levels of government and assure
support from the centre

20. Conclusions and policy recommendations
Student assessment

21. Student Assessment
 Strengths
 Assessment is seen as part of the professional role of teachers
o Teachers play the major role in assessing and reporting
o Schools decide and publish assessment criteria
o Variety of approaches
 An external dimension to assessment was introduced
o National component of school-leaving examination
o Common assignments for the apprenticeship certificate
o Addresses the need for checks and balances to ensure reliability in the
application of standards
 There is an increased focus on student outcomes
o Including a move to identify expected minimum standards of achievement for
students at key points in their education

22. Student Assessment
 Challenges
 Approaches to learning and assessment remain markedly traditional
o Traditional approach to the organisation of classrooms
o Little emphasis in assessment practices on providing student feedback
 Assessment for learning is not systematically used in Czech schools
o Feedback often understood as ‘summative assessment done more often’
 Summative assessment of students raises some concerns
o Assessment is often norm-referenced
o Achievement mixed with effort and motivation
 There is limited consistency of student assessment across schools and
classes
 The national-level support for teacher-based student assessment is
limited
o Lack of guidance to assess against FEPs (e.g. in the form of exemplars),
insufficient assessment tools for teachers, limited professional development.

23. Student Assessment
 Challenges (continued)
 The national standardised tests entail a range of limitations and risks
o Development of the standards being rushed
o May be more appropriately regarded as specifications for the national tests,
rather than standards
o Potential negative consequences of high-stakes use: (i) ‘teaching to the test’;
(ii) cross-curricular competencies ignored; (iii) classroom time spent on
preparation of the test; (iv) schools selecting the students taking the test.
 The assessment of students leads to little interaction among teachers
o Low levels of teacher moderation of student assessments
 Multiple purposes to school-leaving examinations raise some concerns
o To provide a certificate of achievement to students
o To compare performance of schools: need other sources of information to
achieve this and value-added techniques

24. Student Assessment
 Policy recommendations
 Develop educational standards covering the breadth of student learning
objectives prior to developing national standardised tests
o There is a need for clear external reference points in terms of expected levels of
student performance at different levels of education
o National tests not to be the vehicle to develop standards
 Limit the undesired effects of national standardised tests
Reflect further on the purposes of the national tests; should be trialled first; minimise
negative impacts
 Develop a broad strategy for student assessment and strengthen the role of
formative assessment
Assessment for learning; provide tools supporting the assessment of students; broad
range of assessment approaches; formative ass.
 Build teachers’ capability for student assessment

25. Student Assessment
 Policy recommendations (continued)
 Develop a range of tools at the central level to support teacher-based student
assessment
e.g. marking rubrics; exemplars illustrating student performance, assessment tools
 Put in place moderation processes to ensure the consistency of student
summative assessment
 Student assessment should be criterion-based rather than norm-referenced
 Ensure student assessment is inclusive
 e.g. students with special needs, avoid biases by socio-economic background and
minority status (e.g. Roma students)
 Build capacity of markers of external tests and examinations

26. Conclusions and policy recommendations
Teacher appraisal

27. Teacher Appraisal
 Strengths
 The principle that teachers should be evaluated is widely accepted
o Teacher appraisal takes place in all schools and is an important and normal
part of school activities
 Teacher appraisal is focused on evaluating classroom teaching
 The importance of teacher professional development is recognised in the
legislation
 Some structures for co-operation and exchange among teachers are in
place
o E.g. Subject commissions bringing together all teachers teaching a particular
subject; lesson preparation; exchange of pedagogical approaches

28. Teacher Appraisal
 Strengths (continued)
 There are plans to develop teaching standards and a new career system
for teachers
 The link between teacher appraisal and pay increments has potential to
incentivise high performance
o Teachers may be awarded additional pay increments and bonuses that are
determined by the school leadership team
o However, there are important questions regarding the transparency of how
salary rewards are implemented

29. Teacher Appraisal
 Challenges
 There is currently no shared understanding of what constitutes high quality
teaching
o Lack of a national framework defining standards for the teaching profession
o There are no uniform performance criteria or reference frameworks against
which teachers could be appraised
 Teacher appraisal is not systematically implemented for all teachers
The quality and extent of teacher appraisal approaches in individual schools depend
on the capacity and leadership style of the school principals.
 There is little tradition for educational leadership in schools
Many school leaders have not been sufficiently prepared for their wide range of
tasks, in particular leading teaching and learning processes in the school.

30. Teacher Appraisal
 Challenges (continued)
 There are tensions between the accountability and improvement functions of
teacher appraisal
o Teacher appraisal traditionally been conceived as a summative and
accountability-oriented process
o There are risks involved in trying to achieve both the accountability function and
the improvement function of teacher appraisal in one single process
 The link between teacher appraisal and rewards is not transparent
 Links between teacher appraisal and professional development could be
enhanced
o The provision of prof. development appears fragmented and not systematically
linked to teacher appraisal
o There is scope to better link teacher professional development to school
development and improvement

31. Teacher Appraisal
 Policy recommendations
 Develop a professional profile or standards for the teaching profession
A clear and concise statement of what teachers are expected to know and be able
to do is a key element in any teacher appraisal system as it provides a credible
reference to make judgements about teacher competence
 Strengthen teacher appraisal for improvement purposes (developmental
appraisal)
o Strengthening regular school-based formative appraisal with a professional
development focus which is separate from the more summative appraisal
processes.
o To be validated externally by Schools Inspectorate
 Further enhance the role of educational leadership

32. Teacher Appraisal
 Policy recommendations (continued)
 Consider establishing a system of teacher certification to determine career
progression
 Access to each of the key stages of a career could be associated with formal
processes of summative appraisal that complement the regular formative
appraisal
 The different career stages should match the different levels of expertise reflected
in teaching standards
 Need a stronger component external to the school to validate the process and
ensure that practices are consistent across the country
 Three core instruments: classroom observation, self appraisal and documentation
of practices in a simplified portfolio.
 Ensure links between developmental appraisal and appraisal for certification
 Ensure appropriate articulation between teacher appraisal and school
evaluation

33. Conclusions and policy recommendations
School evaluation

34. School Evaluation
 Strengths
 External school evaluation is established
o Clear commitment to external accountability based around school evaluation
with a regular cycle of external school evaluations
o External school evaluation valued by stakeholders
 The external evaluation model embodies a number of features of best
practice
o Well structured and systematic
o Set of publicly-available criteria for external inspection is drawn up every year
o Publication of inspection reports
o A range of sources of information
o Thematic reports
 Schools facing greater challenges benefit from some follow-up
The CSI undertakes a follow-up inspection to assess whether improvements were
undertaken to address the challenges previously identified.

35. School Evaluation
 Strengths (continued)
 Classroom observation is part of school evaluation processes
Placing learning and teaching at the heart of the evaluation process sends clear
signals about what matters.
 There is a new emphasis on schools’ self-evaluation
Has the potential to encourage schools and principals to place a greater emphasis
on school improvement and development planning.
 School leadership is promoted in school evaluation
There is an explicit recognition that the processes of self-evaluation and external
evaluation are hugely dependent on a principal’s capacity to stimulate engagement,
to mobilise resources and to ensure appropriate training and support.

36. School Evaluation
 Challenges
 External school evaluation seems to have limited emphasis on school
improvement
o The accountability function tends to emphasise compliance with legislation
rather than the promotion of school improvement.
o Advice is only given to “weaker” schools which are identified as those that do not
meet the minimum standards as set by law.
o There is not enough guidance about what will lead to school improvement and
little attention is paid to identifying and disseminating best practice
 There are a number of limitations in external school evaluation
Difficult to take account of the socio-economic context of the school; not enough
emphasis on pedagogical aspects, Criteria used in the CSI external evaluations are
not stable enough.
 There is little emphasis on student results/progress
Limited ability to assess quality of learning and student progress.

37. School Evaluation
 Challenges (continued)
 School self-evaluation needs to be strengthened
Its penetration across the school system remains at an early stage of development
 The use of data for school development is limited
 The evaluation by organising bodies has a limited scope and impact
 There is no full recognition of the role of school leaders and their appraisal is
limited
o While the school principal has a key role in the system and considerable
responsibilities, this has not as yet been translated into a dedicated career
structure
o Limited preparation for the role of school principal
o Little recognition and financial reward
o The evaluation of school principals, conducted by organising bodies, is largely
absent except in terms of the financial aspects of budget management.

38. School Evaluation
 Policy recommendations
 Strengthen external school evaluation
o Strengthen focus on school improvement and move away from the current
“compliance” driven model.
o Provide advice for improvement to all schools evaluated.
o The school evaluation framework, the criteria and questions governing
judgements and the methods employed should all focus much more directly on
the quality of learning and teaching and their relationship to student outcomes.
 Improve the alignment between external and self-evaluation and raise the
profile of self-evaluation
Criteria used in both processes sufficiently similar as to create a common language
 Give stronger emphasis to the follow-up to external evaluation
 Improve the capacity of schools to engage in school evaluation

39. School Evaluation
 Policy recommendations (continued)
 Improve the instructional leadership skills of school principals
 Plan to use data on student results effectively
Any publication of results of school performance in students’ school-leaving
examinations and/or national tests should be presented in ways that take account
of intake differences including, for example, the socio-economic background of
students.
 Strengthen the evaluation of school principals
 Improve the scope and role of organising bodies in school quality
improvement
o The regional and municipal authorities should strengthen their role in supporting
school improvement.
o Collaboration and networking amongst schools could be encouraged to help
develop and spread good practice and enhance teachers’ professional skills.

40. Conclusions and policy recommendations
Education system evaluation

41. Education System Evaluation
 Strengths
 An Education Indicators Framework is established
It involves well-established procedures for data collection in close articulation with
schools.
 There is a concern to assess the progress of the education system towards
pre established objectives
Principle of establishing educational objectives and the subsequent monitoring of
the progress towards achieving them
 The qualitative analysis of thematic reports provides valuable information for
system monitoring
Richness of contextualised qualitative analysis
 The participation in international surveys is instrumental for system
evaluation

42. Education System Evaluation
 Challenges
 There is little emphasis on the evaluation of the education system
o Limited policy attention thus far and there is no comprehensive strategic
approach to it
o The current narrow approach to system evaluation does not allow a broad
enough assessment of the extent to which student learning objectives are being
achieved
 The absence of student performance data is a major gap in system
monitoring
Presently there is no mechanism for the Czech Republic to monitor at a national
level the achievement of its students against learning objectives specified in the
Framework Education Programmes
 There are key information gaps at the system level
There are no measures on students’ socio-economic background; There are gaps in the
data collected from schools; Little emphasis on investigating inequities in the system;
Limited information on the teaching and learning environment

43. Education System Evaluation
 Challenges (continued)
 It is not possible to monitor student outcomes over time and across schools
o Mostly due to the absence of national data on student performance
o School-level results of national tests might be disclosed with no account for
schools’ particular contexts. This can considerably distort considerations about
the effectiveness of each school
 Monitoring at the region and municipality levels is faced with considerable
challenges
o Regions have a limited intervention in quality assurance with their main tool
being the evaluation of school principals
o No overview of the different quality assurance systems in the regions, including
strategies for school improvement
o Limited articulation between Regions and municipalities
 System-level information is not fully exploited
Little analysis to inform educational planning; Limited use to inform school
management; No comprehensive information system; Systematic sharing of data
between schools is limited

44. Education System Evaluation
 Policy recommendations
 Raise the profile of system evaluation within the evaluation and assessment
framework
The challenge for system-level evaluation is to ensure that the measures of system
performance are broad enough to capture the whole range of student learning
objectives – policy making at the system level needs to be informed by high
quality data and evidence, but not driven by the availability of such information.
 Develop national student performance data for system monitoring
o Design national standardised tests for national monitoring and as a pedagogical
tool [closely aligned with student learning objectives; can measure only a limited
range of student learning objectives; publish test results at the school level is
premature]
o Develop strategies to monitor a wider range of curricular areas and broader
outcomes [develop sample-based surveys]
o Ensure that national monitoring covers broader outcomes [e.g. higher-order
thinking skills and cross-curricular competencies]

45. Education System Evaluation
 Policy recommendations (continued)
 Prioritise efforts to meet information needs for national monitoring
o Develop measures of the socio-economic background of students
o The data collection from schools needs to be improved
o Give more prominence to the analysis of inequities in the system
o Improve the information on the teaching and learning environment
 Explore ways to more reliably track educational outcomes over time and across
schools
Make meaningful comparisons across schools if test results are published at the
school level
 Strengthen the role of regions and municipalities in quality improvement
 Optimise the reporting and use of system-level data
o Strengthen the analysis for educational planning and policy development
o Improve feedback for local monitoring
o Integrate available data and facilitate access by key agencies
o Facilitate the sharing of student information across schools

46. Thank you
(paulo.santiago@oecd.org)
(www.oecd.org/edu/evaluationpolicy)