PISA-D Annual Report and Progress

3rd meeting of the International Advisory Group
Granados Park Hotel – Asunción, Paraguay
30 March – 1 April 2016
PISA for Development
Expected results from the meeting

• Election of two co-chairs: one representative of the participating
countries and one representative of the development partners
• Presentationand review of annual report on project activities, as
well as the contribution of OECD and the project to education in
the 2030 sustainable development agenda
• Presentationsby ETS and The Learning Bar on the progress being
made on all strands of the project and the plan for next year
followed by discussion with the IAG
• Presentationsfrom participating countries on capacity-building
activities, including feedback on the international/NPM meetings,
peer-to-peer learning and other arrangements

continued…
• Presentationfrom Kosovo and GIZ of their experiences of
launching PISA for the first time in the country; agenda setting,
engagement and communication with external stakeholders
prior to the tests;embedding PISA in a broader discussion of
the value and standards of assessment in the Kosovo education
contextin general; and preparing to use the results in a
National PISA Report
• Presentationby the OECD on an up-date of the PISA-D
engagement and communication strategy
• Presentationsfrom each country on their respective
engagement and communication strategies since project launch

continued…
• Presentationsby OECD and Contractors of analysis and reporting
strategy, including dissemination tools and activities for the
main study, the analysis and assessment frameworks and the
technical report
• Presentationby UIS on the statusof system level data in the
participating countries and discussion of this, followed by
commentsfrom each country
• Presentationby OREALC/UNESCOSantiago on lessons from
LLECE related to engagement and communicationwith
stakeholders and dissemination of results, particularly in respect
of TERCE, followed by discussion

continued…
• Presentationby OECD of the final version of the Terms of
Reference for an independent review of the project; the IAG will
then discuss and agree upon the timing of the review and the
call for tender arrangements
• Presentationby OECD on PISA development strategy, including
the planned scaling up of PISA-D, reporting on preparations for
PISA 2021 and exploring opportunities for expanding PISA to a
wider set of countries, particularly in Latin America
• Conclusions of the meeting, next steps to be taken, agreement
of the draft of the summary record and plans for 4th IAG
Meeting

Day One sessions
1. Introductions, Purpose of Meeting, Election of Co-
Chairs, Project Progress Report, establishment of the
TAG and the contribution of the OECD and the
project to education in the post-2015 agenda
2. Presentation by the International Contractors of
detailed progress report for Strands A, B and C,
including the draft frameworks, draft instruments,
sampling design and detailed plans for 2016-17
3. Participating countries report on capacity-building
activities and outcomes

Day Two sessions
4. Presentation by the PISA nationalteam from
Kosovo and GIZ: engaging with stakeholders
and preparing these for PISA, national reports
and the disseminationof PISA results
5. PISA for Development engagement and
communicationstrategy – participating
countries to report on stakeholder engagement
and communicationactivities
6. Presentation by OECD and contractors of
analysis and reporting plan

Day Three sessions
7. Presentation by UIS on the status of system level data
in the participating countries and discussion of this
8. Lessons from LLECE – TERCE engagement and
communication strategy
9. Agreement of the ToR for an independent review of
the project to be conducted in 2017/2018
10. PISA development strategy and outreach to potential
PISA countries in Latin America
11. Next steps and meeting conclusion

Key documents in folder
• Agenda
• Participants list
• Annual progress report & summaryrecord of last meeting
• Draft cognitive assessmentframeworks (reading, maths, science)
• Draft framework for contextual questionnaires
• Descriptions of items and tasks for all 3 domains in the cognitive assessments
• Student questionnaire; teacher questionnaire; school questionnaire
• Translation and adaptation guidelines and procedures
• Sampling design and plans for 2016
• PISA-D engagementand communicationstrategy
• Draft analysis and reporting plan
• Report on the status of system level data in the participating countries
• Terms of reference for an independent review of the project
• PISA development strategy

Other documents in the room
• Capacity Building Plans from the participating countries
• Presentation on the Kosovo experience
• Country and LLECE engagement and communications
presentations

IAG Agenda
• The International Advisory
Group is invited to adopt the
agenda for its third meeting

PISA-D Annual Report on project activities

Summary of achievements since last
IAG• The project plan for 2015 has been implemented in accordance
with the 13 next steps agreed upon at the 2nd IAG meeting. There
have been no major deviations.
• The project is on track: major technical milestones have been
achieved – frameworks, tests,questionnaires, manual, standards,
sampling plan, adaptation and translation guidelines are all in
place and the field trial is in sight.
• Plans for Strand C have been agreed and way ahead is now clear
for this component.
• Capacity building activities are under way and peer-to-peer
learning partnerships are in place and working.

Reasons for success
• The commitment and performance of participating countries and their
national teams has been outstanding and momentum is building,
expectations are high.
• The international contractors have done a great job and delivered on
their terms of reference in exactly the right way.
• Development partners have delivered their support (international and
in-country) and technical guidance on time and are helping us to move
forward.
• Technical partners, especially UNESCO and UNICEF, have provided
guidance and co-operation and facilitated the project’s contribution to
wider processes; i.e., Education SDG and 2030 agenda.
• PISA countries and economies (Belgium, Brazil, Canada, Chile, China,
Hong Kong, Korea, Iceland, Mexico, Peru, Spain, Uruguay, United States)
have been generous and valuable contributors to peer-to-peer learning

Annual report format
From 2015 onwards, a progress report:
• actual outputs compared to planned outputs
• summary of the use of funds compared to budget,
• explanation of major deviations from plans,
• assessment of problems and risks,
• assessment of the need for adjustments to activity plans
and/or inputs and outputs, including actions for risk
mitigation,
• assessment of achievements in relation to project purpose
15

13 next steps agreed upon at 2nd IAG
1. OECD to complete the remaining planned expert papers and to make all
documentation available on its website
2. OECD to finalise ToR and tendering documents for international contractor(s)
for Strand C
3. OECD to sign participation agreements with all the remaining countries
participating in the project – Paraguay and Cambodia
4. Participating countries to finalise Project Implementation Plans
5. Participating countries to finalise outstanding agreements with development
partners regarding contributions and support (e.g., international costs, in-
country costs and activities)
6. OECD to complete the design of capacity-building plans for all participating
countries

13 next steps agreed upon at 2nd IAG
7. OECD to finalise with development partners outstanding agreements for support
to the project – general contributions and country-specific contributions
8. First meeting of Technical Advisory Group in August (webinar)
9. Complete tendering process to commissioninternational contractor (s) for Strand
C by OECD
10. Project implementation: Strands A and B -- Technical development
– First international/NPM meeting (September 2015)
– Second international/NPM meeting (January 2016)
11. OECD Secretariat with International Contractors to facilitate translation of key
materials, such as manuals, into French and Spanish
12. Participating countries to link the communication focal points in each National
Centre with the OECD Secretariat to facilitate the implementation of the
engagementand communication strategy
13. Third meeting of IAG in March 2016 in Asunción, Paraguay.

i) Expert papers (continued)
Review of system-level data in the participating
countries
• Written by UNESCO Institute of Statistics (UIS)
• Final draft circulated to the IAG at the end of
February 2016 for written feedback
• Final draft will be presented at 3rd IAG meeting,
and countries will provide oral feedback based on
their review of the report
• OECD will implement all feedback after the 3rd IAG
meeting and publish the report mid-2016

i) Expert papers (continued)
The ten indicators that have the most impact on
learning in middle-income countries
• Written by Lucia Tramonte
• Explores in more detail than was possible in the
Lockheed report on the PISA results of the middle-
income countries to identify indicators with the
most impact on learning in those contexts
• The 1st draft of the paper has been received by the
OECD and is under review

ii) Finalise ToR and tendering documents for
Strand C
• OECD finalised ToR and tendering
documents for Strand C immediately
following the last IAG meeting
• Call for tender for Strand C launched in
April 2015
• Review panel took place in June 2015

iii) OECD to sign participation agreements
with remaining countries
• OECD signed participation agreements with:
Paraguay in September 2015
Honduras in January 2016
Cambodia in February 2016
• OECD is in the process of signing a participation
agreement with Panama for Strand C
• OECD is exploring engagement with Ukraine,
supported by Poland, as part of scaling up PISA-D

iii) OECD to sign participation
agreements with remaining countries
(cont’d)PISA-D has been launched in
Ecuador, Guatemala, Honduras,
Paraguay, Senegal & Zambia
Launches planned in: Panama
(22 April) & Cambodia (30 May)

iv) Countries to finalise PIPs
• Project Implementation Plans (PIPs) have been
completed for all participating countries with the
exception of Honduras and Panama
• Honduras’ and Panama ’s PIPs will be completed by
August 2016, following completion of their capacity
needs analysis and capacity building plans
• Discussions have started for Ukraine to elaborate a
PIP for PISA 2018 building on PISA-D PIPs

v) Countries to finalise outstanding
agreements with development partners
• All 8 countries have finalised or are in the process of
finalising their respective budgets for financing their
participation in PISA-D
‒ Countries’ contributions to the international costs
of participation (budget managed by the OECD)
‒ In-country costs of participation (budget managed
by the countries themselves and, in some cases,
their development partners), which is described in
the PIPs

vi) OECD to complete the design of CBPs for
all countries
• CBPs have been developed for Cambodia, Ecuador,
Guatemala, Paraguay, Senegal and Zambia –
available on the OECD’s PISA-D website
• Honduras’ and Panama’s CBP will be completed by
end of May 2016
• Ukraine will also develop a CBP for PISA 2018
building on the model and tools developed by PISA-D

vii) OECD to finalise with development partners
outstanding agreements for support to the project
• All outstanding agreements for support to the
project – general contributionsand country-
specific contributions– have been finalised
• Fund-raising is an on-going process
• Since the 2014-15 progress report, new
development partners include:
‒ Sunny Varkey Foundation
‒ Dubai Cares and Microsoft Corporation

viii) First meeting of the TAG in August
• 1st meeting of the PISA-D Technical Advisory
Group (TAG) took place on 10 September 2015
Webinar focused on inducting the TAG into the project
and a discussion of the initial plans for Strands A and B
• 2nd meeting took place on 19 November 2015
Webinar focused on initial plans for Strand C, PISA
Technical Standards and their application in PISA-D,
design and sampling for the teacher questionnaire and
data administration procedures

ix) Complete tendering process to commission
international contractor(s) for Strand C
• Review panel in June 2015 selected Educational
Testing Service (ETS) as the preferred bidder
• OECD negotiated with ETS on the basis of the
following agreed alterations to the design of
Strand C:
‒ Drop science from the out-of-school assessment
‒ Extend the timeline for Strand C to July 2019.
• Contract between OECD and ETS for Strand C
agreed in March 2016, to be signed…

x) Project implementation
• ETS, TLB, Westat, Pearson and cApStAn with the
Secretariat and countries successfully
completed the planned technical development
work in the months following the last IAG
• Subject Matter Expert Groups (SMEGs) meeting
took place in Mexico from 20-24 July 2015 –
consultationwith national experts followed

x) Project implementation (continued)
• PISA-D Questionnaire Expert Group (QEG)
meeting took place in South Africa from
20-24 August 2015 – consultation with
national experts followed
• 1st international/NPM meeting from 28
September to 2 October 2015 in Quito,
Ecuador – frameworks and items

x) Project implementation (continued)
• 2nd international/NPM meeting was held in
January 2016 at the offices of Westat in
Rockville, US
Concentrated on preparing the participating countries
for the tasks of adapting, translating, verifying and
printing the instruments and managing the survey
operations and initial plans for Strand C

xi) Translation of key materials into French
and Spanish
• PISA-D manual and the PISA Technical
Standards have been translated into
Spanish and French translation is
under way
• PISA-D framework documents are
being translated into French and will
also be translated into Spanish

xii) Countries to link communication focal
points in National Centre with OECD
• Following the last IAG meeting, each
participating country linked its
communication focal point in its
National Centre with the OECD
• Together, they generated web pages
for each country on the OECD’s
PISA-D website

xii) Countries to link communicationfocal points in
National Centre with OECD (continued)
• Countries are in the process of
implementing their national
communication strategies in
alignment with the PISA-D project’s
overall engagement and
communication strategy

xiii) 3rd meeting of IAG in March 2016 in
Asunción, Paraguay

PISA for Development use of funds
Expenditure Analysis (EUR)
Expenditures Commitments
Total 6
years2014
Y1
2015
Y2
2016
Y3
2017
Y4
2018
Y5
2019
Y6
A1. InternationalAdvisory
Group
162 406 62 473 60 000 31 000 30 000 30 000 375 879
A2. Instrumentdevelopment,
field trials, local assessment
implementation and related
services
56 638 810 460 2 200 000 2 000 000 880 000 389 000 6 336 098
A3. Technical oversight, co-
ordination, analysis and
reporting
344 454 391 106 440 000 440 000 440 000 215 000 2 270 560
A4. Engagement for peer-to-
peer learning and contribution
to UN-led post-2015 process
17 070 91 236 100 000 40 000 248 306
A5. Tablets* 300 000
Overhead 147 715 88 236 75 000 25 000 335 951
Total 728 283 1 443 511 3 175 000 2 536 000 1 350 000 604 000 9 836 794
* There is a possibility that the tablets will be donated, in which casethis costwould not be factored into the
international costs of the project.

Challenges, risks and assumptions
1. The institutional, technical and administrative capacity
and conditions that are needed to successfully
implement PISA at country and sub-national levels –
risks low
2. The technical challenges are many and various and
relate to the central question of how far PISA for
Development can go in enhancing and adapting PISA to
be more relevant to developing countries while still
ensuring that the results of the assessment contribute
to the establishment of an international benchmark in
the context of PISA – risks are manageable

Challenges, risks and assumptions
(cont’d)
3. PISA is changing rapidly and PISA-D needs to ensure that
it provides an effective bridge for participating countries
into main PISA – risks low
4. In all countries, PISA-D is already helping to strengthen
institutions devoted to the production of reliable data
and evidence for informing policy design. As the project
moves forward, it would be essential to maintain the
teams, sustain the support for the project,
institutionalise a culture of evaluation and draw support
for PISA in the country across the entire political
spectrum – risks are manageable

Adjustments: Project ImplementationSchedule to 2019

Annual Report on Progress
The International Advisory Group is
invited to:
• Note the progress that has been made
since March 2015
• Discuss any issues arising from the
progress report
• Approve the annual report on progress

PISA
for Development
Progress Report
PISA for DevelopmentIAG Meeting
30 March 2016
Asunción, Paraguay
Ann Kennedy, ETS

Overview of Progress Update
 Timeline and Key Phases
 Key Activities and Milestones (March 2015-March
2016)
 Integrated Assessment Design
 Frameworks and Assessment Instruments
 Translation and Verification
 Target Population and Sampling
 Key Activities and Milestones through the next IAG
meeting in 2017
44

Timeline Summary
45
2015 2016 2017 2018 2019
Develop assessment
and questionnaire
frameworks, test
design and master
instruments
Develop survey
materials, including
national assessment
instruments
Carry out FT
activities,
including coding
and data entry
Analyze FT data
and prepare for
MS
Carry out MS
activities,
including coding
and data entry
Analyze MS
data
Report planning
and
dissemination
Phase 1
Phase 2
Phase 3
Phase 4

Key Activities and Milestones
March 2015 – March 2016
46

Update: Cognitive Instruments
 Frameworks
• Expert group revision of descriptions and sample items (July 2015)
• Expert group webinar (October 2015)
• Country reviews in September (NPM1) and November 2015
 Cognitive Item Review and Selection
• Reviews of existing item pools
• Selection of units for the Field Trial (NPM1)
• Country reviews (November 2015)
• Final selection and assignmentof units to clusters
• Preparation of units for translation and adaptation
47

Update: Questionnaire Instruments
 Frameworks
• Expert Group Meeting (August 2015)
• Definitions of Constructs
• Presentationat the 1st NPM Meeting
• Expert Group Consultation (October 2015)
 Questionnaire Item Development
• Selection from Existing Measures
• Development of New Items
• Preparation of Notes for Adaptations
48

Update: Sampling and
Translation/Adaptation
 Sampling
• Country Questionnaires
• Informationaboutthe EducationalSystem and Eligible Students for PISA-D
• Sampling Tasks
• Test Languages
• Target Population,School SamplingInformation(StrandsA-B)
 Translation
• Translation Plan
• Translation and Adaptation Guidelines
• Release of Strands A-B Cognitive Units and Contextual
Questionnaires for Translation/Adaptation
49

Update: SharePoint Site
 Announcements, Calendar, Tasks
 Documents Libraries (General Access)
• General Documents
• Field Trial Resources and Instruments
• Main Survey Resources and Instruments
 Country sub-sites
• Share materials with international contractors in
secure site with country-specific permissions
• Highlight specific tasks with deadlines
50

Update: Key Documents
 PISA-D NPM Manual Part 1: First Review and Field
Trial Preparations Phase
• Communicationswith Contractors
• Overview of SharePoint Site
• Overview of Tasks and Key Dates for Phase 1
 Technical Standards
• PISA 2018 Technical Standards (Strands A-B)
• Strand C Technical Standards (ForthcomingJune 2016)
51

Goal of PISA-D
 Large scale assessmentssuch as PISA and PISA-D are
designed to inform governments and other key
stakeholders on a variety of issues
 To meet this goal, each assessment is developed around
three key design criteria
• Relevance
• Comparability
• Interpretability
53

Overview
Strands A-B
• 15-year-olds in school
• School-based
• Group data collection
• Self-administered tests and questionnaire
• Paper-based
• Reading, Mathematical and Scientific Literacy
• Questionnaires: Students,Schools and Teachers
54

Student
Data Collection
Strands A-B
(Group Administration)
55
Cognitive Assessment
(Reading, Mathematics and
Science)
Student Questionnaire
School
Questionnaire
Teacher
Questionnaire

Instrumentation
Strands A-B
• Tests:
– Paper booklets (rotated cluster design)
– Common booklet parts: general instructions and mathematical
formulae
– Clusters of Units/items translated through Word files
• Questionnaires
– Paper booklets (Student Q with multiple forms in FT)
– Adapted and translated through Word files
56

Frameworks and
Assessment Instruments
57

General Characteristics
of a Framework
 Frameworks should:
• Reflect the current thinking of experts in the field along with
the views of the participating countries
• Consider the target population
• Define the constructand discuss how results should be
reported
• Guide test development:
• Identify and define key task characteristics
• Recommend the relative importance of these
characteristics in the final set of items
• Provide a scheme for mapping these items to the
construct
• Guide interpretation of results

A Process for
Developing a Framework
 Identify and meet with key experts
 Develop a sequence of steps:
• Reach consensuson the survey goals and constraints
• Create a working definition that will explain the construct,its
relevance, and how it will be measured.
• Identify and define key characteristics for test development
• Recognize that each task provides a piece of evidence about
the knowledge and skills mapping to the construct
• Think about and identify key variables associated with each
task that help to explain placement along each scale
• Create or revise a described scale
59

Frameworks and Item Pools
60

PISA-D Cognitive Frameworks
 PISA Reading, Mathematical, and Scientific
Literacy Frameworks as the basis
• Attention to consistency of language across domains
• Better inform participating countries about
performance at lower levels while maintaining full
range of the PISA scale
• Extension of descriptions of proficiency Level 1
• Clarify basic processes of each domain
• Addition of sample items from PISA released item pool
• Consideration of Strand C population
61

PISA-D Expert Groups
 Experts from PISA and PISA-D
• Framework revisions
• Item reviews
 Composition per domain:
• Chair from ‘main’ PISA 2015/2018
• Content lead from Pearson
• Core expert representative from participating countries
• Extended experts from all participating countries
62

PISA-D Item Selection
Item selection began with understanding the
goals and constraints of the assessment:
• Provide a link to PISA (using data from PISA 2015
paper-based trend items)
• Enhance the lower end of the proficiency scale for
each domain while keeping measurement of the
higher levels
• Maintain adequate construct coverage
• Each student will respond to 2 hours of cognitive
items
• No new item development
63

Assembling the Available Pool of Items
PISA 2015 Trend (items from 2000-2012)
New PISA 2015 (Science only)
PISA for Schools
International Adult Literacy Surveys:
• PIAAC (literacy and numeracy)
• STEP (literacy)
• LAMP (literacy and numeracy)
64

Reviewing and Evaluating the Item Pool
Test developers reviewed items based on:
• Data Quality
• Psychometricquality of items
• Scoring reliability
• Expert reviews
• Construct representation
• Content appropriateness
• Factors that affect difficulty
65

Selecting Items
• Approximately 75 items per domain (5 clusters
of 15 items each) for the FT
• Include 50-60% of items from PISA 2015 trend
item pool; keep intact units where possible
• Greater distribution of items at or below Level
2
• Follow guidance of frameworks for distribution
of items across aspects
66

Goals for Item Distribution and Totals
from Existing Item Pools (Strand A)
Level 1 Level 2 Level 3 Level 4
Levels
5-6
PISA
2015
Trend
13 13 7 7 5 Total=
45
Other
Item
Pools
9 9 5 5 3 Total=
31
Total 22 22 12 12 8 Total=
76
67
Approximately
60% of the total
number of items

• Reading and Mathematical Literacy
• Approximately 45 items per domain (3 clusters
of 15 items each) for the FT
• Subset of Strand A selection
• Distribution of items at or below Level 3
• Adaptability for tablet-based administration
(including computer scoring)
68
Item Distribution and Totals
for Strand C

Reading Framework
and Item Pool
69

Reading Literacy: Framework
Modifications for PISA-D
 Framework based on the PISA 2012 Reading Literacy
Framework
 Incorporated Reading Components (PISA 2012
optional, PIAAC)
 Extended the description of Integrate and Interpret
process to include comprehension of the literal
meaning of text
 Added category (1C) to the levels of proficiency to
indicate expectations of students performing at the
lowest level of the proficiency scale
70

PISA 2009/2015 Processes
with PISA-D reading components
71
Comprehend
literal
information
Access and
retrieve
Integrate and
interpret
Reflectand
evaluate
Reading literacy
Use content primarily
within the text
Draw primarily upon outside
knowledge
Retrieve
information
Forma broad
understanding
Developan
interpretation
Reflect onand
evaluate content of
text
Reflect onand
evaluate formof
text

Reading Items by Process
72
Process
PISA-D FT
StrandA
PISA 2015
Trend
%
Framework
Goal
Number %
Access and
retrieve
25 31% 29% 25-30%
Integrateand
interpret
34 43% 49% 45-55%
Reflect and
evaluate
21 26% 23% 15-25%
Total 80 100% 100% 100%

Reading Items by Situation
73
Situation
PISA-D FT
Strand A
PISA 2015
Trend
%
Framework
Goal
Number %
Personal 25 31% 27% 36%
Educational 25 31% 28% 33%
Occupational 12 15% 23% 20%
Public 18 23% 22% 11%
Total 80 100% 100% 100%

Mathematics Framework
and Item Pool
74

Mathematical Literacy: Framework
 Added categories (1B and 1C) to indicate expectations
of students performing at the lowest level of the
proficiency scale
 Expanded descriptions of the following mathematical
processes:
• Formulate Situations Mathematically
• Employ Mathematical Concepts, Facts, Procedures
• Interpret, Apply, and Evaluate Mathematical Outcomes
 Added sample tasks to illustrate descriptions
75

Mathematics Items by Process
76
Process
PISA-D FT
StrandA
PISA 2015
Trend
%
Framework
Goal
Number %
Formulating
situations
mathematically
19 25% 29% Approx. 25%
Employing
mathematical
concepts, facts,
procedures
30 39% 43% Approx. 50%
Interpreting,
applying, and
evaluating
mathematical
outcomes
27 36% 28% Approx. 25%
Total 76 100% 100% 100%

Mathematics Items by Content
77
Content
PISA-D FT
StrandA
PISA 2015
Trend
%
Framework
Goal
Number %
Change and
relationships
13 17% 27% 25%
Space and shape 16 21% 23% 25%
Quantity 28 37% 25% 25%
Uncertainty and
data
19 25% 25% 25%
Total 76 100% 100% 100%

Science Framework
and Item Pool
78

Scientific Literacy: Framework
 Added category (1C) to indicate expectations of
students performing at the lowest level of the
proficiency scale
 Expanded descriptions of the following scientific
competencies:
• Explain Phenomena Scientifically
• Evaluate and Design Enquiry
• Interpreting Data Scientifically
 Added sample tasks to illustrate descriptions
79

Science Items by Competency
80
Scientific Competency
PISA-D FT
Strand A
PISA 2015
Trend
%
Framework
Goals
Number %
Explaining phenomena
scientifically
41 54% 46% 40-50%
Evaluate anddesign
scientific enquiry
12 16% 19% 20-30%
Interpreting dataand
evidence scientifically
23 30% 34% 30-40%
Total 76 100% 100% 100%

Science Items
by Types of Scientific Knowledge
81
Knowledge
Types
PISA-D FT
PISA 2015
Trend
%
Framework
Goals
Number %
Content 50 66% 62% 55-65%
Procedural 19 25% 27% 20-30%
Epistemic 7 9% 12% 10-20%
Total 76 100% 100% 100%

Translation and Adaptations
of Assessment Instruments
82

 In general, a translation should be fit for purpose
 In the assessment context: each item should
examine the same skills and invoke the same
cognitive processes as the original version, while
being culturally appropriate within the target
country—this is highly important for high-quality,
comparable data
 ‘Localization’ consists of preparation, translation,
adaptation, and validation of the assessment
instruments
 Training and documentation provided in January
2016 NPM meeting
83
Overview of Translation and
Adaptation for PISA-D

 Countries received the following materials:
• Materials Preparation documents for the NPM and
Translators
• Presentationsfor Translator Training Workshop
• Translation and Adaptation Guidelines (Cognitive Items
and Contextual Questionnaires)
• Practice Materials
85
Translation and Adaptation Kit

 Countries negotiated a Translation Plan with cApStAn
based on FT data collection schedule
 Most countries will submit translations and adaptations
of cognitive materials to cApStAn by April
 Adaptations of Background Questionnaires must first be
negotiated and approved by TLB prior to translations are
submitted to cApStAn for verification
 Verification of all instruments is scheduled to be
complete by June 2016
 Strand C Translation/Adaptation begins May 2016
86
Translation/Adaptation and
Verification Timeline

Sampling OverviewStrands A-B
• Sampling Tasks focus on
1) School sample
2) Within-school sample
• Sample Design presented in January 2016 NPM meeting
– Definition of Target Population
– PISA Technical Standards
– Sample Sizes
• Student Sampling Training will take place during April 2016 NPM Meeting
88

Key Activities and Milestones
Through the Next IAG Meeting
89

Upcoming Activities and Milestones:
Phase 1 (Preparation for the Field Trial)
March – September/ December 2016
 Preparation and Adaptation of School-Level Materials
 Finalize FT instruments
 Select FT samples and contact schools
 DataManagement training (NPM 4/4a)
 Coder training (NPM 4/4a)
 Test Administrator training (NPM 4/4a)
 Preparation of National Codebooks for Data Entry
 National FT Training
90

Upcoming Activities and Milestones:
Phase 2 (FT Data Collection)
September / December 2016 – March 2017
 FT Data Collection
 Data Entry
 Coding
91

NPM Meetings
92
Meeting Description Date
4 Training: Coding, Data management, Test Administration July 2016
5 Analysis and interpretation of FT results, plans for analysis of
contextual questionnaires, and preparation for MS data
collection
May 2017
6 MS survey operations, MS student sampling, test
administrator training, scoring, and data management
July 2017
7 Scaling methodology, and data analysis and reporting tools Mar. 2018
8 Analysis and interpretation of MS results and preparations for
reporting and dissemination
July 2018

Objectives
 To yield a large enough sample size to test the
validity of the questionnaire and the assessment
items
 To explore various approaches and evaluate
various options to arrive at a recommendation for
identifying and assessing a nationally
representative sample of out-of-school 15-year-
olds in future cycles of PISA-D Strand C
95

Assessment Overview
Strands A-B
• 15-year-olds in school
• School-based
• Group data collection
• Self-administered tests and
questionnaire
• Paper-based
• Reading, Mathematical and
Scientific Literacy
• Questionnaires: Students,
Schools and Teachers
Strand C
• 14 to 16-year-olds
• Household-based
• Individual data collection
• Questionnaire interview and
self-administered tests
• Computer-based*
• Reading and Mathematical
Literacy
• Questionnaires: Respondents
and Parents
*Paper-based Parent Questionnaire
96

Strand C Target Population
 15-year-olds in-school at grades 6 and lower
 Out-of-school 15-year-olds
 Definition
• “15-year-olds” means 14 to 16 year olds in Strand C
97

Two Selection Approaches for
Strand C (Pilot Project)
 Primary component -- Probability sampling
(required)
 Secondary component – Non-probability sampling
(must do for field test)
98

 Primary component -- Probability sampling
(required)
• To be conductedby all countries
• Generalizable results to target population and
estimation of precision
• At each stage, all sample units have a non-zero
probability of selection
• Gives all individuals in the target population a chance
of selection
• All cases fully worked to standard number of attempts
99

 Secondary component -- Non-probability sampling
(must do for field test)
• Set of respondents represent themselves
• Link-tracing network sampling
• Snowballing
• Respondent driven sampling
100

 Importance of primary component -- Probability
sampling
• Critical due to producing results that represent the
country
• While non-probability sample represents only itself
101

Sampling Strategy
 Strand A
• Administered in schools in group setting
• 1st stage unit = School
• 2nd stage unit = Student
102

Sampling Strategy
 Strand A
 Strand C
• Administered in dwellings in individual setting
• Typically, a probability-based area sample is selected
• 1st stage unit = Geographic area
• 2nd stage unit = Dwelling unit (DUs) and Group quarters (GQs)
– Create list of DUs/GQs from listing operation
• 3rd stage unit = Person
– Create list of persons using a screener questionnaire to arrive at a
list of eligible persons within the DU/GQ
103

Sampling Strategy
 Strand A
 Strand C
• Administered in dwellings in individual setting
• Typically, a probability-based area sample is selected
• 1st stage unit = Geographic area
• 2nd stage unit = Dwelling unit (DUs) and Group quarters (GQs)
– Create list of DUs/GQs from listing operation
• 3rd stage unit = Person
– Create list of persons using a screener questionnaire to arrive at a
list of eligible persons within the DU/GQ
• Next step, the non-probability sample
– Link-tracing
104

Two Major Strata
 Stratum H: Areas with high concentration of target
population
 Stratum L: Areas with low concentration of target
population
105

Sampling Strategy Overview
106

Sampling Strategy
 Primary -- Probability-based area sample
• Most efficient designs give higher probability of selection to
areas with the largest target population size
• Clustering reduces cost at expense of increasing
variances associated with estimates
• Discuss with countries to tailor an area sample design
• 1st stage unit = Census Enumeration Area (EA)
• Discuss with countries their typical approach to
probability sampling
• Determine geographic spread of target population
• Identify areas with large concentrations of out-of-school
15-year-olds
• Assign higher sampling rate in Stratum H
107

Sampling Strategy
 Secondary -- non-probability sample
• Goals of link-tracing as applied to Strand C Pilot Project
• Obtain more sample yield, beyond yield from
probability sample
• Recover something like a probability sample
• Option 1. Among initially selected households (seeds)
through probability sampling, ask the respondent to
recruit k subjects from the target population within the
EA
• Continue for s waves
• Option 2. Start with a convenience sample of
households in the EA
108

Out-of-school sample options
 Within each major strata
• Primary -- Probability sample
• Household area sample
• Secondary -- Non-probability sample options
– Household area sample with link-tracing
– School frame approach -- Strand A schools
» Recruit through students
» Apply link-tracing within X kms
• Street children
109

In-School lower than 7th grade
sample options
 Within each major strata
• Primary -- Probability-based
• Household area sample
• Secondary -- Non-probability sample options
• Household area sample with link-tracing
• School frame approach
– List all schools associated with Strand A schools with 4th,
5th, or 6th grade
– List all students 14 to 16 years old within the schools
– Take-all, or sample, depending on the number of
students listed
– Administer test either at school or household
110

Sample Size
 Field Test
• n = 1 200
 Main Study
• n = 2 700
 May need more sample if more than one language
administered for the assessment
111

RespondentStudent
Data Collection
Strands A-B
(Group Administration)
Strand C
(Interview, paper-based)
112
(Reading, Mathematics and
Science)
Student Questionnaire
(Reading and Mathematics)
School
Questionnaire
Teacher
Questionnaire
Background Questionnaire
Parent
Questionnaire

Reading Items by Process
113
Process
PISA-D FT
StrandA
PISA-D FT
StrandC
PISA 2015
Trend
%
Framework
Goal
Number % Number %
Access and
retrieve
25 31% 14 39% 29% 25-30%
Integrateand
interpret
34 43% 13 36% 49% 45-55%
Reflect and
evaluate
21 26% 9 25% 23% 15-25%
Total 80 100% 36 100% 100% 100%

Reading Items by Situation
114
Situation
PISA-D FT
StrandA
PISA-D FT
StrandC
PISA 2015
Trend
%
Framework
Goal
Number % Number %
Personal 25 31% 5 14% 27% 36%
Educational 25 31% 12 33% 28% 33%
Occupational 12 15% 10 28% 23% 20%
Public 18 23% 9 25% 22% 11%
Total 80 100% 36 100% 100% 100%

Mathematics Items by Process
115
Process
PISA-D FT
StrandA
PISA-D FT
StrandC
PISA 2015
Trend
%
Framework
Goal
Number % Number %
Formulating
situations
mathematically
19 25% 8 18% 29% Approx. 25%
Employing
mathematical
concepts, facts,
procedures
30 39% 21 47% 43% Approx. 50%
Interpreting,
applying, and
evaluating
mathematical
outcomes
27 36% 16 36% 28% Approx. 25%
Total 76 100% 45 100% 100% 100%

Mathematics Items by Content
116
Content
PISA-D FT
StrandA
PISA-D FT
StrandC
PISA 2015
Trend
%
Framework
Goal
Number % Number %
Change and
relationships
13 17% 7 16% 27% 25%
Space and shape 16 21% 10 22% 23% 25%
Quantity 28 37% 19 42% 25% 25%
Uncertainty and
data
19 25% 9 20% 25% 25%
Total 76 100% 45 100% 100% 100%

Instrumentation
Strands A-B
• Tests: paper booklets,
translated through Word
files
• Questionnaires: paper
booklets, translated
through Word files
Strand C
• Tests: computer-based,
authored, translated through
xliff files
• Questionnaires: computer-
based, authored, translated
through an authoring tool;
paper-based (parent)
117
Common Materials
(same translations)
Unique Strands A-B Materials Unique Strand C Materials

PISA
for Development
Contextual Questionnaires for
Educational Prosperity
J. Douglas Willms
University of New Brunswick
and
The Learning Bar

Strand B – The Learning Bar
 April 2014 – PISA-D Technical Workshop
• Seven core themes
• Willms and Tramonte working paper
 May 2014 – IAG Paris
• ToR Meeting: discussion of life-course approach for contextual framework
 Dec 2014 – UNESCO workshop on inequalities
 Mar 2015 – IAG Paris
• Draft contextual framework
 Feb – May 2015 – country visits – Educational Prosperity model
• Ecuador, Guatemala, Senegal, Paraguay, Zambia
 Aug 2015 – QEG meeting: Educational Prosperity and draft questionnaires
 Sep – Oct 2015 – Quito NPM – further consultation on questionnaires and
framework
 Nov 2015 – Paris summit - Finalized Questionnaires
 Jan 2015 – Washington – NPM – Adaptations and Translation, Country Options
 Mar-Apr 2016 – Asuncion NPM – CF, Qs, Translation process complete

Most countries have evaluation systems that
involve testing students at certain grades during
primary and secondary schooling.
Many of these countries also participate in large-
scale assessment programs, such as the
Programme for International Student Assessment
(PISA).

• An underlying assumption of these
evaluations is that students’test scores are
mainly the result of the efforts of teachers
and principals.
• However, students’results in local, national
and international assessments are the
cumulative result of children’s environments
during the prenatal period, their learning
experiences at home and in centers during
the pre-schoolperiod, and their family and
school experiences after they enter
kindergarten.
• Consequently, the results are often used
inappropriately to develop educational policy
and to hold educators accountable.

Educational
Prosperity:
A Life-Cycle Approach

• Educational Prosperity includes a core
set of metrics for success at six key
stages of development across the life-
course from conception to
adolescence.
• These metrics include a set of key
outcomes for each developmental
stage, called ‘prosperity outcomes,’
and a set of family, institutional, and
community factors, called ‘foundations
for success,’ which drive the prosperity
outcomes.
• The approach considers four ways that
success accumulates over the life-span.

Success
accumulates in four
ways:
1. Biological
embedding
2. Foundations for
success
3. Cumulative effects
4. Selection

1. Biological
Embedding
Differential social experiences get under the skin in early
life and, through their effects on developing
neurobiological pathways, affect later trajectories in
human health, learning, and behaviour?
Differentialsocial experiences
Socioeconomic gradients are not evident in most
social outcomes, they are also evident in children’s
early experiences; for example,
 Breast-feeding
 Smoking during pregnancy

under the skin
A key development in the field of human
development is the work defining neuro-biological
development and its effects on health, learning, and
behaviour.
 Brain development from conception to age one
is rapid and extensive, much more so than
previously believed, and is heavily influenced
by the infant’s environment (Carnegie
Corporation of New York, 1994).

• A newborn has billions of neurons, which, during
the course of development, form connections
called synapses. These synapses are formed in
response to environmental stimuli, and while this
is occurring, many of the neurons that are not
being used are pruned away.
• This process of synapse formation and neuron
pruning is often referred to as the “wiring” or
“sculpting” of the brain. Moreover, there are
critical periods, especially during the first three
years, when particular areas of the brain are
sculpted
• Longitudinal studies that have followed children
who have received intensive interventions aimed
at increasing stimulation and providing parent
training and support have demonstrated long-
lasting effects on their social, behavioural, and

2. Foundations for
SuccessIn addition to the effects that are biologically
embedded, children’s outcomes are directly
affected by the foundations for success at
each stage of development.
For example, during the formative period
from birth to age 2, children’s development is
affected by parents’ engagement with the
child and intra-family relations. From age 2 to
age 5, children’s development is affected by
these factors as well as their child care
experiences at home and in early childhood
centers.

Foundation of Success factors
are:
Potent (strong effects on
outcomes)
Pervasive (effect a range of
outcomes)
Proximal (has a direct effect on
the outcomes)

The timely transition from learning-to-read to reading-to-learn, which
for most children occurs at about age 8 or 9, is essential to academic
success,school attainment and well-being.
During the primary school years, from kindergarten to Grade 3,
considerable emphasis is placed on the development of reading skills.
Of course, children learn subject-mattercontentand acquire many
skills during the learning-to-read period.
But after Grade 3 there is a tacit assumption that children can read
fluently and comprehend curricular materials in subject domains such
as health, social studies, and science. The curriculum changes:
studentsare expected to learn the languages of subject domains and
use that language to think critically, solve problems, and create new
knowledge. The demands for strong reading skills increase as
The critical transitionfromlearning-to-readto
reading-to-learn

ConfidentLearners
A whole-school literacy programfor vulnerable
schools
‘The simple view of reading’ (Rose,
2006) has two critical,
complementary dimensions:
Code-related skills – the ability to
recognize and understand particular
words, and
Language skills – being able to
understand and interpret spoken
and written language.

Code-Related Skills
Conce
pts
About
Print
Letter
Knowle
dge
Phonologica
l Awareness
Supra-
Phonemic
Awareness
(Large Units)
Phonemic
Awareness
(Small units)
Reading
Fluency
Accuracy:
Word
Decoding
and Spelling
Accuracy:
Word
Recognition
and Spelling
Speed
and
Prosody

Language Skills
Vocabulary
Receptive
Language
Expressive
Language
Written
Language

3. Cumulative
DevelopmentChildren develop their skills in a cumulative
process as they make the transition from one
stage to the next. The skills they attain at one
stage are an asset that they can use to
develop skills at the next stage.
For example, children’s acquisition of
language skills at age 2 is a strong predictor of
their pre-literacy skills at age 5.

12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Child's Age (Months)
0
100
200
300
400
500
600
700
800
VocabularySize(Words)
Average Growth
Trajectory
Females
Males
Huttenlocher et al., 1991

The Early Years Evaluation assesses skills in
five domains
• Awareness of Self and Environment- a child's
understandingof the world and his or her abilityto
make connections with home and community
experiences.
• Social Skills and Approaches to Learning - a
child’s attentivenessduring classroom activities and
his or her abilityto interact with peers while
respecting the classroom rules.
• Cognitive Skills - a child's basic math and pre-
reading skills and his or her abilityto solve problems.
• Language and Communication- a child's
understandingof spoken language and his or her
abilityto express thoughtsand feelings.
• Physical Development
Fine motor - a child's abilityto perform small
movements that require hand-eyecoordination.
Gross motor - a child's abilityto perform large

The Early Years Evaluation assesses skills in
five domains
In the Colonia pilot study sample (N = 3,425), the reliability
coefficients (Cronbach’s alpha) of the EYE-TA subtests were as
follows:
Awareness of Self and Environment: 0.91
Social Skills and Approaches to Learning: 0.93
Cognitive Skills: 0.93
Language and Communication: 0.93
Physical Development: 0.90
The Latin American version of the Early Years Evaluation will be
used with all kindergarten children in Uruguay.

4. Selection
When students are successful at one stage of
development, their life-course can be altered if
they are selected into certain classes, school
programs or schools.
For example, children who have strong reading
and language skills are more likely to be streamed
into classes or school programs where they benefit
from positive peer interactions, a higher quality of
instruction, and other factors that enable them to
develop their skills at a faster pace.
Children who experience learning difficulties at a
particular stage are more likely to be streamed into
lower ability classes and have less access to the
factors that improve their skills.

The Role of Contextual Questionnaires
in

The goals of PISA for Development
1. Increase countries use of PISA assessmentsfor
monitoring progress towards nationally-set
targets for improvement.
2. Analyse factors associated with student learning
outcomes, particularly for poor and marginalized
populations; and
3. Build capacity for tracking international
education targets in the post-2015 framework
being developed within the UN’s thematic
consultations.

Educational Prosperity builds on key findings
of PISA
Educational Prosperity provides an opportunity for countries to
build on the knowledge acquired with PISA over the six cycles since
2000.
1. An understanding of why students with similar levels of
socioeconomicstatus achieve higher levels of performance in
some schools and school systems compared with others.
- the quest for ‘school effects’
- input-process-outputmodels
2. Identify subject-mattervalues and attitudes related to
performance that may lead students to follow a particular
career path; and
3. Measure general ‘global competencies’ or 21st Century skills
that are related to success in the workplace after students leave

The contextual questionnaires for
PISA for Development will …
1. Provide valid and reliable measures of a set of prosperity
outcomes and foundations for success.
The goal is not to measure 20 or 30 factors that are directly or indirectly
related to student performance. Rather, it is to measure in greater depth a
small set of outcomes relevant to participating countries, and the key factors
that drive these outcomes.
2. Provide an extended measure of socioeconomic status and
poverty that can be used alongside other demographic
measures to assess equality and equity.
3. Enable countries to use the data from PISA for Development in
a straightforward and explicit way to inform policy and practice
at all levels of the school system.

Contextual questionnaire content for the
Field Trial
Principles for the development of measures
1. Inclusion of core content from main PISA
2. Content that is high priority for participating countries.
3. Measures that are consistent with the Educational Prosperity
framework.
4. Readability – Grade 5 or lower and words with ‘age-of-
acquisition’ of age 10 or lower.
5. Tractable measures with direct links to policy and practice.

Contextual questionnaire content for the
Field Trial
School questionnaire – 28 questions
focusing mainly to student demographics
and the foundations for success.Incudes
detailed measures of material resources
and inclusive environments.
Teacher questionnaire – 36 questions
pertaining mainly student demographics
and the foundations for success.Incudes
detailed measures of quality instruction
and inclusive environments.
Student questionnaire – 83 questions
rotated over three booklets. It includes
several questions on students’ language
and their socioeconomic status.
 Trend
questions
 New
questions

Capacity Building Activities
• 4th PISA-D objective: country capacity in assessment and
analysis strengthened

Preparation for participation and support during
implementation:
Technical
support and
assistance
Project
implementation
plan
Capacity
building plan
Capacity needs
analysis

OECD support for and collaborative working to
produce:
Policy
dialogue
Dissemination
of results
Preparation
of a country
report
Country-
specific
data
analysis
Preparation
of in-country
stakeholders

• International/NPM meetings and follow-up
• Manuals, guidelines, procedures – translations of these
• Peer-to-peer learning and support
• Additional training and capacity building workshops
and events, engagement and communication efforts
• Country visits by OECD and contractors
• Support for country specific analysis of results and
collaboration with OECD over the writing of a national
report and dissemination activities, including support
for policy dialogue

International/NPM meetings
Meeting Venue Dates Content Peers
2015 International Advisory Group
meeting (3 days)
Paris, France 11-13 March 2015 Project planning and
thematic discussions.
Belgium
Spain
International/NPM#1/CapacityBuilding
meeting (4-5 days)
Quito, Ecuador 28 September – 2
October 2015
PISA cognitive and
contextual
frameworks,
characteristics of the
available itempools.
Chile
Korea
Mexico
Peru
Uruguay
meeting(4-5 days)
Rockville, MD,
United States
25-29 January
2016
Adaptation, translation
and verification of
survey materials and
sampling.
Canada
US
meeting (3 days)
Asunción,
Paraguay
30 March – 1 April
2016
Project planning and
Kosovo
meeting (4-5 days)
Asunción,
Paraguay
4-8 April 2016 Student sampling and
field trial survey
operations.
Brazil

International/NPM#4/Capacity Building
meeting (4-5 days)
Livingston, Zambia 4-8 July 2016 Scoring and coder
training and data
management for the
field trial.
China
International/NPM#4a/CapacityBuilding
meeting for Strand C only (2-3 days)
Barcelona, Spain 1-3 November
2016
Quality control sample
selection forms, and
quality assurance
procedures, data
management
(software, codebook,
etc.)
Spain
2017 International Advisory Groupmeeting
(3 days)
Cambodia 17-19 May 2017 Project planning and
Iceland
meeting (4-5 days)
Cambodia 22-26 May 2017 Analysis and
interpretation of field
trial results and
preparation for main
study.
Korea
meeting (4-5 days)
Princeton 4-8 July 2017 Student sampling and
main study survey
operations.
Canada
US
International/NPM#6a/Capacity Building
meeting Strand C only (2-3 days)
Panama city,
Panama
14-18 January
2018
MS interviewer training
and data management
training.

meeting (3 days)
Saly, Senegal 14-16 March 2018 Project planning and
meeting (4-5 days)
Saly, Senegal 19-23 March 2018 Data processing,
results, scaling
methodology and
preparation for
analysis.
meeting (8-10 days)
Antigua,
Guatemala
July 2018 Analysis and
interpretation of main
study results, reporting
and dissemination of
results.

Peer-to-peer learning
The goals of peer-to-peer learning in PISA-D are:
• help key staff in PISA-D countries to attain the level of project
implementation outlined in PISA’s Standards, capacity building
plans and other project documents,
• enhance the management of large-scale assessmentsin PISA-D
countries,
• provide key staff in PISA-D National Centres with opportunities
for professional growth and development,
• provide opportunities for collegial sharing and reflective practice,
• accelerate the capacity building objectives of the project.
*Guidelines to support the process*

Peer learning partnerships:
Peers Peer Learners
Korea Cambodia
Uruguay Ecuador
Peru Guatemala
Brazil Honduras
Chile Paraguay
Canada Senegal
Hong Kong/China/Iceland Zambia

• Peer learning partnerships involving exchange
visits, workshops,tele-conferences, email
exchanges
• Contributionsto international/NPMmeetings,
IAG meetings, other workshops and training
events
• Case studies of participating in PISA, institutional
histories of PISA national centres

Additional training and capacity building workshops and
events, engagement and communication efforts
• Project launches and facilitation of project and national
engagement and communication strategies
• With IDB, capacity building workshops for Latin
American countries on frameworks and item
development; Item Response Theory; secondary
analysis of data; and communication and engagement
with stakeholders
• Item Response Theory workshop for Senegal
• Capacity building workshops and partnership between
Cambodia and Korea Institute for Curriculum and
Evaluation

Feedback from the countries
• How many members of your team have benefited from
participation in an international NPM meeting or other capacity
building event since the last IAG? How did you disseminate what
you learnt at these meetings to other members of your team?
• For Ecuador, what was it like to host an NPM meeting? What
advice would you give to future hosts?
• For Cambodia (and Korea) what progress have you made with
capacity building workshops?
• For the Latin American countries (and IDB) what progress have
you made with capacity building workshops?

Feedback from the countries
• How are the peer-to-peer learning partnerships working out?
• Which aspect of the project so far has been most helpful in
building your capacity? Has this benefited in any way your
national assessment?
• What are you most looking forward to in the next 12 months for
capacity building?
• Are there any important areas of capacity building included in
your plans that the project or other partnerships not yet
addressing or planning to address?

3th PISA-D International Advisory Group
Meeting
Capacity Building Activities and
Outcomes
Granados Park Hotel, Asuncion, Paraguay

Structure
• Steering Committee
provides strategic
direction and
oversight
• Technical Task Force
performs project tasks
and develops capacity
that can be used by
other projects
• Quality Assurance
Department is
responsible for all
PISA-D activities
Steering
Committee
Task Force
Policy
Makers
Develop
ment
Partners
Quality
Assurance
Departmen
t
ERC/RUPP
OECD-
PISA
Peer
Countries
 Development
Partners,
 Curriculum,
 National assessment,
 Regional assessment,
 Research,
 Teacher Training,
 Labour,
 Private sector (foreign
and domestic)

Structure, cont.
• Steering Committee Memberships: Minister,
Secretary of States, Under Secretary of States,
Director Generals, Deputy Director Generals,
Department Directors, and Development
Partners
• Technical Task Force Memberships: Director
General (leader), Department Directors,
Education Research Council and Development
Partners
• Education Quality Assurance Department
Members: 12, 7 are full time staff and
receiving on the job training

Capacity Building
• February 2016: a workshop on PISA-D was held in Phnom Penh,
hosted by Ministry of Education and KICE, supported by
UNESCO Bangkok
 Participants: related departments/institutions, NGOs,
Development Partners, provincial officials and 8 NEQMAP
(Network on Education Quality Monitoring in the Asia-Pacific)
member countries (Bhutan, Laos, Mongolia, Myanmar, Nepal,
Pakistan, Sri Lanka)
 The topics: Introduction of PISA and PISA-D, PISA Implementation
(Case Study of Korea), PISA results implementation for
participating countries (KICE), PISA experience of middle-income
countries and its results (OECD), Cambodia Case Study (EQAD),
AssessmentFramework and Item development (by KICE),
Secondary data analysis and reports (KICE)
• Peer-to-peer learning partnerships: KICE and Cambodia are
discussing on the activities for the capacity building after the
Launching event

Capacity Building, cont.
• For the near future, below topics are be
expected to get support from KICE: Item
and Questionnaire Developments, Sampling
Design, Data Collection and Security
Protocols, Data Analysis and Report Writing
• Suggested topics: The use of the findings to
inform the policy, curriculum,
teaching/learning should be added to be a
complete set of assessment cycle

Q1. How many members of your team have benefited from
participation in an international NPM meeting or other capacity
building event since the last IAG? How did you disseminate what you
learnt at these meetings to other members of your team?
• 4 members of the team
All meetings are subject to a report sent to the Minister of Education, the
Secretary-General and the departments concerned by PISA -D.
The results of the meetings of the steering committee and technical
committees are shared with the technical and financial partners during the
meetings of the Steering Committee and the National Technical
Committee.
Every year, Senegal organizes two meetings of this kind for information on
progress made and difficulties noted. Important decisions are selected at
the end of these interministerial meetings.
Coordination meetings at the Ministry of Education are also sharing
opportunities with other directors.
Moreover, INEADE the technical team meets regularly to exchange around
PISA-D. Roles and responsibilities are distributed among all members of the
technical team.

Q2. What progress have you made with capacity building
workshops?
• The CBP was finalized interactively with the consultant Satya
Brink.
• The summary document prepared by the NPM was sent to the
Minister and all training activities were planned.
• The budget for the organization of training sessions is validated
by all stakeholders in the Ministry (DAGE - DPRE - DMSG ).
• The INEADE forwarded the terms of reference for the training of
20 members of the technical team at the OECD and DPRE for
execution. The team is looking for a consultant to the holding of
the first session which covers the IRT and writing items. It is
imperative to receive three offers to enable the Department to
choose the most appropriate profile. We had planned to hold
this session in the month of April 2016 if all constraints are lifted.

Q3. How are the peer-to-peer learning partnerships working
out?
• For language reasons, Senegal has chosen Canada as the
sponsor in the peer-to -peer learning.
• Meetings of peer-to- peer learning as part of the NPM Peru,
Brasil, Belgium, France and Canada were given the
opportunity to learn from the experience of these countries.
• In the absence of a budget for this activity,experience tours
that would have given the opportunity to understand more
about how countries organize their teams and solve
problems , peer- to-peer learning has difficultyfunctioning
well. However, Senegal appreciates the openness of the
NPM of Canada and its embassy who are willing to
accompany us in the PISA -D.

Q4. Which aspect of the project so far has been most helpful in
building your capacity? Has this benefited in any way your national
assessment?
PISA-D provides lifelong learning in all aspects from planning,
implementation and the sharing of results. All activities
conducted support the idea that PISA is the most rigorous
evaluation. The Senegal boost:
- The collaborative approach in developing the terms of referenceof the
meetings, feedbacks studies and reports submitted by firms (ETS, Pearsons ,
TLB, IUS)
- Shared the quality of scientific papers with the teams that are immense
treasures and heritage for conducting evaluations (reference frameworks ,
guides , tools)
- The multiplicity of tools and communication management (Sharepoint ,
permanent mail reminders)
- The feedback and support provided to countries to enable them to make
progress in the assigned tasks (sampling task, action plan, negotiations with
financial and technical partners)

Q5. What are you most looking forward to in the next 12
months for capacity building?
Strengthening capacity of Senegal’s PISA-D team in the techniques
of multivariate analysis and sharing methodologies and reporting of
assessmentdata (datasvisualization)

Q6. Are there any important areas of capacity building included in
your plans that the project or other partnerships not yet addressing
or planning to address?
The only area not covered by the project is peer-to-peer learning,
which has not yet received funding. We hope to revive the Canadian
Embassy or other partner to afford to visit the Canadian experience
in PISA . Overall, the plan has good coverage.

Country Report on Capacity
Building Activities
IAG Meeting (30th March, 2016), Asunción,
Paraguay
Zambia

Outline of Presentation
• Overview of Progress on PISA – D
• Capacity Building Plans
• NPM Meetings
• International Contractors
• Peer to Peer
• Way Forward

Overview on Progress
• Signed Agreement
• Launched Project
• Paid International Fees
• Attended IAG Meetings
• Attended NPM Meetings
• Visits from Contractors & Secretariat
• Video Conferencing, E-mails and Calls with
Secretariat & Contractors
• In Country Meetings with Key Stakeholders
• Working with Contractors on Adaptation and
Sampling

Capacity Building Plans
• Worked with International Contractor on Capacity
Needs.
• Identified Capacity needs were costed and
budgeted.
• Worked with Secretariat to align the capacity
needs to international project implementation
timeline.
• Streamlined capacity needs with those to be
covered by the International Contractors.
• Zambia is sourcing funding for activities not
covered by International Contractors.

NPM Meetings
• Zambia has attended all the two NPM and
technical meetings held so far.
• Participated in the PISA Strand A Assessment
Framework development.
• Appreciated considerations on development of
Frameworks for Strands A, B & C.
• Training on Sampling design, Adaption and
Translation received.
• Interacted with PISA countries, Experts,
Developmental Partners, Other Assessment
organizations and Institutions.

International Contractors
• During Meetings InternationalContractors
have built capacity.
• International Contractors have visited Zambia
to work with local team.
• International Contractors have been guiding
Country team through e-mail on various
activities through e-mail, Video Conferencing
and phone calls.

Peer to Peer
• Through meetings such as NPM peer to peer
learning has taken place.
• The hosting of NPM and IAG meetings has
facilitated understanding other members
situations and how best Zambia can implement
the project.
• Involvement non OECD countries who have
participated in PISA has been a positive influence.
• Meeting and talking with OECD Countries with
experience in PISA has also been beneficial.

Way Forward
• Zambia to secure resources to undertake capacity
building activities not covered in the NPM
Meetings and International Contractors.
• NC to partner with local institutions who have
capacity in some areas of need such as Strand C’s
household surveys.
• Zambia will continue cherishing the support of
the Secretariat , International Contractors and
other partners in the successful implementation
of the PISA – D project in the country.

Conclusions Day One

Strong representation at the meeting from:
• Countries signed up (Cambodia, Ecuador,
Guatemala, Honduras, Paraguay, Senegal,
Zambia) or committed (Panama) to participation
in PISA for Development
• Development partners (France, Germany
(BMZ/GIZ), IDB, Sunny Varkey Foundation,
Japan(JICA), Norway, UK (DFID), World Bank)
• International agencies (UNESCO, UIS, UNICEF,
LLECE)
• International contractors.
Conclusions Day One

Conclusions Day One
• Two co-chairs elected:
– one representative of the participating countries: Rodolfo Elias (Paraguay)
– one representative of the development partners: Maria Soledad Bos (IDB)
• Annual report on project activities reviewed and approved with
suggestions for improving presentation of project progress and
highlighting risks (particularly related to risk 4 on project impact), and
praised the substantial progress that has been made
• IAG members noted the new development partners and discussed the
role of these within the IAG
• IAG members also noted the challenge of countries for financing their
participation in international meetings and asked OECD to continue
reporting on this

Conclusions Day One
• OECD confirmed that no more countries will join PISA-D
• IAG members noted that Strand C activities are on track despite
there not yet being a signed contract with ETS and encouraged
OECD to sign the contract as soon as possible
• Members noted the presentationsby ETS on Strand A and
commended the countries, contractors and Secretariat for the
progress that has been made
• IAG members discussed issues related to comparability in the
contextof PISA-D having no major domain, computer-basedvs.
paper-based, and trend links

Conclusions Day One
• Members noted the presentationsby TLB on Strand B and
progress that has been made
• There was a discussion of how the framework for the
questionnaires are reflected in the items selected for the
instruments
• Members noted the presentation by ETS on Strand C and
progress that has been made in preparing this component
• There was a discussion of the timeline for Strand C and countries
noted the short window of time available for the field trial data
collection

PISA-D Annual Report and Progress

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (15)

Ähnlich wie PISA-D Annual Report and Progress

Ähnlich wie PISA-D Annual Report and Progress (20)

Mehr von EduSkills OECD

Mehr von EduSkills OECD (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

PISA-D Annual Report and Progress