The document discusses limitations of international standardized data collection in assessing learner diversity and inclusive practices. It acknowledges progress made through comparative analyses but notes significant resistance from teachers due to misuse of data to push political agendas. Key limitations of current data are the failure to record student challenges to learning, school environments' ability to be inclusive, and impacts of school culture. There is a need for large-scale qualitative tools to evaluate ecological factors influencing inclusion and better reflect realities on the ground. Key indicators like ableism, racism, and socioeconomic challenges remain largely ignored.

1. What is the importantdata that is not beingrecordedin
comparativeinternationalanalyses on studentsuccess?
Examiningthechallengesof addressinglearnerdiversity
proactivelyand of quantifyingthe impactof inclusivepractices.
CIMQUSEF18: 18th Edition of the International Conference on the Quality of Education
The main determinants of improving learning outcomes: Analysis of PISA, TIMSS &
PIRLS data
Frederic Fovet, School of Education and Technology, RRU

2. Objectives of the Session
• Acknowledge the global push back against national and international data analysis in
education in the current neo-liberal landscape
• Explore some of the motivations behind that push back
• Examine the extent to which inclusive practices currently fail to be recorded by
comparative international data collection
• Explore avenues for reform and for the development of pertinent large scale
qualitative tools for the evaluation of ecological factors that impact inclusion

3. Phenomenal progress
• There have been giant steps made in the last decade with regards to the ways data
on student performance is collected, analyzed and used for school improvement
(Breakspear, 2014; Rozgonjuk et al., 2019; Wu et al., 2020).
• Much of the impact of the analysis of this data lies in the fact that it has allowed for
large international comparative studies that yield important conclusions on the
effectiveness of teaching practices, curriculum, and modes of assessment (Dickinson,
2019; OECD 2000-2015).
• The PISA framework and annual PISA results have in particular allowed for revealing
reflections, at international level, in relation to the objectives, ethos and
performance of national educational structures (Krieg, 2019; Patrinos & Angrist,
2018).

4. But also unprecedented resistance
• There is widespread resistance, misunderstanding and fear among teachers and
administrators against all forms of standardized testing.
• Continued disquiet - Pons (2017) talks about the “ongoing PISA shock”
• Lundahl & Serder (2020) discuss the dangerous precedence of PISA stats over
educational research. Tveit & Lundahl (2018) also discuss the misuse of international
comparative data on education to push political reform
• Explicit push back in the field: Tension with regards to purpose of assessment
(Penninckx, 2020); tension with regards to goals and values of teacher training
(Brooks, 2021); clash with learning outcomes and teaching philosophies (Shelton &
Brooks, 2019)
• Also considerable evidence of en masse push-back from parents (Simon, 2012)

5. Causes of this resistance
• Much of the push back emerges from fears of the Neoliberal agenda that
underlies much of the discourse on quality indicators in education.
• Though the essential objective is to obtain a snapshot of practices and
standards for comparative purposes, the objective is now widely seen as
simply ‘scoring high’ in standardized testing.
• Has led to a process where this creates tension on school leaders to ‘dribble
down’ pressure on teachers rather than on examining the root cause of low
performance.
• In many contexts, this pressure is then passed on to the student to ‘over-
perform’ in standardized screenings.
• It can be said that at this stage, on the ground, international efforts to gather
standardized data are misunderstood and misused.

7. A disquiet that is not going away… And
should it?
• International comparative studies are carried out on the data collected in
these large surveys/ standardized exercises.
• However, these studies have yet to examine learner diversity or an
educational system’s ability to develop, grow and sustain inclusive practices in
schools (Krammer et al., 2021).
• As a result, a significant gap exists in the quantitative data that is emerging
from international comparative studies (Ainscow, 2015; Booth & Ainscow,
2002; Poulsen & Hewson, 2014).
• Much of the disquiet felt by teachers relates the increasing chasm between
awareness of/ policy around learner diversity and inclusion vs. global
quantitative outcomes.

8. Limitationsof international, comparativestandardizeddata
on the issuesof learner diversityand inclusivepractices
• Three angles that are under-reported or not reported at all:
(1) Students’ challenges with access to learning (acknowledging student diversity)
In the simplest of ways, learner diversity is entirely erased from the tools for data
collection and analysis. In most contexts, students with disabilities for example are not
given access to accommodations. Nor are students allowed to use assessment
strategies they may have used for years to achieve positive outcomes.
(2) Environments’ ability to design inclusively (social model perspective)
Current international comparative tools dismiss the importance of the design of the
learning environment and experience/ the role of the educator as designer of an
inclusive experience. Tools quantify learners as though they remain unaffected by the
practices of educators beyond competencies themselves.
(3) School culture and its impact on learning outcomes (systemic dimension)
There is no provision currently to assess school culture and its impact on learning
outcomes, even of the literature constantly reiterates its key role in student success

9. The COVID Crisis and its impact on this
tension
• The school closures and the pivot to online teaching and assessment has led to a
cathartic understanding of the limitations of current national and international data
practices in education.
• Inequities have increased, the gap has widened. Some learners and communities
have been disastrously left out.
• Current quality insurance tools in the K-12 sector have been unable to reflect these
challenges and these realities.

10. Quantitative tools that do exist but are currently
under-utilized in terms of data mining
• Strength through Diversity: Education for Inclusive Societies. Design and Implementation Plan
(OECD, 2019) is promising but mostly focused on refugees and migrants – somewhat limited
concept of learner diversity
• Some large scale projects produced by private foundations, such as the Century Foundation’s
How Racially Diverse Schools and Classrooms Can Benefit All Students (Wells, Fox & Cordova-
Cobo, 2016). There are, however, inherent issues with work from the private sector in this
area: scope, lens, international and comparative flavour.
• Some key national reports on inclusive practices in specific sectors, such as the Interagency
Working Group on Inclusion in STEM’s (2021) Best practices for diversity and inclusion in
STEM Education and research: A guide by and for Federal Agencies. However, many of these
reports are proactive recommendations rather than the analysis of existing outcomes/ data.
• All in all, credible efforts are observed but we are still very far from functional, sustainable,
quantitative, international and comparative tools that have the ability to offer a pertinent
and valid snapshot of inclusive practices in the K-12 sector and of their impact on outcomes.

11. Need for large scale qualitative tools that
go beyond existing tools
• The Index for Inclusion (Booth & Ainscow, 2002) is a set of materials to guide schools through
a process of inclusive school development. It is about building supportive communities and
fostering high achievement for all staff and students.
• “The Index is a set of materials to support schools in a process of inclusive school
development, drawing on the views of staff, governors, school students, parents/carers and
other community members. It is concerned with improving educational attainments through
inclusive practice and thus provides an attempt to redress a balance in those schools which
have concentrated on raising student attainment at the expense of the development of a
supportive school community for staff and students.” (CSIE, 2019)
• Three objectives: (i) schools assess their practices, (ii) planning for improvement, (iii) ensures
participation of all stakeholders.
• However, the lack of impact of the Index for Inclusion in the last decade is evidence that it is
perhaps not the most malleable of tools. Onerous, complex, and does not allow for easy
comparisons across jurisdictions. It is more focused on the transformation of individual sites.

12. Need for large scale qualitative tools that
go beyond existing tools (contd.)
• Some challenges lie ahead in relation to the development of sustainable qualitative
tools that might allow for comparative analysis of the various ways national
education systems tackle the task of differentiating education.
• Epistemological hurdle: quantitative stats cannot be sufficiently deep to examine and
analyze the impact of the various ecological factors that impact the inclusion of
diverse learners. It is urgent that large scale qualitative tools become available.
• Persistent distrust with regards to qualitative tools in the field of education;
represents an epistemological malaise (Cunningham, 2019)
• The sector now has tools that are sufficiently refined and sophisticated to support
large scale data collection and analysis, and the exploration of complex ecological
indicators (Cope & Kalantzis, 2016).
• There is not political will, however, currently to invest in the creation and application
of these tools.

13. Key indicators that remain largely ignored
in learning outcomes
• Ableism: There is a need for subtlety and nuance here as much of the discourse on inclusion remains bio-
medical, instead of embracing the Social Model of Disability ().
• Racism: The Black Lives Matter movement has made most jurisdictions realize to what extent racism has been
ignored in all aspects of society and public governance. Education is not an exception
• Cultural ethno-centrism: There is increasing awareness that cultural diversity must be addressed effectively in
school, in the midst of a disrupted geo-political landscape where mass migration represents specific
challenges.
• Sexism: The #MeToo movement has increased public awareness of sexual and gender power inequities.
These exists in schools and the K-12 sector currently has no effective indicators to track their impact.
• Socio-economic challenges: While economic or sociological reports will often highlight the crucial impact of
poverty on education, standardized large scale data collection in the K-12 sector currently fails to
acknowledge how these factors shape learning outcomes.
• Decolonizing the curriculum: This is a key process in the K-12 sector (both with regards to ongoing internal
colonization in the Global North – TRC in Canada for example - and to post-colonial processes in the Global
South). There is a pressing need for indicators of success with regards to efforts to decolonize the K-12
curriculum.
• Importance of examining EDI, in terms of tangible ecological indicators, in a decade where the rest of society
finally grasps its impact on all sociological outcomes (employment, justice, law enforcement, media, etc.).
This momentum cannot be successfully driven and assessed if there are no indicators available in the K-12
sector.

14. References
Ainscow, M. (2015). Struggles for Equity in Education: The Selected Works of Mel Ainscow. Routledge. DOI:10.4324/978131156882213.
Booth, T., & Ainscow, M. (2002). Index for Inclusion: Developing Learning and Participation in Schools. Centre for Studies on Inclusive Education.
Breakspear, S. (2014) How does PISA shape education policy making? Why how we measure learning determines what counts in education. Centre for Strategic Education. https://allchildrenlearning.org/wp-content/uploads/2019/11/Breakspear-PISA-Paper.pdf
Brooks, C. (2021) The quality conundrum in initial teacher education. :Teachers and Teaching, 27(1-4), 131-146, DOI: 10.1080/13540602.2021.1933414
Cope, B., & Kalantzis, M. (2016). Big Data Comes to School: Implications for Learning, Assessment, and Research. AERA Open. https://doi.org/10.1177/2332858416641907
CSIE (2019) Breaking down the barriers: the Index for Inclusion. Centre for Studies on Inclusive Education supporting inclusion, challenging exclusion. http://www.csie.org.uk/resources/breaking-barriers.shtml
Cunningham, J. (2019). Missing the mark: Standardized testing as epistemological erasure in U.S. schooling. Power and Education, 11(1), 111–120. https://doi.org/10.1177/1757743818812093
Dickinson, K. (2019, February 15) Standardized tests: Finland’s education system vs. the U.S. Big Think. https://bigthink.com/politics-current-affairs/standardized-testing?rebelltitem=2#rebelltitem2
Interagency Working Group on Inclusion in STEM (2021) Best practices for diversity and inclusion in STEM Education and research: A guide by and for Federal Agencies. National Science and Technology Council. https://www.whitehouse.gov/wp-content/uploads/2021/09/091621-Best-Practices-for-Diversity-
Inclusion-in-STEM.pdf
Krammer, M., Gasteiger-Klicpera, B., Holzinger, A., & Wohlhart, D. (2021) Inclusion and standards achievement: the presence of pupils identified as having special needs as a moderating effect on the national mathematics standards achievements of their classmates. International Journal of Inclusive Education,
25(7), 795-811. DOI: 10.1080/13603116.2019.1573938
Krieg, J. (2019) International Student Assessments Performance and Spending. Fraser Institute. https://www.fraserinstitute.org/sites/default/files/international-student-assessment.pdf
Lundahl, C., & Serder, M. (2020) Is PISA more important to school reforms than educational research? The selective use of authoritative references in media and in parliamentary debates. Nordic Journal of Studies in Educational Policy, 6(3), 193-206,. DOI: 10.1080/20020317.2020.1831306
Organisation for Economic Co-operation and Development [OECD] (2000–2015). Program for International Student Assessment (PISA), 2000, 2003, 2006, 2009, 2012, and 2015 Mathematics, Reading, and Science Assessments. OECD
Organisation for Economic Co-operation and Development [OECD](2019) Strength through Diversity: Education for Inclusive Societies. Design and Implementation Plan. EDU/EDPC(2019)11/REV2. Organisation for Economic Co-operation and Development. https://www.oecd.org/education/strength-through-
diversity/Design-and-Implementation-Plan.pdf
Patrinos, H.A.., & Angrist, N. (2018) Global Dataset on Education Quality A Review and Update (2000–2017). World Bank Group. https://documents1.worldbank.org/curated/en/390321538076747773/pdf/WPS8592.pdf
Penninckx, M. (2020). Quality education without central examinations: The case of Flanders. Management in Education, 34(1), 40–43. https://doi.org/10.1177/0892020619881045
Pons, X. (2017) Fifteen years of research on PISA effects on education governance: A critical review. Eur J Educ., 52: 131– 144. https://doi.org/10.1111/ejed.12213
Poulsen, J., & Hewson, K. (2014) Standardized Testing: Fair or Not? Teaching Centre, University of Lethbridge. https://www.ulethbridge.ca/teachingcentre/standardized-testing-fair-or-not
Rozgonjuk, D., Täht, K. & Vassil, K. (2019) Internet use at and outside of school in relation to low- and high-stakes mathematics test scores across 3 years. International Journal of STEM Education, 8, 27. https://doi.org/10.1186/s40594-021-00287-y
Shelton, S., & Brooks, T. (2019) “We Need to Get These Scores Up”: A Narrative Examination of the Challenges of Teaching Literature in the Age of Standardized Testing. Journal of Language and Literacy Education. 19(2). https://files.eric.ed.gov/fulltext/EJ1235207.pdf
Simon, S. (2012, June 12) Parents Protest Surge in Standardized Testing. Reuters, U.S. edition . http://www.reuters.com/article/us-usa-education-testing-idUSBRE85B0EO20120612.
Tveit, S., & Lundahl, C. (2018). New modes of policy legitimation in education: (Mis)using comparative data to effectuate assessment reform. European Educational Research Journal, 17(5), 631–655. https://doi.org/10.1177/1474904117728846
Wells, A.S., Fox, L., & Cordova-Cobo, D. (2016) How Racially Diverse Schools and Classrooms Can Benefit All Students. The Century Foundation. https://tcf.org/content/report/how-racially-diverse-schools-and-classrooms-can-benefit-all-students/?agreed=1
Wu, X., Wu, R., Chang, H. H., Kong, Q., & Zhang, Y. (2020). International Comparative Study on PISA Mathematics Achievement Test Based on Cognitive Diagnostic Models. Frontiers in psychology, 11, 2230. https://doi.org/10.3389/fpsyg.2020.02230

15. Contact details
• Frederic Fovet (PhD.)
• Associate Professor, School of Education and Technology, Royal Roads
University
• Frederic.fovet@royalroads.ca
• @Ffovet
• www.implementudl.com