This presentation was given by Stéphan Vincent-Lancrin of the OECD at the project meeting “Fostering and assessing students' creativity and critical thinking in higher education” on 20 June 2016 in Paris, France.

1. ASSESSING PROGRESSION IN CREATIVE
AND CRITICAL THINKING SKILLS
Stéphan Vincent-Lancrin
Deputy Head, Senior Analyst and Project Lead,
OECD Centre for Educational Research and
Innovation
Paris, 20 June 2016

2. objectives of the meeting

3. • What are we talking about when we talk about creativity and
critical thinking in the higher education context?
• Peer learning about pedagogies trying to intentionally foster
students’ creativity and critical thinking
– Can we group them in categories?
– What characterises them?
• How are students assessed to show the acquisition of these
skills?
• What advice to start an action research in education aligned
with what we are doing in schooling?
Objectives of the meeting

4. • First day mainly about pedagogies:
– Design thinking
– Entrepreneurship
– Other approaches
• Second day morning mainly about evaluation and assessment:
– Standardised tools
– Faculty-designed asssessments
– Qualification frameworks
• Second day afternoon
– Discussing a possible international action research to showcase how
higher education institutions can foster their students’ creativity and
critical thinking
Structure of the meeting

5. skills for innovation

6. Skills and education for innovation
« 21st Century Skills »
Innovation
Skills
Education
and
training

7. Skills that tertiary-educated professionals
report as very important in their job
Percentage of employees reporting the following skills as very important in their job
Source: OECD, based on REFLEX and HEGESCO data
22.7
30.6
40.2
40.3
40.4
40.4
40.5
41.8
46.5
48.0
50.0
53.4
54.2
55.0
56.5
56.9
58.6
60.8
61.7
0 10 20 30 40 50 60 70
knowledge of other fields
write and speak a foreign language
assert your authority
negociate
alertness to opportunities
present ideas in audience
willingness to question ideas
mobilize capacities of others
analytical thinking
come with news ideas/solutions
write reports or documents
acquire new knowledge
master of your own field
coordinate activities
use computers and internet
make your meaning clear
work productively with others
perform under pressure
use time efficiently

8. Critical skills for the most innovative jobs
(according to tertiary-educated workers)
Likelihood (odds ratios) of reporting the following skills: people
in the most innovative jobs vs. least innovative jobs
Source: Avvisati, Jacotin and Vincent-Lancrin (2013), based on REFLEX and HEGESCO data
1.83
2.05
2.08
2.09
2.15
2.19
2.19
2.20
2.35
2.36
2.36
2.42
2.51
2.58
2.60
2.71
2.81
3.00 3.90
1
assert your authority
knowledge of other fields
negociate
perform under pressure
use time efficiently
work productively with others
use computers and internet
write and speak a foreign language
write reports or documents
master of your own field
make your meaning clear
mobilize capacities of others
acquire new knowledge
coordinate activities
analytical thinking
alertness to opportunities
present ideas in audience
willingness to question ideas
come with news ideas/solutions

9. Top 10 skills for future jobs according to
employers (2016)
2020
1. Complex problem solving
2. Critical thinking
3. Creativity
4. People management
5. Coordinating with others
6. Emotional intelligence
7. Judgment and decision
making
8. Service orientation
9. Negotiation
10. Cognitive flexibility
2015
1. Complex problem solving
2. Coordinating with others
3. People management
4. Critical thinking
5. Negotiation
6. Quality control
7. Service orientation
8. Judgment and decision
making
9. Active listening
10. Creativity
World Economic Forum

10. What individual skills should
education systems foster?
Technical skills
(know-what and know-
how)
Creativity and
critical thinking
skills
(Critical thinking,
observation, curiosity,
ability to make
connections,
imagination,...)
Social and
behavioural skills
(Self-confidence, energy,
perseverance, passion,
leadership, collaboration,
communication)

11. Some comments on these skill
categories
• They are domain-specific
– Skills are generally domain-specific: one is creative in a field, one knows how to
behave/communicate in a specific context, one has problem-solving skills in a field,
one has content knowledge in a field
• They can become « domain-generic »
– A skills becomes « domain-generic » when one has gained it in a number of
domains or settings, so that it becomes a « habit of mind » (a disposition or a
stabilised skill) that one can apply to new fields
• They overlap and may reinforce
each other
But
• They are different and cannot
be reduced to a single skill (or
measure)
Technical skills
Skills in
thinking and
creativity
Behavioural and
social skills
(character)

12. Skills and education for innovation
« 21st Century Skills »
Well-
being
Skills
Education
and
training

13. higher education for innovation

14. Higher education for innovation
Innovation
Skills
Education
and
training
• Which graduates get
highly innovative jobs?
• Do higher education
systems foster skills for
innovation?
• Which pedagogic
practices are associated
with highly innovative
jobs?

15. What share of graduates of a given field
have a highly innovative job?
20.3
22.9
23.4
25.4
27.6
28.1
28.1
28.4
28.8
30.9
32.7
36.5
37.6
0 10 20 30 40 50
law
health
humanities
others
business
sciences & maths
education
average
social sciences
architecture
agriculture
arts
engineering & computing
Product /service innovation

16. What share of graduates of a given field
have a highly innovative job?
12.4
12.6
13.2
15.2
16.0
18.4
18.6
21.0
23.9
24.2
28.6
28.8
38.4
0 10 20 30 40 50
law
others
humanities
social sciences
education
health
business
average
arts
agriculture
architecture
sciences & maths
engineering & computing
Technology, tool, instrument innovation

17. What share of graduates of a given field
have a highly innovative job?
30.4
31.5
35.1
36.1
37.6
38.0
38.3
38.5
39.5
39.9
41.7
41.7
48.3
0 10 20 30 40 50
law
others
business
humanities
social sciences
arts
average
health
agriculture
architecture
engineering & computing
education
sciences & maths
Knowledge / method innovation

18. Higher education for innovation
Innovation
Skills
Education
and
training
• Which graduates get
highly innovative jobs?
• Do higher education
systems foster skills for
innovation?
• Which pedagogic
practices are associated
with highly innovative
jobs?

19. Strong points of higher education
0 2 4 6 8 10 12 14 16 18 20
assert your authority
mobilize capacities of others
negociate
alertness to opportunities
willingness to question ideas
coordinate activities
write and speak a foreign language
make your meaning clear
come with news ideas/solutions
present ideas in audience
knowledge of other fields
use time efficiently
use computers and internet
perform under pressure
work productively with others
write reports or documents
acquire new knowledge
analytical thinking
master of your own field
Ranking of 3 top strong skills by graduates
Source: OECD, based on REFLEX, HEGESCO and PROFLEX data

20. Strong points of higher education
0 2 4 6 8 10 12 14 16 18 20
assert your authority
mobilize capacities of others
negociate
alertness to opportunities
willingness to question ideas
coordinate activities
write and speak a foreign language
make your meaning clear
come with news ideas/solutions
present ideas in audience
knowledge of other fields
use time efficiently
use computers and internet
perform under pressure
work productively with others
write reports or documents
acquire new knowledge
analytical thinking
master of your own field
Ranking of 3 top strong skills by graduates
Source: OECD, based on REFLEX, HEGESCO and PROFLEX data

21. Weak points of higher education
0 2 4 6 8 10 12 14 16 18 20
acquire new knowledge
master of your own field
make your meaning clear
coordinate activities
work productively with others
willingness to question ideas
analytical thinking
perform under pressure
use time efficiently
alertness to opportunities
come with news ideas/solutions
write reports or documents
mobilize capacities of others
knowledge of other fields
use computers and internet
present ideas in audience
assert your authority
negociate
write and speak a foreign language
Ranking of 3 top weak skills by graduates
Source: OECD, based on REFLEX, HEGESCO and PROFLEX data

22. Weak points of higher education
0 2 4 6 8 10 12 14 16 18 20
acquire new knowledge
master of your own field
make your meaning clear
coordinate activities
work productively with others
willingness to question ideas
analytical thinking
perform under pressure
use time efficiently
alertness to opportunities
come with news ideas/solutions
write reports or documents
mobilize capacities of others
knowledge of other fields
use computers and internet
present ideas in audience
assert your authority
negociate
write and speak a foreign language
Ranking of 3 top weak skills by graduates
Source: OECD, based on REFLEX, HEGESCO and PROFLEX data

23. Higher education for innovation
Innovation
Skills
Education
and
training
• Which graduates get
highly innovative jobs?
• Do higher education
systems foster skills for
innovation?
• Which pedagogic
practices are associated
with highly innovative
jobs?

24. Innovation and the relative emphasis on
practice- and theory-based instruction
0.95
1
1.05
1.1
1.15
1.2
engineering business health education science others
practice score theory score
Odds ratios between innovators and non-innovators, by field of study
Source: OECD, based on REFLEX and HEGESCO data

25. Innovation and the relative emphasis on
practice- and theory-based instruction
0.95
1
1.05
1.1
1.15
1.2
any innovation technology, tools product, service knowledge,
methods
practice score theory score
Odds ratios between innovators and non-innovators, by type of innovation
Source: OECD, based on REFLEX and HEGESCO data

26. ICT-enhanced pedagogic models to
foster « skills for innovation » in STEM
• Virtual and remote
laboratories
• Educational games
• Technology-enhanced
cooperative learning
• Real-time formative
assessment
• Skills-based
assessment

27. Problem-based learning in higher
education
Please cite this paper as:
Hoidn, S. and K. Kärkkäinen (2014), “Promoting Skills for
Innovation in Higher Education: A Literature Review on the
Effectiveness of Problem-based Learning and of Teaching
Behaviours”, OECD Education Working Papers, No. 100,
OECD Publishing.
http://dx.doi.org/10.1787/5k3tsj67l226-en
OECD Education Working Papers
No. 100
Promoting Skills for
Innovation in Higher
Education
A LITERATURE REVIEW ON THE
EFFECTIVENESS OF PROBLEM-BASED
LEARNING AND OF TEACHING BEHAVIOURS
Sabine Hoidn, Kiira Kärkkäinen
• PBL more beneficial
regarding long-term
retention and application
of knowledge and skills
• PBL has positive impact
on students’ motivation,
satisfaction, and attitudes
toward learning
• Traditional approaches
seem to have an edge for
test scores and knowledge
acquisition, but not
significant

28. fostering and assessing creativity and
critical thinking in education

29. • 2 networks:
– Higher education
– School (primary and secondary)
• 3 possible domains:
– One STEM discipline
– One arts education discipline
– One generic approach to innovation/creativity
• Activities in primary-secondary education:
– Develop a pedagogical toolkit (rubric, pedagogical activities and
assessments, student work)
– Document pedagogies to foster the selected skills/dispositions
• Activities in tertiary education:
– Share about pedagogies and identification of skills
– Pedagogical intervention like in primary and secondary?
Ongoing OECD project on assessing
progression in creativity and critical thinking

30. Participants in 14 countries (June 2016)
• Brazil
• China
• Finland
• France (3 teams)
• India
• Italy
• Hungary
• Netherlands
• Russia
• Slovakia
• Spain
• Thailand
• UK (Wales)
• USA (4 teams)

31. Initial version of the rubric (not to be shared)
CREATIVITY
(Coming up with ideas and solutions)
CRITICAL THINKING
(Questionning and evaluating ideas and
solutions)
Progression
INQUIRE
 Feel, empathise, observe,
describe relevant experience
and information
 Explore, seek and generate
ideas
 Understand context/frame and
boundaries of the problem
 Review alternative theories and
opinions and compare/find
perspectives on the problem
 Level 1
 Level 2
 Level 3
 Level 4
IMAGINE
 Make connections, integrate
other disciplinary perspectives
 Stretch and play with
unusual/risky/radical ideas
 Identify strengths and weaknesses
of evidence, arguments, claims and
beliefs
 Challenge assumptions, check
accuracy, analyse gaps in knowledge
 Level 1
 Level 2
 Level 3
 Level 4
DO / SHARE
 Envision / Express / Produce /
Prototype new product /
solution / performance
 Appreciate the novelty of
solution and/or possible
consequences
 Appraise / Base / Justify
opinion/products on logical, ethical
or aesthetic criteria/reasoning
 Acknowledge own bias (as perceived
by others) and uncertainty/limits of
endorsed opinion/solution
 Level 1
 Level 2
 Level 3
 Level 4

32. Creativity
Inquire
Imagine
Do /
Share
 Feel, empathise, observe, describe relevant
experience and information
 Explore, seek and generate ideas
 Make connections, integrate other
disciplinary perspectives
 Stretch and play with
unusual/risky/radical ideas
 Envision / Express / Produce / Prototype new
product / solution / performance
 Appreciate the novelty of solution and/or
possible consequences

33. Critical thinking
Inquire
Imagine
Do /
Share
 Understand context/frame and boundaries of
the problem
 Review alternative theories and opinions and
compare/find perspectives on the problem
 Identify strengths and weaknesses of evidence,
arguments, claims and beliefs
 Challenge assumptions, check accuracy,
analyse gaps in knowledge
 Appraise / Base / Justify opinion/products on
logical, ethical or aesthetic criteria/reasoning
 Acknowledge own bias (as perceived by others)
and uncertainty/limits of endorsed
opinion/solution

34. A rubric to help develop lesson plans
• Mapping of the different steps of a lesson plan on the rubric
• Not all dimensions need to be covered
• Generally task are mainly about creativity or critical
thinking (but can cover some habits of both)

35. • Develop scoring rubrics
• Develop student self-assessment rubrics
• Develop domain-specific rubrics
• To be further simplified
A rubric to…

36. The NCU rubric on creative thinking
• A, B: Defining problem, use of
evidence
• C: Identifies/challenges
assumptions
• D: Logical and inventive
combination of ideas
• E: Contradictory perspectives
• F, I: Multiple solutions to
problem
• G, J: Originality
• H: Changes when necessary
• K: Concludes by weighing
evidence
• L, M: Shares coherently

37. Creativity / match with NCU rubric
Inquire
Imagine
Do /
Share
 Feel, empathise, observe, describe relevant
experience and information - A
 Explore, seek and generate ideas – F, G
 Make connections, integrate other
disciplinary perspectives - D
 Stretch and play with
unusual/risky/radical ideas - G
 Envision / Express / Produce / Prototype new
product / solution / performance
 Appreciate the novelty of solution and/or
possible consequences – K, L?

38. Critical thinking/ match with NCU rubric
Inquire
Imagine
Do /
Share
 Understand context/frame and boundaries of the
problem - A
 Review alternative theories and opinions and
compare/find perspectives on the problem – E, F
 Identify strengths and weaknesses of evidence,
arguments, claims and beliefs
 Challenge assumptions, check accuracy, analyse
gaps in knowledge - C
 Appraise / Base / Justify opinion/products on logical,
ethical or aesthetic criteria/reasoning – B, M
 Acknowledge own bias (as perceived by others) and
uncertainty/limits of endorsed
opinion/solution

39. • What is missing?
• What would you omit?
• How would you simplify?
Your feedback on the current version of
the OECD rubric

40. a controlled pedagogical
intervention in higher education?

41. A pedagogical toolkit
A. The toolkit
A1.
Rubric for
assessment
Dimensions
Levels of
progression
A2.
Pedagogical
activities
Specific for
each domain
Designed to
test
dimensions
and levels of
progression
A3.
Set of
exercises
Specific for
each domain
Designed to
prepare for
the
assessment
A4.
Portfolio of
student work
For each
domain
As examples
of different
skill levels

42. Contextual data collection
B1.
Subject-based
assessment
Standardised
assessment of
academic
achievement
(maths and
science; visual
arts and
music)
B2.
Creativity
assessment
Standardised
test for
creativity
(EPoC)
(domain-
specific)
B3.
Survey
questionnaires
School
principals
Teachers
Students
B4.
Interviews /
focus groups
Teachers
Students

43. School-based intervention
School year
A A A A
Primary school group - Ages 8-9. STEM, Arts or interdisciplinary domain
Secondary school group - Ages 13-14. STEM, Arts or interdisciplinary domain
A
What factors influence the outcomes?
• Pre-tests and questionnaires at the beginning of the
intervention:
– Are differences related to baseline in achievement, creativity, to student
beliefs, to pedagogies, to socio-economic bacground, etc.?
• Observations and discussions within the network
B
B

44. School-based intervention
School year
A A A A
Primary school group - Ages 8-9. STEM, Arts or interdisciplinary domain
Secondary school group - Ages 13-14. STEM, Arts or interdisciplinary domain
A
What effects for the intervention?
• Measures after the intervention:
– Post-tests and -questionnaires
– Qualitative observations of pedagogies
– Interviews, ESM, executive functions
• Matched control group (with some kind of intervention as
well)
Control group
Control group
B
B
B
B

45. • What are the differences across country school networks in terms
of beliefs/attitudes and pedagogies (and baselines)?
• Is there change in student and teacher beliefs/attitudes and
practices?
• What effect on standardised achievement tests?
• What effect on standardised creativity tests?
• Any difference by socio-economic background, by achievement
level, by type of pedagogy, etc.?
• What are the perceived hurdles to implementing these
pedagogical approaches?
Questions we would like to answer

46. What domains (ideally)?
Domains
• Engineering
• Teacher
training/education
• Humanities / Social
science
• Innovation projects
• Arts
• Science
• Professional / technical
education
Levels
• 20 year olds
• First year students
• Last year of bachelor
Institutions
• Top tier
• Average
• Short cycles

47. • Questionnaires (beliefs/attitudes, pedagogy, anchoring
vignettes)
– Students
– Teachers
– Deans
• Domain-specific achievement tests (with embedded
questions on interest, curiosity, attitudes, and domain-specific
pedagogies)
• Domain-specific creativity tests
• Domain-specific critical thinking tests
• (Good domain general instruments?)
What instruments?

48. • Under what conditions could HEI be interested?
– Should we work with a few institutions?
– Could we have a large network of institutions?
– Could we build on existing initiatives?
• What funding model?
– Would HEIs cover their own costs? Who else could fund
their costs?
– Who could fund the international costs (OECD
development and coordination work)?
Size of pilot and funding model

49. other suggested ways
to support the agenda

50. www.oecd.org/edu/innovation
www.oecd.org/edu/universityfutures
www.oecd.org/edu/internationalisation
Stephan.Vincent-Lancrin@oecd.org
THANK YOU