What is evaluative bias, and what are we doing to reduce it from selection and development solutions? Lumina Learning CEO Dr Stewart Desson discusses his PhD research into this topic. Watch a video of Stewart giving this presentation: https://youtu.be/86OOo3Kw8m0 Read his article on the topic: https://luminalearning.com/evaluative-bias-minimisation

1. Exploring The Impact On
Criterion Validity Of Reducing
Evaluative Bias In Measuring
the Big Five
Dr. Stewart Desson,
CEO Lumina Learning
stewartdesson@luminalearning.com https://www.linkedin.com/in/stewartdesson/

2. Structure of presentation
• 3 Main Achievements
• Why is this research of interest?
• 5 Aims & 9 Research questions
• Highlights of the work
• Key Findings
• Limitations and Suggestions for Further Research
• Conclusions

3. Three Main Achievements
Lumina Spark personality instrument designed to:
1. Reduce Evaluative Bias
- Improves “user validity” (MacIver, Anderson, Costa & Evers, 2014)
- Values diversity
2. Measure Personality Adaptively and Maladaptively
- Without pathologising
- Discovered new blended personality trait
3. Improve our scientific understanding of performance at work
- Bandwidth fidelity debate explored
- Greater fidelity established
MacIver, R., Anderson, N., Costa, A. C., & Evers, A. (2014). Validity of Interpretation: A user validity
perspective beyond the test score. International Journal of Selection and Assessment, 22(2), 149-164.

4. Why is this research of interest?
• Contribution to science
- There is bias in current personality assessment tools
- Need to measure dysfunctional behaviour at work, without recourse to mental
health models
- Greater fidelity in understanding the relationship between ‘personality’ and
‘performance at work’
- Through greater fidelity, measuring a new ‘blended trait’
• Contribution to organisational psychology
- Enhanced interpersonal relationships
- More effective organisations, through better developed leaders
• This research is timely
- Changing culture puts more focus on the need to avoid bias in the workplace
- Need to value “deep diversity”

6. Five Aims
1. Measure both polarities of each Big Five dimensions
- as scalar opposites and independent constructs
- explore evaluative bias (Bäckström et al., 2014) and impact on user validity
2. Measure adaptive and maladaptive scales
- explore their relationship with highly dysfunctional constructs (Judge, Piccolo &
Kosalka, 2009) such as the "dark side" traits measured in the HDS (Hogan & Hogan,
1997; Hogan, Hogan & Kaiser, 2010)
3. Establish usefulness of the Spark model
- understand convergent and divergent validity
- locate scales in the periodic table of personality traits (Woods & Anderson, 2016)
4. Establish the criterion validity of the model
- positive and negative correlations between workplace performance and adaptive /
maladaptive traits at both poles of each Big Five construct
5. Explore bandwidth / fidelity debate
- empirically test higher and lower level models on the data gathered

7. Why measure both ends?
If you stick your head in the oven
and your feet in the freezer,
on average you'll be comfortable
Attributed to statistician Bruce Grossman
19 January 1960, The Guthrian (Guthrie Center, IA), pg. 2, col. 4:

8. Methodology
1.Five Aims
2.Nine Research Questions
3.Forty-Two Hypothesis tests

9. Evaluative Bias
in humour

10. Q. Why do we have Evaluative Bias in psychometrics?
A. One reason is item selection based on Factor Analysis
“When an evaluatively unbalanced set of descriptors
such as the Big Five adjectival markers (Goldberg,
1992) is subjected to a simple structure rotation
algorithm, the resulting factors almost invariably end
up contrasting positive versus negative descriptors
(Goldberg, 1992).”
Pettersson, E., Mendle, J., Turkheimer, E., Horn, E. E., Ford, D. C., Simms, L. J., & Clark, L. A. (2014). Do
maladaptive behaviors exist at one or both ends of personality traits? Psychological assessment, 26(2), 433.

11. Trait Descriptive Adjectives (TDA)
The adjectives that describe “Extraversion”:
• Extraverted
• Unrestrained
• Energetic
• Active
• Daring
• Vigorous
• Bold
• Verbal
• Assertive
• Talkative
Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological assessment, 4(1), 26.
The adjectives that describe “Introversion”:
• Introverted
• Unexcitable
• Inhibited
• Untalkative
• Timid
• Withdrawn
• Reserved
• Bashful
• Shy
• Quiet

12. TDA scored using a 5-point Likert format (N = 40)
1 - ‘Highly Undesirable’ through to 5 - ‘Highly Desirable’
“Please score each of these statements according
to how socially desirable you think they would be
in another person. Do not consider whether you
yourself possess them or not. Instead, just
intuitively rate them based on how much you
think others may find them desirable.”

13. TDA - Average Social Desirability Score Quantifiying
Evaluative Bias
Item Categories O C E A N
Plus (O+,C+,E+,A+,N-) 3.8 4.0 3.7 4.3 1.9
Minus (O-,C-,E-,A-,N+) 2.1 1.8 2.4 1.8 3.2
Difference is a Measure of
Evaluative Bias
1.8 2.2 1.3 2.6 -1.3

14. Evaluative Bias (Peabody, 1967) still an issue
• Research shows many Big Five models have an evaluative bias
(Bäckström, Björklund & Larsson, 2014)
• Negative impacts:
- on user validity
- on construct validity
• Questions:
- has the construct been measured in a comprehensive and non
biased way?
- how does the user feel about reading about aspects of their
personality in a biased and unbalanced personalised report?

15. A+ Agreeable A- Direct
E- IntrovertedE+ Extraverted
Measured
Intimate
Observing
Purposeful
Structured
Reliable
Accommodating
Collaborative
Empathetic
Tough
Competitive
Logical
Demonstrative
Takes Charge
Sociable
Adaptable
Flexible
Spontaneous
Cautious
Practical
Evidence Based
Radical
Conceptual
Imaginative
C+ Conscientious C- Flexible
O- PragmaticO+ Open
Agreeableness
Conscientiousness
Extraversion
Openness to Experience
Measuring the Big 5 Factors at ‘Both Ends’ – OCEAN – with Jungian Type Overlay
Even-Tempered
Optimistic
Resilient
Confident
N- Emotional Stability
Impassioned
Vigilant
Responsive
Modest
N+ Neurotic
Neuroticism
Feeling or Thinking
Judging or Perceiving
Extraversion or Introversion
Intuition or Sensing
ADAPTIVETRAITS
ADAPTIVETRAITS
Maladaptive Adaptive

16. A+ Agreeable to People Pleaser A- Direct to Aggressive
E- Introverted to PassiveE+ Extraverted to Overbearing
Measured
Intimate
Observing
Purposeful
Structured
Reliable
Accommodating
Collaborative
Empathetic
Tough
Competitive
Logical
Demonstrative
Takes Charge
Sociable
Adaptable
Flexible
Spontaneous
Cautious
Practical
Evidence Based
Radical
Conceptual
Imaginative
C+ Conscientious to Bureaucratic C- Flexible to Chaotic
O- Pragmatic to ClosedO+ Open to Dreamer
Agreeableness
Conscientiousness
Extraversion
Openness to Experience
Measuring the Big 5 Factors at ‘Both Ends’ – OCEAN – with Jungian Type Overlay
Even-Tempered
Optimistic
Resilient
Confident
N- Emotional Stability
Impassioned
Vigilant
Responsive
Modest
N+ Neurotic
Neuroticism
Feeling or Thinking
Judging or Perceiving
Extraversion or Introversion
Intuition or Sensing
MALADAPTIVETRAITS
MALADAPTIVETRAITS
Maladaptive Adaptive

17. I make new friends easily
Illustrative Example: Measuring Extraversion
Sometimes I listen too much and don’t
give my view
Extraversion
Introversion

18. I make new friends easily
Illustrative Example: Measuring Extraversion
Sometimes I talk too much
I choose my words carefully
before I speak
Sometimes I listen too much and don’t
give my view
Extraversion
Adaptive
Introversion
Maladaptive
Extraversion
Maladaptive
Introversion
Adaptive

19. 4 Points of the Personality Compass
• Aggregating the 4
measures reduces bias
AND
• Exploring the 4
measures supports
richer interpretation and
dialogue
Extraversion
Introversion
Adaptive
Maladaptive

20. Lumina Spark Reliability and Validity
• Lumina Spark Psychometric is reliable
- Cronbach’s Alphas
- Test-re-test
• Lumina Spark Psychometric is valid for use in the workplace
- Construct Validity
- Convergent and Divergent Validity
- Criterion Validity
- Consensual Validity
- User Validity increased
- Evaluative Bias decreased

21. IPIP NEO - Average Social Desirability Score Quantifiying
Evaluative Bias
Item Categories O C E A N
Plus (O+,C+,E+,A+,N-) 3.4 4.3 3.7 3.9 2.0
Minus (O-,C-,E-,A-,N+) 2.6 2.1 2.5 2.0 3.8
Difference is a Measure of
Evaluative Bias
0.7 2.2 1.2 1.8 -1.8

22. Lumina Spark - Average Social Desirability Score
Quantifiying Evaluative Bias
Item Categories O C E A N
Plus (O+,C+,E+,A+,N-) 3.2 3.3 3.0 3.1 1.9
Minus (O-,C-,E-,A-,N+) 2.6 2.6 2.6 2.8 4.3
Difference is a Measure of
Evaluative Bias
0.6 0.7 0.4 0.2 -2.4

23. Lumina Spark has Reduced Evaluative Bias
Hypothesis Outcome
H39: The TDA and IPIP-NEO measures of Openness will possess more
evaluative bias than the Spark measure of Openness
Supported
H40: The TDA and IPIP-NEO measures of Conscientiousness will possess
more evaluative bias than the Spark measure of Conscientiousness
Supported
H41: The TDA and IPIP-NEO measures of Extraversion will possess more
evaluative bias than the Spark measure of Extraversion
Supported
H42: The TDA and IPIP-NEO measures of Agreeableness will possess more
evaluative bias than the Spark measure of Agreeableness
Supported

24. Great Eight
• Kurz and Bartram (2002) defined the Great Eight
• Kurz (2003) developed variation of Great Eight with more applied
titles
• Bartram (2005) meta analysis showed OPQ scale mappings
- Double weight for top scales
- Single weight for two further scales
• Saville, Maclver, & Kurz (2009) repeated Bartram (2005) method
- showed scale mappings of OPQ, Saville Wave Professional, Wave Focus,
SPQ, NEO, HPI, 16PF onto Great Eight
• This study applies Bartram (2005) and Saville et al. (2009) approach
to Lumina Spark

26. Great Eight Items
Analysing Situations:
Demonstrating Analytical Thinking; Solving Complex Problems; Critically Evaluating Information
Creating Concepts:
Being Creative and Innovating; Thinking Strategically; Driving Organisational Change
Relating to People:
Displaying Good Interpersonal Skills; Exercising Active Listening; Communicating Effectively
Controlling Resources:
Leading and Directing Others; Managing People and Resources Effectively; Being Decisive; Making Sound Judgments
Respecting People:
Giving Support; Building Team Spirit; Showing Compassion and Being Approachable
Adapting to Demands:
Showing Composure; Working Effectively Under Pressure; Dealing with Ambiguity
Delivering Results:
Planning and Organising Efficiently; Working Diligently; Completing Tasks on Time
Driving Performance:
Having Career Ambition; Setting and Achieving Ambitious Work Objectives; Showing Business Acumen

27. 14 “Point to Point” Spark to Great Eight Hypotheses Created
Analysing
Situations
Creating
Concepts
Relatingto
People
Controlling
Resources
Respecting
People
Adaptingto
Demands
Delivering
Results
Driving
Performance
SumofAll
Eight
O+ Big Picture Thinking .09 .22** -.07 .03 -.04 .01 -.12* .03 .03
O+ Overextended .10 .16* -.08 -.03 -.10 -.04 -.19** .03 -.02
O- Down to Earth .02 -.25** -.08 -.06 -.04 -.06 .13* -.06 -.08
O- Overextended -.03 -.26** .01 -.03 .00 -.01 .13* .01 -.04
C+ Discipline Driven .07 -.06 -.02 .04 .06 .08 .28** .08 .09
C+ Overextended .02 -.16* -.08 -.08 -.03 -.15* .11 -.02 -.07
C- Inspiration Driven -.13* .06 -.02 -.05 -.06 -.08 -.11 -.11 -.09
C- Overextended -.08 -.01 -.02 -.12 -.04 -.17** -.25** -.12* -.14*
E+ Extarversion -.05 .12 .21** .06 .11 .10 .03 .08 .12*
E+ Overextended .02 .09 .09 -.04 -.01 -.01 -.10 .10 .03
E- Introversion .10 -.10 -.18** -.02 -.07 -.09 .03 -.06 -.07
E- Overextended .09 -.10 -.20** -.06 -.14* -.14* .04 -.05 -.10
A+ People Focused -.09 .04 .19** -.05 .20** -.03 -.01 -.06 .03
A+ Overextended -.09 -.08 .01 -.16* .04 -.13* -.10 -.13* -.12
A- Outcome Focused .12* .02 -.14* .11 -.22** .05 .04 .12* .03
A- Overextended .13* -.10 -.22** .02 -.19** -.05 .02 .05 -.06
N+ Risk Reactor Overextend -.05 -.20** -.20** -.17** -.15* -.35** -.13* -.13* -.25**
N- Reward Reactor .06 .22** .13* .21** .07 .25** .07 .13* .20**
Note: N = 254; * p<0.05 ** p<0.01

28. Analysing
Situations
Creating
Concepts
Relatingto
People
Controlling
Resources
Respecting
People
Adaptingto
Demands
Delivering
Results
Driving
Performance
SumofAll
Eight
O+ Big Picture Thinking .22** -.12*
O+ Overextended .16* -.19**
O- Down to Earth -.25** .13*
O- Overextended -.26** .13*
C+ Discipline Driven .28**
C+ Overextended -.16* -.15*
C- Inspiration Driven
C- Overextended -.17** -.25** -.12* -.14*
E+ Extarversion .21** .12*
E+ Overextended
E- Introversion -.18**
E- Overextended -.20** -.14* -.14*
A+ People Focused .19** .20**
A+ Overextended -.16* -.13* -.13*
A- Outcome Focused .12* -.14* -.22** .12*
A- Overextended .13* -.22** -.19**
N+ Risk Reactor Overextend -.20** -.20** -.17** -.15* -.35** -.13* -.13* -.25**
N- Reward Reactor .22** .13* .21** .25** .20**
12 out of 14 hypotheses supported (shaded green)
2 out of 14 hypotheses not supported (shaded red)
8 additional interesting and important findings (shaded orange)

29. Nine Research Questions
1. Is the proposed Spark model of personality compatible with the Big Five factor
structure?
2. Where do the Spark scales sit in the personality periodic table (Woods & Anderson,
2016) of blended Big Five factors?
3. Do the Spark adaptive scales correlate more highly than the Spark maladaptive scales,
with other “bright side” Big Five traits?
4. Do the Spark’s maladaptive scales correlate more highly than the Spark adaptive scales,
with the HDS “dark side” traits?
5. What evidence is there to support the conceptualisation of Spark maladaptive scales as
overplayed / overextended / extreme ends of the "bright side" Big Five traits?
6. How well does the Spark comply with a priori hypothesized criterion validity
relationships with the Great Eight competency model?
7. Is there a differential pattern of criterion validities between the Spark adaptive and
maladaptive scales and if so, what can be learnt from this?
8. Compared to the Spark five-dimensional bandwidth approach, can the higher fidelity
Spark eighteen scales explain more of the variance in the personality criterion
relationship?
9. Can the Spark reduce the impact of evaluative bias in the Big Five?

30. Limitations
• Weird sample - “Western, educated, industrialised,
rich, democratic countries”
- Henrich, Heine and Lorentzian (2010)
• All criterion measures are 360-degree observations
• Causal inference?

31. Suggestions for further research
• Replicate the confirmatory factor analysis on new samples
• Explore if certain maladaptive traits may be
- detrimental to performance in many roles
- essential to performance in other specific roles e.g. a
soldier about to go into combat
• Can Neuroticism be expressed adaptively?
• Could Emotional Stability be expressed in a maladaptive
form?

32. Suggestions for further research
• Build a maladaptive periodic table
- measuring blends of opposite polarities
- both adaptively and maladaptively
• Use Lumina Spark as an instrument for dissecting and
assessing the construct validity of existing
instruments
- in a way that other instruments may not be able to
- as a compliment to the TDA periodic table analysis

33. Five Conclusions
1. Evaluative bias is (increasingly) an organisational issue
2. Issue addressed by measuring both ends of the Big Five dimensions
3. Conceptualising maladaptive traits as the more extreme ends of the Big Five (“too
much of a good thing”)
- Helps explain enablers and blockers to performance at work
- Avoids risk of pathologising people
4. Spark approach does reduce evaluative bias
- Compared to TDA and IPIP-NEO
- Enables test user to crack open their Big Five dimensions and see their adaptive and
maladaptive traits at both ends of the polarities
5. Spark has explored elements of the periodic table (Woods & Anderson, 2016) less
well researched by other top psychometrics
- Explored element not yet researched elsewhere
- Blend that leads with Conscientiousness and is supported by Neuroticism
stewartdesson@luminalearning.com
https://www.linkedin.com/in/stewartdesson/