SlideShare ist ein Scribd-Unternehmen logo
1 von 9
Downloaden Sie, um offline zu lesen
Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718
1877-0428 © 2013 The Authors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license.
Selection and peer-review under responsibility of Cognitive-counselling, research and conference services (c-crcs).
doi:10.1016/j.sbspro.2014.01.1221
ScienceDirect
International Conference on Education & Educational Psychology 2013 (ICEEPSY 2013)
The Development of the Heuristics and Biases Scale (HBS)
Marcin Skladab
*,
Rene Diekstrab
a
Utrecht University: University College Roosevelt, Lange Noordstraat 1 4330AB Middelburg, The Netherlands
b
The Hague Uuniversity of Applied Sciences,Johanna Westerdijkplein 75, 2521EN Den Haag The Netherlands
Abstract
Problem Statement:
There is no comprehensive tool capturing general vulnerability to biases caused by the use of heuristics. Existing tools focus
only on one specific bias or on personality traits.
Research Questions:
Can general vulnerability to heuristic thinking be assessed and what are the sub-dimensions of this construct? Can
undergraduate students be successfully involved in the research process?
Purpose of the Study:
To demonstrate the results of an educational experiment in which undergraduate students are involved in the first stage of
development of the Heuristics and Biases Scale (HBS).
Research Methods:
After getting acquainted with the underlying theory, students chose one specific bias or heuristic, investigated related results
of the experiments and paradigms. At the later stage, under supervision, students developed items intended to capture the
chosen bias. Finally, positively evaluated items were combined together and piloted. The psychometrical properties of the
items and course outcomes were assessed.
Findings:
Developed items formed scales with satisfactory reliability. Course received positive student evaluations, and the assessment
indicated that the majority of students achieved intended learning outcomes.
Conclusions:
The study indicates that it is possible to develop a psychometrically sound assessment to measure vulnerability to a range of
common cognitive biases. Moreover, it is also possible to successfully involve undergraduate students in the development of a
psychometrical tool.
Acknowledgments: Authors would like to thank the Students of University College Roosevelt contributing to this work as their
Independent Research Projects or as a part of Social Psychology or Statistics Course.
© 2013 The Authors. Published by Elsevier Ltd.
Selection and peer-review under responsibility of Dr Zafer Bekirogullari.
* Corresponding author. Phone: +31 118 655 528 fax: +31 (0)118 655 50.
E-mail address: m.sklad@ucr.nl
Available online at www.sciencedirect.com
© 2013 The Authors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license.
Selection and peer-review under responsibility of Cognitive-counselling, research and conference services (c-crcs).
711
Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718
Key Words: Cognitive Heuristics Biases Education Psychometric Scale
1. Introduction
According to classic normative model of rational choice, when a choice has to be made, the expected utility of
each option can be calculated given the probabilities and utilities of all possible outcomes of each option as the
weighted average of the utilities of the outcomes. A rational actor making a decision should choose the option
maximizing the expected utility (Neumann & Morgenstern, 1953). Obviously, excluding simple gambling
problems, complete rationality defined in the above way is unachievable for human beings. Firstly, in case of
most real life decisions the list of potential consequences of each option is very long, if not infinite, with many
unknowns. Secondly, probabilities of various outcomes are usually not known and at best they can be only
roughly estimated. Herbert Simon (1957) coined the term bounded rationality to describe limitations set by
human’s computational and deliberative capacity on rationality of decision making. He argued that we seek to
find satisfying solutions rather than optimal. Social psychologists went even further claiming that people often
behave like cognitive misers trying to save cognitive resources by using as little information as possible to reach
a conclusion and only if motivated opt for more elaborated cognitive strategies (Fiske and Taylor, 1991).
The distinction between deliberate, strenuous, conscious mental processes and automatic processes requiring
much less cognitive resources has been well described by a group of psychological theories described collectively
as dual process theory (Evans, 2011). Independently on whether there are two discrete modes of reasoning
(Sloman, 1996) or a continuum between implicit and explicit processes (Osman, 2004), there might be individual
differences in relying on these two modes of reasoning (Stanovich & West, 2000). These differences might for
instance be due to differences in cognitive abilities (Stanovich & West, 2000, Sattler, Hellix, & Neher, 1970) or
to differences in preferred cognitive style and motivation (Fiske and Taylor, 1991; Petty, DeMarree, Brinol,
Horcajo, & Strathman, 2008). Kahneman (2011) refers to dual-processing as two system thinking, with System 1
corresponding to intuitive unintentional, involuntary, and effortless, automatic thinking, and System 2 is
responsible for reflective, controlled, conscious effort consuming large amount of mental energy and being
limited by capacity of working memory.
According to Kahneman, errors in judgment produced by System 1 are not random. Theory proposed by
Tversky & Kahneman (1974) suggested that the process guiding intuitive judgment is not just an incomplete
version of the rational model of judgment that could be executed by System 2. They postulated that judgments
made by System 1 are qualitatively different, since they are made using different algorithms than judgments
made by System 2. System 1 substitutes harder questions by simpler in order to produce solutions. In other
words, System 1 relies on a limited number of heuristics - highly efficient mental shortcuts, which reduce
complex tasks of assessing probabilities and predicting values to simple judgmental operations. Even though the
use of heuristics most of the time provides a satisfying answer, it also leads to systematic biases. The nonrandom
nature of the departures from the normative rational theory demonstrated in many experiments has been treated
as an indicator of the existence of underlying heuristics producing these errors (Gilovich & Griffin, 2002).
According to Kahneman (2011), there is also a certain degree of control over the mode of decision making.
While System 1 is usually a default way of processing, one can make a decision to switch to System 2 in attempt
to correct for these biases. This also suggests that there might be individual differences in the extent to which we
rely on System 1 and System 2, due to personal preferences.
2. Problem
Even though System 1 judgments based on heuristics provide answers of sufficient quality for most everyday
decisions, they may also lead to serious errors. Unfortunately, there is ample evidence that people rely on
712 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718
heuristics also in case of decisions having extremely significant consequences, for instance: large investments
(O’Donnell & Schultz, 2005), court sentences (e.g. Danziger, Levar and Avnaim-Pesso, 2011; Sigall and
Ostrove, 1975), and medical decisions (e.g. Marewski & Gigerenzer, 2012). In this context, it is really relevant to
be able to evaluate an individual’s susceptibility to cognitive biases. To authors best knowledge there is no
behaviorally based measurement tool assessing this susceptibility. The closest existing tools are IQ tests
measuring cognitive capacity and a self report tool measuring self reported motivation (Cacioppo & Petty, 1982).
Therefore, the purpose of this Study is: 1) to establish whether different biases indeed reflect consistent
individual differences in general tendency to rely on System 1, or each particular heuristic or bias is fairly
independent, and 2) to determine whether it is possible to develop a measurement tool which measures
individuals’ vulnerability to cognitive biases. Originally, Kahneman and Tversky (1974) proposed three main
heuristics responsible for the systematic bias of judgment: availability heuristic, representativeness heuristic, and
anchoring and adjustment heuristic. Later, in order to accommodate for the unquestionable pivotal role affective
judgment plays in decision making (e.g. Damasio, 1994), affect heuristic (Slovic at al., 2002) has been added to
the list and incorporated to the theory (Kahneman 2011).
Kahneman (2011) postulated that several cognitive biases can be attributed to each heuristic. For instance,
availability heuristic, which allows the actor to use ease of retrieval to estimate the frequency of a class or the
probability of an event, can lead to availability bias and to the phenomenon of rare events bias. In case of both
biases, the probability of events which can be easily retrieved from the memory is systematically overestimated.
Therefore, recent and vivid events may have a disproportionate impact on decisions.
In the representativeness heuristic, the resemblance of an object to the prototype serves as a proxy for
estimating probabilities. The use of this heuristic can produce biases like: conjunction fallacy, base rate
negligence, and gamblers fallacy. In all of these biases, estimation of the likelihood is biased because people to
large degree base their judgment on information about the similarity of the evaluated event to the prototype.
Opposite to formal logic, in case of conjunction fallacy, the chance of occurrence of a target with one unlikely
feature is rated lower than the chance of occurrence of a target with the same unlikely feature plus one likely
feature (Tversky & Kahneman, 1983). For instance, a white crow with two wings is rated more probable than a
white crow. In base rate negligence (e.g. Kahneman & Tversky 1972; 1983Bar-Hillel, 1980), participants in
experiments overestimate the chance that a target belongs to a relatively rare category if it, he or she resembles a
prototypical member of this category. For instance, somebody who looks like a librarian is most likely a librarian.
Gamblers fallacy makes people believe that an independent single random outcome will depend on previous
outcomes in order to make series fit the image of random distribution. For instance, after three tails one expects
head.
Anchoring and adjustment heuristic allows us to estimate a value starting from a known position and adjusting
the value based on facts that come to mind. Unfortunately, if the original anchor is off, adjustment is usually not
sufficiently big, which leads to a bias.
Reliance on feelings associated with these affective responses in judgment has been defined as the affect
heuristic. Almost any potential characteristic of a person or object has a certain affective valence. Thanks to
affect heuristic, when people need to evaluate any property of the target, they can substitute the property in
question with general affective evaluation of the target (Slovic, et al. 2002). The most well known bias related to
this heuristic is halo effect (Thorndike, 1920; Asch, 1946). In the halo effect, one vivid characteristic of a person
determines how this person is judged on all unrelated characteristics. For example, a beautiful person is perceived
as kind. According to Tversky (2011), the use of judgmental heuristics is synonymous with automatic System 1
processing. For, heuristics replace estimating characteristics (like probability) requiring elaborate formal thinking
by estimates of characteristics automatically and effortlessly provided by System 1: similarity, ease of recall and
affective value.
713
Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718
If the assumption that cognitive biases are a product of relying on an automatic processing style is correct, it
should be possible to evaluate the general reliance on automatic system 1 processing by the amount and intensity
of judgmental biases demonstrated by an individual.
In the presented research, the initial development of the items constituting cognitive bias scales has been
carried out as a part of problem based learning (PBL, Barrows 1996) by students in the intermediate level
psychology class. Aim was to test whether undergraduate university college students can be successfully
involved in academic research this way, to the benefit of both students and research.
3. Research Method
As was mentioned before, the purpose of the study was: 1) to develop and pilot the scales to assess
individual’s reliance on heuristics and vulnerability to cognitive biases and 2) to pilot the procedure of involving
university college students in the development and to enhance their learning through PBL (Barrows 1996). In this
approach, students start with the problem, then work in a peer group to explain the problem and to construct the
needed knowledge. Meanwhile, the teacher provides the framework for it. Later, the students work on their
individual solution and then regroup at the end to share their findings. To reduce issues related to potential excess
of cognitive load in the initial stage of PBL (Sweller, 1988), in this project, students initially got acquainted with
the general underlying theory and heuristics and biases approach (Kahneman, 2011) and chose one specific bias
or heuristic to study. In the next stage, they wrote a short essay summarizing the processes postulated to be
involved in the bias. Moreover they investigated related results of experiments and experimental paradigms used
to demonstrate the chosen bias. At this time, they were also confronted with the problem of creating a test item
taping into the selected heuristic or bias, and they attempted to produce an initial solution. Next, based on their
choices, they were assigned to workgroups of three to four people and worked together towards the goal of
developing a handful of fully functional items. This process, guided by the framework provided by the teacher,
took several stages in which students worked individually and in the workgroups to polish and exchange
feedback on their items under supervision. During these stages, students were also given the necessary
psychometrical background. Finally, after a pilot of studying individual items, students in each group selected the
best items intended to capture the chosen bias. Finally, items positively evaluated on their face validity were
combined with other items developed by researchers and piloted. Finally, an assessment was carried out of both
the enhancement of learning by students as well as the developed scales.
The problem based learning project was evaluated based on self-reported learning outcomes. At the end of the
project, besides a regular course evaluation form, students filled a short self report questionnaire. In this self
report questionnaire, they could indicate if the HBS project reached its learning objectives and they could
indicate if, given the opportunity, they would like to develop their work on HBS further. Psychometrical
properties of the items were assessed based on the results of the pilot.
The group of students involved in the development of the scales consisted of a regular class of 25 English
speaking students of a Dutch residential university college. The students were mostly in their second year of
undergraduate honors program. Before being elected to do the second year level intermediate psychology course,
they had already successfully completed an introductory psychology course and a methodology and statistics for
social sciences course.
The pilot questionnaire including the items based on the students’ work contained three parts. The first part
concerned the demographical background of the participant. The second part included 59 heuristics and biases-
related items. Each item consisted of one up to three interconnected questions, which were combined to produce
one unique item score. The biases-related questions covered the following biases: Hindsight bias, Conjunction
fallacy, Planning fallacy, Halo effect, Phenomenon of rare Events, Overconfidence bias, and Susceptibility to
714 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718
priming. Items concerning Gamblers fallacy, Availability bias, Anchoring bias and Sunk cost fallacy were
abandoned due to dissatisfactory properties.
After the first small scale pilot, an adjusted final questionnaire was distributed to a convenience sample of
respondents. The questionnaire was distributed in several clusters in order to diversify the sample. The clusters
included: high school students, elderly house residents, students of University College, relatives and
acquaintances of students participating in the class.
The questionnaire was either handed out in a paper form and collected later, or filled under supervision on the
spot, or it was delivered and returned by email. All participants received the questionnaire with the explicit
instruction to fill it in individually without asking for help. Completing the questionnaire took approximately 45
minutes on average.
Out of the 163 questionnaires that were distributed, 128 complete samples were collected. In the sample,
gender was not equally distributed, for the sample consisted in 63.5% of female participants. The average age of
the participants was equal to 25.91 years (S=13.44) and ranged from 15 to 60 years. The majority of the sample
(65.7%) consisted of students.
The analysis will be presented in two parts. The first part describes the results of the questionnaire considering
students’ learning experience. The latter part concerns initial psychometrical evaluation of the produced items by
analysis of their distribution, intercorrelations, reliability, and exploratory factor analysis.
4. Findings
4.1. Learning Experience
Evaluation of learning objectives of the course in the self report (see Table 1) indicated that, in students
reports, the project met its learning objectives. Each item concerning a learning objective received a statistically
significant (p<.005) positive answer on average. Learning from peers also received significantly (p=.011)
positive evaluation. Among students, 40% agreed that given the opportunity they would like to develop their
work further.
Table 1. Students Evaluations of course objectives (1- strongly disagree, 7-strongly agree)
Mean Std. Deviation
learned about biases and heuristics influencing human decision making 5.80 1.24
learned how cognitive biases manifest themselves in practice 5.50 1.05
I learned how to communicate about cognitive biases and heuristics 5.00 1.26
I gained a better understanding of limitations to our rationality 5.85 0.88
I learned a lot from my fellow students (discussing the items together, listening to the
presentations and reviewing their papers)
4.85 1.35
I gained interest in our cognitive processes 5.50 1.10
Note: N=20,
715
Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718
4.2. Heuristics and biases tool
From the initial pool of 39 items with satisfactory face validity, items were selected for the factor analysis
only if there were at least two of them concerning the same cognitive bias and only if they correlated at least .25
with at least one other item, suggesting reasonable factorability. This procedure left 23 items related to
vulnerability to priming, hindsight bias, conjunction fallacy, planning fallacy, halo effect and phenomenon of rare
events.
Screening revealed that the data was suitable for factor analysis. The minimum amount of data for factor
analysis was satisfied, with a final sample size of 128 (using list-wise deletion), providing a ratio of over 5 cases
per variable. The Kaiser-Meyer-Olkin measure of sampling adequacy was .53, which renders the data acceptable
for factor analysis, and Bartlett’s test of sphericity was significant (χ2
(253) = 766.50, p < .001). For all items,
measures of sampling adequacy and the communalities were all acceptable above .5 and .4 respectively (see
Table 2), later confirming that each item shared some common variance with other items. Due to exploratory
nature of the study, principal components analysis has been chosen to identify factors underlying the set of items.
One (System 1 thinking), two (heuristics), and six (biases) factors solutions were theoretically supported. Initial
eigen values indicated that the first factors explained 12%, 10.5%, 9.5%, 8.5%, 7.5%, 6.2%, 5.7% of the variance
respectively. The analysis of eigen values on the scree plot indicated leveling off after five, seven factors; and 11
factors. The 11 factor solution was rejected because last factors yielded eigen values below 1.0 and the small
number of variables per factor. The five factor solution failed to assign primary loadings to all items. The seven
factor solution, which explained 59.7% of the variance has been chosen as a result. There was no difference
between the orthogonal and non orthogonal rotation in terms of assignment of items to components and of clarity
of the assignment, thus varimax rotation has been used for the final solution. Factors extracted by rotated
solutions had roughly the same explanatory power with the first accounting for 9.8% of total variance and the last
for 7.2% of variance. All items had primary factor loading or above .4 and no item had a cross-loading above .4.
716 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718
Table 2. Factor loadings and communalities based on a principal components analysis with varimax rotation for 23 items included in the
HBS questionnaire (N = 228)
Communality 1
CF
2
PF1
3
HB
4
PRE
5
HE
6
PF2
7
PRI
Priming 1 .621 .778
Priming 2 .704 .796
Hindsight Bias 2 .604 .739
Hindsight Bias 3 .613 .768
Hindsight Bias 4 .458 .635
Hindsight Bias 5 .418 .466
Conjunction Fallacy 1 .622 .723
Conjunction Fallacy 2 .407 ,557
Conjunction Fallacy 3 ,458 .614
Conjunction Fallacy 4 .545 .710
Conjunction Fallacy 5 .535 .685
Planning Fallacy 1 .687 .820
Planning Fallacy 2 .481 .631
Planning Fallacy 3 .682 .796
Planning Fallacy 4 .642 .553
Planning Fallacy 6 .768 .848
Planning Fallacy 7 .693 .813
Halo Effect 1 .508 .615
Halo Effect 2 .476 .589
Halo_Effect 3 .446 .656
Halo Effect 4 .582 .721
Phenomenon of rare events 3 .873 .923
Phenomenon of rare events 4 .918 .942
Note. Factor loadings < .4 are suppressed. CF-Conjunction Fallacy, PF-Planning fallacy, HB- Hindsight bias,
PRE- phenomenon of rare events, HE-halo effect, PRI-Priming.
The analysis revealed a factor structure closely matching the pattern of biases addressed by the items. Thus,
the 7 factors were given labels corresponding to the biases the items were intended to measure. The items related
to planning fallacy formed two distinct factors. The first planning fallacy related to fallacy concerning
underestimating the amount of time and work endeavors consume. The second related to the amount of money
one spends on it. Internal consistency for each of the scales was examined using Cronbach’s alpha, and median
inter-item correlation. The alphas were moderate: .60 for priming (2 items, r=.46), .62 for Hindsight Bias (5
items, r=.28), .60 for Conjunction Fallacy (4 items, r=.26), .69 for Time Planning Fallacy (3 items, r=.39), . 67
for Money Planning Fallacy (3 items, r=.47) ,. 92 for Phenomenon of rare events (2 items, r=.80). No increases in
alpha for any of the scales could have been achieved by eliminating any item. The seven factor scores were
computed as average of corresponding items. The analysis of correlations between the factors revealed two
717
Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718
statistically significant (p<.05) correlations: The first between two types of planning fallacy (r=.28), the second
between priming and halo effect (r=-.20).
5. Conclusion
The study had twofold purposes. First, to test whether it is possible to involve undergraduate students in the
development of a psychometrical tool with benefits to them and the product. The second purpose was to initiate
the construction of a set of scales allowing for estimation of individuals’ vulnerability to biases and for estimation
of the use of heuristical thinking.
The results seem to confirm the educational benefit of the project. However, since the conclusion concerning
educational benefit is based on self reports and a small sample, rather than a controlled experiment, it should be
taken with caution.
The pilot of the measurement tool revealed several interesting results. Firstly, the items addressing biases were
able to form internally consistent scales. Medians of inter item-correlations for all factors exceeded .25
suggesting good reliability of individual items (Robinson, Shaver, & Wrightsman, 1991). It also suggests that if
the number of items in each scale would be increased to a reasonable amount, all scales would reach a
satisfactory alpha (Carmines & Zeller, 1979). Therefore, it can be concluded that biases can be reliably measured
using the design. It is also worth noting that there were no positive correlations between clusters of items
addressing different biases. This indicates that there is no support for the existence of underlying common source
of different biases, like habitual use of System 1 or heuristic (Kahneman, 2011). However, this may be partially
the outcome of the fact that the used sample of items by no means represents well the complete spectrum of
known cognitive biases.
In summary, the results show that developing a scale addressing vulnerability to cognitive biases is possible. It
can be done by extending the current study in three ways: in terms of the sample size, the number of biases
addressed, and the number of items per bias.
6. References
Asch, S.E. (1946) Forming impressions of personality. The Journal of Abnormal and Social Psychology. 41(3),
Jul 1946, pp. 258-290.
Bar-Hillel, M. (1980). The Base Rate Fallacy in Probability Judgements. Acta Psychologica, 44, 211-233.
Barrows, H. S., (1996). Problem-based learning in medicine and beyond: A brief overview. New Directions for
Teaching and Learning 68, 3–12. doi:10.1002/tl.37219966804
Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology,
42(1), 116-131.doi:10.1037/0022-3514.42.1.116
Carmines, E. G. & Zeller, R. A. (1979) Reliability and validity assessment. Thousand Oaks, CA: Sage
Publications. Danziger, S., Levav, J., & Avnaim-Pesso, L. (2011). Extraneous factors in judicial decisions.
Proceedings of the National Academy of Sciences, 108(17), 6889-6892.
Damasio, A. R. (1994). Descartes’ error: Emotion, reason, and the human brain. New York: Avon.
Evans J. S. B. T., (2011). Dual-Process Theories Of Reasoning: Contemporary Issues And Developmental
Applications Developmental Review, 31 (2-3), 86–102.
Fiske, S.T., & Taylor, S.E. (1991). Social cognition (2nd ed.). New York: McGraw-Hill.
Gilovich, T., & Griffin, D.W. (2002). Introduction – Heuristics and biases – Then and now. In T. Gilovich, D.W.
Griffin, & D. Kahneman (Eds.), Heuristics and Biases: The Psychology of Intuitive Judgment (pp. 1-18).
Cambridge: Cambridge University Press.
Kahneman, D. (2011). Thinking, Fast and Slow. London: Penguin Books.
Kahneman, D., Tversky, A. (1972). On prediction and judgment. Oregon Research Institute Bulletin 12 (4).
O'Donnell, E., & Schultz. J. (2005). The halo effect in business risk audits: Can strategic risk assessment bias
auditor judgment about accounting details? The Accounting Review 80(3), 921-38.
718 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718
Marewski J.N., Gigerenzer G., (2012). Heuristic Decision Making in Medicine. Dialogues Clinical Neuroscience
14, 77–99.
Von Neumann, J., & Morgenstern, O. (1953). Theory of Games and Economic Behavior. New York: John Wiley
and Sons.
Osman, M. (2004). An evaluation of dual-process theories of reasoning. Psychonomic Bulletin & Review 11 (6):
988–1010.
Pacini, R., & Epstein, S. (1999). The relation of rational and experiential information processing styles to
personality, basic beliefs, and the ratio-bias phenomenon. Journal of Personality and Social Psychology, 76(6),
972-987. doi: 10.1037/0022-3514.76.6.972
Petty, R. E., DeMarree, K. G., Brinol, P., Horcajo, J., & Strathman, A. J. (2008). Need for cognition can magnify
or attenuate priming effects in social judgment. Personality and Social Psychology Bulletin, 34, 900-912
Robinson, J. P., Shaver, P. R., & Wrightsman, L. S. (1991). Criteria for scale selection and evaluation. In J. P.
Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.). Measures of personality and social psychological attitudes
(pp. 1-16) New York: Academic Press.
Sattler, J. M., Hillex, W. A., & Neher, L. A. (1970). Halo effect in examiner scoring of intelligence test
responses. Journal of Consulting and Clinical Psychology, 34(2), 172-176.
Sigall, H., Ostrove, N. (1975). Beautiful but dangerous: effects of offender attractiveness and nature of the crime
on juridic judgement. Journal of Personality and Social Psychology. 31(3), 410-414.
Simon, H. (1957). "A Behavioral Model of Rational Choice", in Models of Man, Social and Rational:
Mathematical Essays on Rational Human Behavior in a Social Setting. New York: Wiley.
Sloman S.A. (1996) The empirical case for two systems of reasoning. Psychological Bulletin, 119, 3-22.
Slovic, P., Finucane, M., Peters, E., MacGregor D. G., (2002). The Affect Heuristic. In Thomas Gilovich, Dale
Griffin, Daniel Kahneman. Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge University
Press. pp. 397–420.
Stanovich, K. E., & West, R. F. (2000). Individual Differences in Reasoning: Implications for the Rationality
Debate. Behavioral and Brain Sciences, 23, 645-726.
Sweller, J., (1988). Cognitive load during problem solving: Effects on learning, Cognitive Science, 12, 257-285.
Thorndike, E.L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4(1), 25-29.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124-
1131.
Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in
probability judgment. Psychological Review, 90(4), 293-315.

Weitere ähnliche Inhalte

Ähnlich wie The Development of the Heuristics and Biases Scale.pdf

Top of FormPresentation Understanding Research MethodologyIn.docx
Top of FormPresentation Understanding Research MethodologyIn.docxTop of FormPresentation Understanding Research MethodologyIn.docx
Top of FormPresentation Understanding Research MethodologyIn.docxedwardmarivel
 
Knowledge complex Systems, decisions, uncertaintly risk
Knowledge complex Systems, decisions, uncertaintly riskKnowledge complex Systems, decisions, uncertaintly risk
Knowledge complex Systems, decisions, uncertaintly riskFLACSO
 
1609182308Mediation.pdf
1609182308Mediation.pdf1609182308Mediation.pdf
1609182308Mediation.pdfJoshuaLau29
 
The role of social heuristics in deliberative outcomes | María G. Navarro
The role of social heuristics in deliberative outcomes | María G. NavarroThe role of social heuristics in deliberative outcomes | María G. Navarro
The role of social heuristics in deliberative outcomes | María G. NavarroMaría G. Navarro
 
The experiential utility: how behavioural economics can help hci to define qu...
The experiential utility: how behavioural economics can help hci to define qu...The experiential utility: how behavioural economics can help hci to define qu...
The experiential utility: how behavioural economics can help hci to define qu...Stefano Bussolon
 
Ak park zak_2007
Ak park zak_2007Ak park zak_2007
Ak park zak_2007Jang Park
 
Relevance of experimental design
 Relevance of experimental design Relevance of experimental design
Relevance of experimental designAlexander Decker
 
Relevance of experimental design
 Relevance of experimental design Relevance of experimental design
Relevance of experimental designAlexander Decker
 
Relevance of experimental design
 Relevance of experimental design Relevance of experimental design
Relevance of experimental designAlexander Decker
 
Teresa Swain 1 postsReTopic 2 DQ 2Drawing on your knowled.docx
Teresa Swain 1 postsReTopic 2 DQ 2Drawing on your knowled.docxTeresa Swain 1 postsReTopic 2 DQ 2Drawing on your knowled.docx
Teresa Swain 1 postsReTopic 2 DQ 2Drawing on your knowled.docxmehek4
 
The role of theory in research division for postgraduate studies
The role of theory in research division for postgraduate studiesThe role of theory in research division for postgraduate studies
The role of theory in research division for postgraduate studiespriyankanema9
 
Rorschach Measures Of Cognition And Social Functioning Essay
Rorschach Measures Of Cognition And Social Functioning EssayRorschach Measures Of Cognition And Social Functioning Essay
Rorschach Measures Of Cognition And Social Functioning EssayKatherine Alexander
 
The role of theory in research
The role of theory in researchThe role of theory in research
The role of theory in researchJaseme_Otoyo
 
Social_Research_Methods_on25Feb2016.pdf
Social_Research_Methods_on25Feb2016.pdfSocial_Research_Methods_on25Feb2016.pdf
Social_Research_Methods_on25Feb2016.pdfprinterwalasales1
 

Ähnlich wie The Development of the Heuristics and Biases Scale.pdf (20)

Top of FormPresentation Understanding Research MethodologyIn.docx
Top of FormPresentation Understanding Research MethodologyIn.docxTop of FormPresentation Understanding Research MethodologyIn.docx
Top of FormPresentation Understanding Research MethodologyIn.docx
 
Knowledge complex Systems, decisions, uncertaintly risk
Knowledge complex Systems, decisions, uncertaintly riskKnowledge complex Systems, decisions, uncertaintly risk
Knowledge complex Systems, decisions, uncertaintly risk
 
1609182308Mediation.pdf
1609182308Mediation.pdf1609182308Mediation.pdf
1609182308Mediation.pdf
 
The role of social heuristics in deliberative outcomes | María G. Navarro
The role of social heuristics in deliberative outcomes | María G. NavarroThe role of social heuristics in deliberative outcomes | María G. Navarro
The role of social heuristics in deliberative outcomes | María G. Navarro
 
The experiential utility: how behavioural economics can help hci to define qu...
The experiential utility: how behavioural economics can help hci to define qu...The experiential utility: how behavioural economics can help hci to define qu...
The experiential utility: how behavioural economics can help hci to define qu...
 
Neuroeconomics
NeuroeconomicsNeuroeconomics
Neuroeconomics
 
An application of neutrosophic logic in the confirmatory data analysis of the...
An application of neutrosophic logic in the confirmatory data analysis of the...An application of neutrosophic logic in the confirmatory data analysis of the...
An application of neutrosophic logic in the confirmatory data analysis of the...
 
Ak park zak_2007
Ak park zak_2007Ak park zak_2007
Ak park zak_2007
 
Neuroeconomics
NeuroeconomicsNeuroeconomics
Neuroeconomics
 
Relevance of experimental design
 Relevance of experimental design Relevance of experimental design
Relevance of experimental design
 
Relevance of experimental design
 Relevance of experimental design Relevance of experimental design
Relevance of experimental design
 
Relevance of experimental design
 Relevance of experimental design Relevance of experimental design
Relevance of experimental design
 
Chapter 2
Chapter 2Chapter 2
Chapter 2
 
Teresa Swain 1 postsReTopic 2 DQ 2Drawing on your knowled.docx
Teresa Swain 1 postsReTopic 2 DQ 2Drawing on your knowled.docxTeresa Swain 1 postsReTopic 2 DQ 2Drawing on your knowled.docx
Teresa Swain 1 postsReTopic 2 DQ 2Drawing on your knowled.docx
 
The role of theory in research division for postgraduate studies
The role of theory in research division for postgraduate studiesThe role of theory in research division for postgraduate studies
The role of theory in research division for postgraduate studies
 
Rorschach Measures Of Cognition And Social Functioning Essay
Rorschach Measures Of Cognition And Social Functioning EssayRorschach Measures Of Cognition And Social Functioning Essay
Rorschach Measures Of Cognition And Social Functioning Essay
 
The role of theory in research
The role of theory in researchThe role of theory in research
The role of theory in research
 
Project Abstract
Project AbstractProject Abstract
Project Abstract
 
PR2_Wk1-2.pptx
PR2_Wk1-2.pptxPR2_Wk1-2.pptx
PR2_Wk1-2.pptx
 
Social_Research_Methods_on25Feb2016.pdf
Social_Research_Methods_on25Feb2016.pdfSocial_Research_Methods_on25Feb2016.pdf
Social_Research_Methods_on25Feb2016.pdf
 

Mehr von JoshuaLau29

schultz10e_ch13.ppt
schultz10e_ch13.pptschultz10e_ch13.ppt
schultz10e_ch13.pptJoshuaLau29
 
schultz10e_ch12.ppt
schultz10e_ch12.pptschultz10e_ch12.ppt
schultz10e_ch12.pptJoshuaLau29
 
schultz10e_ch10.ppt
schultz10e_ch10.pptschultz10e_ch10.ppt
schultz10e_ch10.pptJoshuaLau29
 
schultz10e_ch08.ppt
schultz10e_ch08.pptschultz10e_ch08.ppt
schultz10e_ch08.pptJoshuaLau29
 
schultz10e_ch07.ppt
schultz10e_ch07.pptschultz10e_ch07.ppt
schultz10e_ch07.pptJoshuaLau29
 
schultz10e_ch06.ppt
schultz10e_ch06.pptschultz10e_ch06.ppt
schultz10e_ch06.pptJoshuaLau29
 
schultz10e_ch05.ppt
schultz10e_ch05.pptschultz10e_ch05.ppt
schultz10e_ch05.pptJoshuaLau29
 
schultz10e_ch04.ppt
schultz10e_ch04.pptschultz10e_ch04.ppt
schultz10e_ch04.pptJoshuaLau29
 
schultz10e_ch03.ppt
schultz10e_ch03.pptschultz10e_ch03.ppt
schultz10e_ch03.pptJoshuaLau29
 
schultz10e_ch02.ppt
schultz10e_ch02.pptschultz10e_ch02.ppt
schultz10e_ch02.pptJoshuaLau29
 
schultz10e_ch01.ppt
schultz10e_ch01.pptschultz10e_ch01.ppt
schultz10e_ch01.pptJoshuaLau29
 
1-s2.0-000187919090008P-main.pdf
1-s2.0-000187919090008P-main.pdf1-s2.0-000187919090008P-main.pdf
1-s2.0-000187919090008P-main.pdfJoshuaLau29
 
減壓創新生_Group proposal (1).pdf
減壓創新生_Group proposal (1).pdf減壓創新生_Group proposal (1).pdf
減壓創新生_Group proposal (1).pdfJoshuaLau29
 
The Belief in Good Luck Scale.pdf
The Belief in Good Luck Scale.pdfThe Belief in Good Luck Scale.pdf
The Belief in Good Luck Scale.pdfJoshuaLau29
 
Interdependence.ppt
Interdependence.pptInterdependence.ppt
Interdependence.pptJoshuaLau29
 
ch08Interdependence and.ppt
ch08Interdependence and.pptch08Interdependence and.ppt
ch08Interdependence and.pptJoshuaLau29
 

Mehr von JoshuaLau29 (20)

schultz10e_ch13.ppt
schultz10e_ch13.pptschultz10e_ch13.ppt
schultz10e_ch13.ppt
 
schultz10e_ch12.ppt
schultz10e_ch12.pptschultz10e_ch12.ppt
schultz10e_ch12.ppt
 
schultz10e_ch10.ppt
schultz10e_ch10.pptschultz10e_ch10.ppt
schultz10e_ch10.ppt
 
schultz10e_ch08.ppt
schultz10e_ch08.pptschultz10e_ch08.ppt
schultz10e_ch08.ppt
 
schultz10e_ch07.ppt
schultz10e_ch07.pptschultz10e_ch07.ppt
schultz10e_ch07.ppt
 
schultz10e_ch06.ppt
schultz10e_ch06.pptschultz10e_ch06.ppt
schultz10e_ch06.ppt
 
schultz10e_ch05.ppt
schultz10e_ch05.pptschultz10e_ch05.ppt
schultz10e_ch05.ppt
 
schultz10e_ch04.ppt
schultz10e_ch04.pptschultz10e_ch04.ppt
schultz10e_ch04.ppt
 
schultz10e_ch03.ppt
schultz10e_ch03.pptschultz10e_ch03.ppt
schultz10e_ch03.ppt
 
schultz10e_ch02.ppt
schultz10e_ch02.pptschultz10e_ch02.ppt
schultz10e_ch02.ppt
 
schultz10e_ch01.ppt
schultz10e_ch01.pptschultz10e_ch01.ppt
schultz10e_ch01.ppt
 
1-s2.0-000187919090008P-main.pdf
1-s2.0-000187919090008P-main.pdf1-s2.0-000187919090008P-main.pdf
1-s2.0-000187919090008P-main.pdf
 
減壓創新生_Group proposal (1).pdf
減壓創新生_Group proposal (1).pdf減壓創新生_Group proposal (1).pdf
減壓創新生_Group proposal (1).pdf
 
137.pdf
137.pdf137.pdf
137.pdf
 
43_MSQ.pdf
43_MSQ.pdf43_MSQ.pdf
43_MSQ.pdf
 
The Belief in Good Luck Scale.pdf
The Belief in Good Luck Scale.pdfThe Belief in Good Luck Scale.pdf
The Belief in Good Luck Scale.pdf
 
Interdependence.ppt
Interdependence.pptInterdependence.ppt
Interdependence.ppt
 
EBP-2.ppt
EBP-2.pptEBP-2.ppt
EBP-2.ppt
 
ch09.ppt
ch09.pptch09.ppt
ch09.ppt
 
ch08Interdependence and.ppt
ch08Interdependence and.pptch08Interdependence and.ppt
ch08Interdependence and.ppt
 

Kürzlich hochgeladen

Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...Lokesh Kothari
 
Animal Communication- Auditory and Visual.pptx
Animal Communication- Auditory and Visual.pptxAnimal Communication- Auditory and Visual.pptx
Animal Communication- Auditory and Visual.pptxUmerFayaz5
 
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral AnalysisRaman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral AnalysisDiwakar Mishra
 
Biological Classification BioHack (3).pdf
Biological Classification BioHack (3).pdfBiological Classification BioHack (3).pdf
Biological Classification BioHack (3).pdfmuntazimhurra
 
Cultivation of KODO MILLET . made by Ghanshyam pptx
Cultivation of KODO MILLET . made by Ghanshyam pptxCultivation of KODO MILLET . made by Ghanshyam pptx
Cultivation of KODO MILLET . made by Ghanshyam pptxpradhanghanshyam7136
 
Nanoparticles synthesis and characterization​ ​
Nanoparticles synthesis and characterization​  ​Nanoparticles synthesis and characterization​  ​
Nanoparticles synthesis and characterization​ ​kaibalyasahoo82800
 
Disentangling the origin of chemical differences using GHOST
Disentangling the origin of chemical differences using GHOSTDisentangling the origin of chemical differences using GHOST
Disentangling the origin of chemical differences using GHOSTSérgio Sacani
 
Is RISC-V ready for HPC workload? Maybe?
Is RISC-V ready for HPC workload? Maybe?Is RISC-V ready for HPC workload? Maybe?
Is RISC-V ready for HPC workload? Maybe?Patrick Diehl
 
CALL ON ➥8923113531 🔝Call Girls Kesar Bagh Lucknow best Night Fun service 🪡
CALL ON ➥8923113531 🔝Call Girls Kesar Bagh Lucknow best Night Fun service  🪡CALL ON ➥8923113531 🔝Call Girls Kesar Bagh Lucknow best Night Fun service  🪡
CALL ON ➥8923113531 🔝Call Girls Kesar Bagh Lucknow best Night Fun service 🪡anilsa9823
 
Stunning ➥8448380779▻ Call Girls In Panchshil Enclave Delhi NCR
Stunning ➥8448380779▻ Call Girls In Panchshil Enclave Delhi NCRStunning ➥8448380779▻ Call Girls In Panchshil Enclave Delhi NCR
Stunning ➥8448380779▻ Call Girls In Panchshil Enclave Delhi NCRDelhi Call girls
 
Artificial Intelligence In Microbiology by Dr. Prince C P
Artificial Intelligence In Microbiology by Dr. Prince C PArtificial Intelligence In Microbiology by Dr. Prince C P
Artificial Intelligence In Microbiology by Dr. Prince C PPRINCE C P
 
Formation of low mass protostars and their circumstellar disks
Formation of low mass protostars and their circumstellar disksFormation of low mass protostars and their circumstellar disks
Formation of low mass protostars and their circumstellar disksSérgio Sacani
 
Call Girls in Munirka Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Call Girls in Munirka Delhi 💯Call Us 🔝9953322196🔝 💯Escort.Call Girls in Munirka Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Call Girls in Munirka Delhi 💯Call Us 🔝9953322196🔝 💯Escort.aasikanpl
 
Biopesticide (2).pptx .This slides helps to know the different types of biop...
Biopesticide (2).pptx  .This slides helps to know the different types of biop...Biopesticide (2).pptx  .This slides helps to know the different types of biop...
Biopesticide (2).pptx .This slides helps to know the different types of biop...RohitNehra6
 
Physiochemical properties of nanomaterials and its nanotoxicity.pptx
Physiochemical properties of nanomaterials and its nanotoxicity.pptxPhysiochemical properties of nanomaterials and its nanotoxicity.pptx
Physiochemical properties of nanomaterials and its nanotoxicity.pptxAArockiyaNisha
 
A relative description on Sonoporation.pdf
A relative description on Sonoporation.pdfA relative description on Sonoporation.pdf
A relative description on Sonoporation.pdfnehabiju2046
 
Botany 4th semester file By Sumit Kumar yadav.pdf
Botany 4th semester file By Sumit Kumar yadav.pdfBotany 4th semester file By Sumit Kumar yadav.pdf
Botany 4th semester file By Sumit Kumar yadav.pdfSumit Kumar yadav
 
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...Sérgio Sacani
 
Hubble Asteroid Hunter III. Physical properties of newly found asteroids
Hubble Asteroid Hunter III. Physical properties of newly found asteroidsHubble Asteroid Hunter III. Physical properties of newly found asteroids
Hubble Asteroid Hunter III. Physical properties of newly found asteroidsSérgio Sacani
 

Kürzlich hochgeladen (20)

Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
Labelling Requirements and Label Claims for Dietary Supplements and Recommend...
 
Animal Communication- Auditory and Visual.pptx
Animal Communication- Auditory and Visual.pptxAnimal Communication- Auditory and Visual.pptx
Animal Communication- Auditory and Visual.pptx
 
CELL -Structural and Functional unit of life.pdf
CELL -Structural and Functional unit of life.pdfCELL -Structural and Functional unit of life.pdf
CELL -Structural and Functional unit of life.pdf
 
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral AnalysisRaman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
Raman spectroscopy.pptx M Pharm, M Sc, Advanced Spectral Analysis
 
Biological Classification BioHack (3).pdf
Biological Classification BioHack (3).pdfBiological Classification BioHack (3).pdf
Biological Classification BioHack (3).pdf
 
Cultivation of KODO MILLET . made by Ghanshyam pptx
Cultivation of KODO MILLET . made by Ghanshyam pptxCultivation of KODO MILLET . made by Ghanshyam pptx
Cultivation of KODO MILLET . made by Ghanshyam pptx
 
Nanoparticles synthesis and characterization​ ​
Nanoparticles synthesis and characterization​  ​Nanoparticles synthesis and characterization​  ​
Nanoparticles synthesis and characterization​ ​
 
Disentangling the origin of chemical differences using GHOST
Disentangling the origin of chemical differences using GHOSTDisentangling the origin of chemical differences using GHOST
Disentangling the origin of chemical differences using GHOST
 
Is RISC-V ready for HPC workload? Maybe?
Is RISC-V ready for HPC workload? Maybe?Is RISC-V ready for HPC workload? Maybe?
Is RISC-V ready for HPC workload? Maybe?
 
CALL ON ➥8923113531 🔝Call Girls Kesar Bagh Lucknow best Night Fun service 🪡
CALL ON ➥8923113531 🔝Call Girls Kesar Bagh Lucknow best Night Fun service  🪡CALL ON ➥8923113531 🔝Call Girls Kesar Bagh Lucknow best Night Fun service  🪡
CALL ON ➥8923113531 🔝Call Girls Kesar Bagh Lucknow best Night Fun service 🪡
 
Stunning ➥8448380779▻ Call Girls In Panchshil Enclave Delhi NCR
Stunning ➥8448380779▻ Call Girls In Panchshil Enclave Delhi NCRStunning ➥8448380779▻ Call Girls In Panchshil Enclave Delhi NCR
Stunning ➥8448380779▻ Call Girls In Panchshil Enclave Delhi NCR
 
Artificial Intelligence In Microbiology by Dr. Prince C P
Artificial Intelligence In Microbiology by Dr. Prince C PArtificial Intelligence In Microbiology by Dr. Prince C P
Artificial Intelligence In Microbiology by Dr. Prince C P
 
Formation of low mass protostars and their circumstellar disks
Formation of low mass protostars and their circumstellar disksFormation of low mass protostars and their circumstellar disks
Formation of low mass protostars and their circumstellar disks
 
Call Girls in Munirka Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Call Girls in Munirka Delhi 💯Call Us 🔝9953322196🔝 💯Escort.Call Girls in Munirka Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
Call Girls in Munirka Delhi 💯Call Us 🔝9953322196🔝 💯Escort.
 
Biopesticide (2).pptx .This slides helps to know the different types of biop...
Biopesticide (2).pptx  .This slides helps to know the different types of biop...Biopesticide (2).pptx  .This slides helps to know the different types of biop...
Biopesticide (2).pptx .This slides helps to know the different types of biop...
 
Physiochemical properties of nanomaterials and its nanotoxicity.pptx
Physiochemical properties of nanomaterials and its nanotoxicity.pptxPhysiochemical properties of nanomaterials and its nanotoxicity.pptx
Physiochemical properties of nanomaterials and its nanotoxicity.pptx
 
A relative description on Sonoporation.pdf
A relative description on Sonoporation.pdfA relative description on Sonoporation.pdf
A relative description on Sonoporation.pdf
 
Botany 4th semester file By Sumit Kumar yadav.pdf
Botany 4th semester file By Sumit Kumar yadav.pdfBotany 4th semester file By Sumit Kumar yadav.pdf
Botany 4th semester file By Sumit Kumar yadav.pdf
 
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
Discovery of an Accretion Streamer and a Slow Wide-angle Outflow around FUOri...
 
Hubble Asteroid Hunter III. Physical properties of newly found asteroids
Hubble Asteroid Hunter III. Physical properties of newly found asteroidsHubble Asteroid Hunter III. Physical properties of newly found asteroids
Hubble Asteroid Hunter III. Physical properties of newly found asteroids
 

The Development of the Heuristics and Biases Scale.pdf

  • 1. Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718 1877-0428 © 2013 The Authors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license. Selection and peer-review under responsibility of Cognitive-counselling, research and conference services (c-crcs). doi:10.1016/j.sbspro.2014.01.1221 ScienceDirect International Conference on Education & Educational Psychology 2013 (ICEEPSY 2013) The Development of the Heuristics and Biases Scale (HBS) Marcin Skladab *, Rene Diekstrab a Utrecht University: University College Roosevelt, Lange Noordstraat 1 4330AB Middelburg, The Netherlands b The Hague Uuniversity of Applied Sciences,Johanna Westerdijkplein 75, 2521EN Den Haag The Netherlands Abstract Problem Statement: There is no comprehensive tool capturing general vulnerability to biases caused by the use of heuristics. Existing tools focus only on one specific bias or on personality traits. Research Questions: Can general vulnerability to heuristic thinking be assessed and what are the sub-dimensions of this construct? Can undergraduate students be successfully involved in the research process? Purpose of the Study: To demonstrate the results of an educational experiment in which undergraduate students are involved in the first stage of development of the Heuristics and Biases Scale (HBS). Research Methods: After getting acquainted with the underlying theory, students chose one specific bias or heuristic, investigated related results of the experiments and paradigms. At the later stage, under supervision, students developed items intended to capture the chosen bias. Finally, positively evaluated items were combined together and piloted. The psychometrical properties of the items and course outcomes were assessed. Findings: Developed items formed scales with satisfactory reliability. Course received positive student evaluations, and the assessment indicated that the majority of students achieved intended learning outcomes. Conclusions: The study indicates that it is possible to develop a psychometrically sound assessment to measure vulnerability to a range of common cognitive biases. Moreover, it is also possible to successfully involve undergraduate students in the development of a psychometrical tool. Acknowledgments: Authors would like to thank the Students of University College Roosevelt contributing to this work as their Independent Research Projects or as a part of Social Psychology or Statistics Course. © 2013 The Authors. Published by Elsevier Ltd. Selection and peer-review under responsibility of Dr Zafer Bekirogullari. * Corresponding author. Phone: +31 118 655 528 fax: +31 (0)118 655 50. E-mail address: m.sklad@ucr.nl Available online at www.sciencedirect.com © 2013 The Authors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license. Selection and peer-review under responsibility of Cognitive-counselling, research and conference services (c-crcs).
  • 2. 711 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718 Key Words: Cognitive Heuristics Biases Education Psychometric Scale 1. Introduction According to classic normative model of rational choice, when a choice has to be made, the expected utility of each option can be calculated given the probabilities and utilities of all possible outcomes of each option as the weighted average of the utilities of the outcomes. A rational actor making a decision should choose the option maximizing the expected utility (Neumann & Morgenstern, 1953). Obviously, excluding simple gambling problems, complete rationality defined in the above way is unachievable for human beings. Firstly, in case of most real life decisions the list of potential consequences of each option is very long, if not infinite, with many unknowns. Secondly, probabilities of various outcomes are usually not known and at best they can be only roughly estimated. Herbert Simon (1957) coined the term bounded rationality to describe limitations set by human’s computational and deliberative capacity on rationality of decision making. He argued that we seek to find satisfying solutions rather than optimal. Social psychologists went even further claiming that people often behave like cognitive misers trying to save cognitive resources by using as little information as possible to reach a conclusion and only if motivated opt for more elaborated cognitive strategies (Fiske and Taylor, 1991). The distinction between deliberate, strenuous, conscious mental processes and automatic processes requiring much less cognitive resources has been well described by a group of psychological theories described collectively as dual process theory (Evans, 2011). Independently on whether there are two discrete modes of reasoning (Sloman, 1996) or a continuum between implicit and explicit processes (Osman, 2004), there might be individual differences in relying on these two modes of reasoning (Stanovich & West, 2000). These differences might for instance be due to differences in cognitive abilities (Stanovich & West, 2000, Sattler, Hellix, & Neher, 1970) or to differences in preferred cognitive style and motivation (Fiske and Taylor, 1991; Petty, DeMarree, Brinol, Horcajo, & Strathman, 2008). Kahneman (2011) refers to dual-processing as two system thinking, with System 1 corresponding to intuitive unintentional, involuntary, and effortless, automatic thinking, and System 2 is responsible for reflective, controlled, conscious effort consuming large amount of mental energy and being limited by capacity of working memory. According to Kahneman, errors in judgment produced by System 1 are not random. Theory proposed by Tversky & Kahneman (1974) suggested that the process guiding intuitive judgment is not just an incomplete version of the rational model of judgment that could be executed by System 2. They postulated that judgments made by System 1 are qualitatively different, since they are made using different algorithms than judgments made by System 2. System 1 substitutes harder questions by simpler in order to produce solutions. In other words, System 1 relies on a limited number of heuristics - highly efficient mental shortcuts, which reduce complex tasks of assessing probabilities and predicting values to simple judgmental operations. Even though the use of heuristics most of the time provides a satisfying answer, it also leads to systematic biases. The nonrandom nature of the departures from the normative rational theory demonstrated in many experiments has been treated as an indicator of the existence of underlying heuristics producing these errors (Gilovich & Griffin, 2002). According to Kahneman (2011), there is also a certain degree of control over the mode of decision making. While System 1 is usually a default way of processing, one can make a decision to switch to System 2 in attempt to correct for these biases. This also suggests that there might be individual differences in the extent to which we rely on System 1 and System 2, due to personal preferences. 2. Problem Even though System 1 judgments based on heuristics provide answers of sufficient quality for most everyday decisions, they may also lead to serious errors. Unfortunately, there is ample evidence that people rely on
  • 3. 712 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718 heuristics also in case of decisions having extremely significant consequences, for instance: large investments (O’Donnell & Schultz, 2005), court sentences (e.g. Danziger, Levar and Avnaim-Pesso, 2011; Sigall and Ostrove, 1975), and medical decisions (e.g. Marewski & Gigerenzer, 2012). In this context, it is really relevant to be able to evaluate an individual’s susceptibility to cognitive biases. To authors best knowledge there is no behaviorally based measurement tool assessing this susceptibility. The closest existing tools are IQ tests measuring cognitive capacity and a self report tool measuring self reported motivation (Cacioppo & Petty, 1982). Therefore, the purpose of this Study is: 1) to establish whether different biases indeed reflect consistent individual differences in general tendency to rely on System 1, or each particular heuristic or bias is fairly independent, and 2) to determine whether it is possible to develop a measurement tool which measures individuals’ vulnerability to cognitive biases. Originally, Kahneman and Tversky (1974) proposed three main heuristics responsible for the systematic bias of judgment: availability heuristic, representativeness heuristic, and anchoring and adjustment heuristic. Later, in order to accommodate for the unquestionable pivotal role affective judgment plays in decision making (e.g. Damasio, 1994), affect heuristic (Slovic at al., 2002) has been added to the list and incorporated to the theory (Kahneman 2011). Kahneman (2011) postulated that several cognitive biases can be attributed to each heuristic. For instance, availability heuristic, which allows the actor to use ease of retrieval to estimate the frequency of a class or the probability of an event, can lead to availability bias and to the phenomenon of rare events bias. In case of both biases, the probability of events which can be easily retrieved from the memory is systematically overestimated. Therefore, recent and vivid events may have a disproportionate impact on decisions. In the representativeness heuristic, the resemblance of an object to the prototype serves as a proxy for estimating probabilities. The use of this heuristic can produce biases like: conjunction fallacy, base rate negligence, and gamblers fallacy. In all of these biases, estimation of the likelihood is biased because people to large degree base their judgment on information about the similarity of the evaluated event to the prototype. Opposite to formal logic, in case of conjunction fallacy, the chance of occurrence of a target with one unlikely feature is rated lower than the chance of occurrence of a target with the same unlikely feature plus one likely feature (Tversky & Kahneman, 1983). For instance, a white crow with two wings is rated more probable than a white crow. In base rate negligence (e.g. Kahneman & Tversky 1972; 1983Bar-Hillel, 1980), participants in experiments overestimate the chance that a target belongs to a relatively rare category if it, he or she resembles a prototypical member of this category. For instance, somebody who looks like a librarian is most likely a librarian. Gamblers fallacy makes people believe that an independent single random outcome will depend on previous outcomes in order to make series fit the image of random distribution. For instance, after three tails one expects head. Anchoring and adjustment heuristic allows us to estimate a value starting from a known position and adjusting the value based on facts that come to mind. Unfortunately, if the original anchor is off, adjustment is usually not sufficiently big, which leads to a bias. Reliance on feelings associated with these affective responses in judgment has been defined as the affect heuristic. Almost any potential characteristic of a person or object has a certain affective valence. Thanks to affect heuristic, when people need to evaluate any property of the target, they can substitute the property in question with general affective evaluation of the target (Slovic, et al. 2002). The most well known bias related to this heuristic is halo effect (Thorndike, 1920; Asch, 1946). In the halo effect, one vivid characteristic of a person determines how this person is judged on all unrelated characteristics. For example, a beautiful person is perceived as kind. According to Tversky (2011), the use of judgmental heuristics is synonymous with automatic System 1 processing. For, heuristics replace estimating characteristics (like probability) requiring elaborate formal thinking by estimates of characteristics automatically and effortlessly provided by System 1: similarity, ease of recall and affective value.
  • 4. 713 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718 If the assumption that cognitive biases are a product of relying on an automatic processing style is correct, it should be possible to evaluate the general reliance on automatic system 1 processing by the amount and intensity of judgmental biases demonstrated by an individual. In the presented research, the initial development of the items constituting cognitive bias scales has been carried out as a part of problem based learning (PBL, Barrows 1996) by students in the intermediate level psychology class. Aim was to test whether undergraduate university college students can be successfully involved in academic research this way, to the benefit of both students and research. 3. Research Method As was mentioned before, the purpose of the study was: 1) to develop and pilot the scales to assess individual’s reliance on heuristics and vulnerability to cognitive biases and 2) to pilot the procedure of involving university college students in the development and to enhance their learning through PBL (Barrows 1996). In this approach, students start with the problem, then work in a peer group to explain the problem and to construct the needed knowledge. Meanwhile, the teacher provides the framework for it. Later, the students work on their individual solution and then regroup at the end to share their findings. To reduce issues related to potential excess of cognitive load in the initial stage of PBL (Sweller, 1988), in this project, students initially got acquainted with the general underlying theory and heuristics and biases approach (Kahneman, 2011) and chose one specific bias or heuristic to study. In the next stage, they wrote a short essay summarizing the processes postulated to be involved in the bias. Moreover they investigated related results of experiments and experimental paradigms used to demonstrate the chosen bias. At this time, they were also confronted with the problem of creating a test item taping into the selected heuristic or bias, and they attempted to produce an initial solution. Next, based on their choices, they were assigned to workgroups of three to four people and worked together towards the goal of developing a handful of fully functional items. This process, guided by the framework provided by the teacher, took several stages in which students worked individually and in the workgroups to polish and exchange feedback on their items under supervision. During these stages, students were also given the necessary psychometrical background. Finally, after a pilot of studying individual items, students in each group selected the best items intended to capture the chosen bias. Finally, items positively evaluated on their face validity were combined with other items developed by researchers and piloted. Finally, an assessment was carried out of both the enhancement of learning by students as well as the developed scales. The problem based learning project was evaluated based on self-reported learning outcomes. At the end of the project, besides a regular course evaluation form, students filled a short self report questionnaire. In this self report questionnaire, they could indicate if the HBS project reached its learning objectives and they could indicate if, given the opportunity, they would like to develop their work on HBS further. Psychometrical properties of the items were assessed based on the results of the pilot. The group of students involved in the development of the scales consisted of a regular class of 25 English speaking students of a Dutch residential university college. The students were mostly in their second year of undergraduate honors program. Before being elected to do the second year level intermediate psychology course, they had already successfully completed an introductory psychology course and a methodology and statistics for social sciences course. The pilot questionnaire including the items based on the students’ work contained three parts. The first part concerned the demographical background of the participant. The second part included 59 heuristics and biases- related items. Each item consisted of one up to three interconnected questions, which were combined to produce one unique item score. The biases-related questions covered the following biases: Hindsight bias, Conjunction fallacy, Planning fallacy, Halo effect, Phenomenon of rare Events, Overconfidence bias, and Susceptibility to
  • 5. 714 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718 priming. Items concerning Gamblers fallacy, Availability bias, Anchoring bias and Sunk cost fallacy were abandoned due to dissatisfactory properties. After the first small scale pilot, an adjusted final questionnaire was distributed to a convenience sample of respondents. The questionnaire was distributed in several clusters in order to diversify the sample. The clusters included: high school students, elderly house residents, students of University College, relatives and acquaintances of students participating in the class. The questionnaire was either handed out in a paper form and collected later, or filled under supervision on the spot, or it was delivered and returned by email. All participants received the questionnaire with the explicit instruction to fill it in individually without asking for help. Completing the questionnaire took approximately 45 minutes on average. Out of the 163 questionnaires that were distributed, 128 complete samples were collected. In the sample, gender was not equally distributed, for the sample consisted in 63.5% of female participants. The average age of the participants was equal to 25.91 years (S=13.44) and ranged from 15 to 60 years. The majority of the sample (65.7%) consisted of students. The analysis will be presented in two parts. The first part describes the results of the questionnaire considering students’ learning experience. The latter part concerns initial psychometrical evaluation of the produced items by analysis of their distribution, intercorrelations, reliability, and exploratory factor analysis. 4. Findings 4.1. Learning Experience Evaluation of learning objectives of the course in the self report (see Table 1) indicated that, in students reports, the project met its learning objectives. Each item concerning a learning objective received a statistically significant (p<.005) positive answer on average. Learning from peers also received significantly (p=.011) positive evaluation. Among students, 40% agreed that given the opportunity they would like to develop their work further. Table 1. Students Evaluations of course objectives (1- strongly disagree, 7-strongly agree) Mean Std. Deviation learned about biases and heuristics influencing human decision making 5.80 1.24 learned how cognitive biases manifest themselves in practice 5.50 1.05 I learned how to communicate about cognitive biases and heuristics 5.00 1.26 I gained a better understanding of limitations to our rationality 5.85 0.88 I learned a lot from my fellow students (discussing the items together, listening to the presentations and reviewing their papers) 4.85 1.35 I gained interest in our cognitive processes 5.50 1.10 Note: N=20,
  • 6. 715 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718 4.2. Heuristics and biases tool From the initial pool of 39 items with satisfactory face validity, items were selected for the factor analysis only if there were at least two of them concerning the same cognitive bias and only if they correlated at least .25 with at least one other item, suggesting reasonable factorability. This procedure left 23 items related to vulnerability to priming, hindsight bias, conjunction fallacy, planning fallacy, halo effect and phenomenon of rare events. Screening revealed that the data was suitable for factor analysis. The minimum amount of data for factor analysis was satisfied, with a final sample size of 128 (using list-wise deletion), providing a ratio of over 5 cases per variable. The Kaiser-Meyer-Olkin measure of sampling adequacy was .53, which renders the data acceptable for factor analysis, and Bartlett’s test of sphericity was significant (χ2 (253) = 766.50, p < .001). For all items, measures of sampling adequacy and the communalities were all acceptable above .5 and .4 respectively (see Table 2), later confirming that each item shared some common variance with other items. Due to exploratory nature of the study, principal components analysis has been chosen to identify factors underlying the set of items. One (System 1 thinking), two (heuristics), and six (biases) factors solutions were theoretically supported. Initial eigen values indicated that the first factors explained 12%, 10.5%, 9.5%, 8.5%, 7.5%, 6.2%, 5.7% of the variance respectively. The analysis of eigen values on the scree plot indicated leveling off after five, seven factors; and 11 factors. The 11 factor solution was rejected because last factors yielded eigen values below 1.0 and the small number of variables per factor. The five factor solution failed to assign primary loadings to all items. The seven factor solution, which explained 59.7% of the variance has been chosen as a result. There was no difference between the orthogonal and non orthogonal rotation in terms of assignment of items to components and of clarity of the assignment, thus varimax rotation has been used for the final solution. Factors extracted by rotated solutions had roughly the same explanatory power with the first accounting for 9.8% of total variance and the last for 7.2% of variance. All items had primary factor loading or above .4 and no item had a cross-loading above .4.
  • 7. 716 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718 Table 2. Factor loadings and communalities based on a principal components analysis with varimax rotation for 23 items included in the HBS questionnaire (N = 228) Communality 1 CF 2 PF1 3 HB 4 PRE 5 HE 6 PF2 7 PRI Priming 1 .621 .778 Priming 2 .704 .796 Hindsight Bias 2 .604 .739 Hindsight Bias 3 .613 .768 Hindsight Bias 4 .458 .635 Hindsight Bias 5 .418 .466 Conjunction Fallacy 1 .622 .723 Conjunction Fallacy 2 .407 ,557 Conjunction Fallacy 3 ,458 .614 Conjunction Fallacy 4 .545 .710 Conjunction Fallacy 5 .535 .685 Planning Fallacy 1 .687 .820 Planning Fallacy 2 .481 .631 Planning Fallacy 3 .682 .796 Planning Fallacy 4 .642 .553 Planning Fallacy 6 .768 .848 Planning Fallacy 7 .693 .813 Halo Effect 1 .508 .615 Halo Effect 2 .476 .589 Halo_Effect 3 .446 .656 Halo Effect 4 .582 .721 Phenomenon of rare events 3 .873 .923 Phenomenon of rare events 4 .918 .942 Note. Factor loadings < .4 are suppressed. CF-Conjunction Fallacy, PF-Planning fallacy, HB- Hindsight bias, PRE- phenomenon of rare events, HE-halo effect, PRI-Priming. The analysis revealed a factor structure closely matching the pattern of biases addressed by the items. Thus, the 7 factors were given labels corresponding to the biases the items were intended to measure. The items related to planning fallacy formed two distinct factors. The first planning fallacy related to fallacy concerning underestimating the amount of time and work endeavors consume. The second related to the amount of money one spends on it. Internal consistency for each of the scales was examined using Cronbach’s alpha, and median inter-item correlation. The alphas were moderate: .60 for priming (2 items, r=.46), .62 for Hindsight Bias (5 items, r=.28), .60 for Conjunction Fallacy (4 items, r=.26), .69 for Time Planning Fallacy (3 items, r=.39), . 67 for Money Planning Fallacy (3 items, r=.47) ,. 92 for Phenomenon of rare events (2 items, r=.80). No increases in alpha for any of the scales could have been achieved by eliminating any item. The seven factor scores were computed as average of corresponding items. The analysis of correlations between the factors revealed two
  • 8. 717 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718 statistically significant (p<.05) correlations: The first between two types of planning fallacy (r=.28), the second between priming and halo effect (r=-.20). 5. Conclusion The study had twofold purposes. First, to test whether it is possible to involve undergraduate students in the development of a psychometrical tool with benefits to them and the product. The second purpose was to initiate the construction of a set of scales allowing for estimation of individuals’ vulnerability to biases and for estimation of the use of heuristical thinking. The results seem to confirm the educational benefit of the project. However, since the conclusion concerning educational benefit is based on self reports and a small sample, rather than a controlled experiment, it should be taken with caution. The pilot of the measurement tool revealed several interesting results. Firstly, the items addressing biases were able to form internally consistent scales. Medians of inter item-correlations for all factors exceeded .25 suggesting good reliability of individual items (Robinson, Shaver, & Wrightsman, 1991). It also suggests that if the number of items in each scale would be increased to a reasonable amount, all scales would reach a satisfactory alpha (Carmines & Zeller, 1979). Therefore, it can be concluded that biases can be reliably measured using the design. It is also worth noting that there were no positive correlations between clusters of items addressing different biases. This indicates that there is no support for the existence of underlying common source of different biases, like habitual use of System 1 or heuristic (Kahneman, 2011). However, this may be partially the outcome of the fact that the used sample of items by no means represents well the complete spectrum of known cognitive biases. In summary, the results show that developing a scale addressing vulnerability to cognitive biases is possible. It can be done by extending the current study in three ways: in terms of the sample size, the number of biases addressed, and the number of items per bias. 6. References Asch, S.E. (1946) Forming impressions of personality. The Journal of Abnormal and Social Psychology. 41(3), Jul 1946, pp. 258-290. Bar-Hillel, M. (1980). The Base Rate Fallacy in Probability Judgements. Acta Psychologica, 44, 211-233. Barrows, H. S., (1996). Problem-based learning in medicine and beyond: A brief overview. New Directions for Teaching and Learning 68, 3–12. doi:10.1002/tl.37219966804 Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42(1), 116-131.doi:10.1037/0022-3514.42.1.116 Carmines, E. G. & Zeller, R. A. (1979) Reliability and validity assessment. Thousand Oaks, CA: Sage Publications. Danziger, S., Levav, J., & Avnaim-Pesso, L. (2011). Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences, 108(17), 6889-6892. Damasio, A. R. (1994). Descartes’ error: Emotion, reason, and the human brain. New York: Avon. Evans J. S. B. T., (2011). Dual-Process Theories Of Reasoning: Contemporary Issues And Developmental Applications Developmental Review, 31 (2-3), 86–102. Fiske, S.T., & Taylor, S.E. (1991). Social cognition (2nd ed.). New York: McGraw-Hill. Gilovich, T., & Griffin, D.W. (2002). Introduction – Heuristics and biases – Then and now. In T. Gilovich, D.W. Griffin, & D. Kahneman (Eds.), Heuristics and Biases: The Psychology of Intuitive Judgment (pp. 1-18). Cambridge: Cambridge University Press. Kahneman, D. (2011). Thinking, Fast and Slow. London: Penguin Books. Kahneman, D., Tversky, A. (1972). On prediction and judgment. Oregon Research Institute Bulletin 12 (4). O'Donnell, E., & Schultz. J. (2005). The halo effect in business risk audits: Can strategic risk assessment bias auditor judgment about accounting details? The Accounting Review 80(3), 921-38.
  • 9. 718 Marcin Sklad and Rene Diekstra / Procedia - Social and Behavioral Sciences 112 (2014) 710 – 718 Marewski J.N., Gigerenzer G., (2012). Heuristic Decision Making in Medicine. Dialogues Clinical Neuroscience 14, 77–99. Von Neumann, J., & Morgenstern, O. (1953). Theory of Games and Economic Behavior. New York: John Wiley and Sons. Osman, M. (2004). An evaluation of dual-process theories of reasoning. Psychonomic Bulletin & Review 11 (6): 988–1010. Pacini, R., & Epstein, S. (1999). The relation of rational and experiential information processing styles to personality, basic beliefs, and the ratio-bias phenomenon. Journal of Personality and Social Psychology, 76(6), 972-987. doi: 10.1037/0022-3514.76.6.972 Petty, R. E., DeMarree, K. G., Brinol, P., Horcajo, J., & Strathman, A. J. (2008). Need for cognition can magnify or attenuate priming effects in social judgment. Personality and Social Psychology Bulletin, 34, 900-912 Robinson, J. P., Shaver, P. R., & Wrightsman, L. S. (1991). Criteria for scale selection and evaluation. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.). Measures of personality and social psychological attitudes (pp. 1-16) New York: Academic Press. Sattler, J. M., Hillex, W. A., & Neher, L. A. (1970). Halo effect in examiner scoring of intelligence test responses. Journal of Consulting and Clinical Psychology, 34(2), 172-176. Sigall, H., Ostrove, N. (1975). Beautiful but dangerous: effects of offender attractiveness and nature of the crime on juridic judgement. Journal of Personality and Social Psychology. 31(3), 410-414. Simon, H. (1957). "A Behavioral Model of Rational Choice", in Models of Man, Social and Rational: Mathematical Essays on Rational Human Behavior in a Social Setting. New York: Wiley. Sloman S.A. (1996) The empirical case for two systems of reasoning. Psychological Bulletin, 119, 3-22. Slovic, P., Finucane, M., Peters, E., MacGregor D. G., (2002). The Affect Heuristic. In Thomas Gilovich, Dale Griffin, Daniel Kahneman. Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge University Press. pp. 397–420. Stanovich, K. E., & West, R. F. (2000). Individual Differences in Reasoning: Implications for the Rationality Debate. Behavioral and Brain Sciences, 23, 645-726. Sweller, J., (1988). Cognitive load during problem solving: Effects on learning, Cognitive Science, 12, 257-285. Thorndike, E.L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4(1), 25-29. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124- 1131. Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90(4), 293-315.