This document provides background on the author's research interests in sensemaking and complexity over several decades. Some key findings from the author's past work include: - People operate at two levels when problem solving - attempting to solve the problem, and also make sense of the nature of the problem. - Older subjects and those with less education showed more variability in performance on novel tasks, suggesting they had to construct strategies to make sense of the tasks. - Subjects in experiments actively attempt to make sense of the experimental situation and optimize their performance rather than just measuring "hardware" abilities. - The author's work on design problem solving found the best designers reformulate problems as they solve them.

1. Sensemaking & Complexity:
Position Paper for CHI 2005 Workshop
John C. Thomas
T. J. Watson Research Center
Background: The topic of this workshop provides a fascinating and insightful way of
viewing many of the research topics that I have been involved in over time. Although the
term “sensemaking” has not always been applied to my previous relevant work, I’ve
been interested in the substance of this topic for several decades. My dissertation, “An
analysis of behavior in the Hobbits-Orcs problem” (Thomas, 1974) offered a
comparison between actual behavior and the then popular notion that problem solving
proceeded as a step-by-step process of minimizing the difference between the current
state and the desired state. I was suspicious of this model, partly because of the
observations of DeGroot (1965) who studied grandmaster chess players and found that
they examined one branch of play, examined another branch of play, and then re-
examined the first branch of play. This behavior cannot be a question of “forgetting”;
rather, it seemed obvious to me, that they had discovered something general about the
situation that they were in during their examination of the second branch of play that
caused them to re-evaluate their thinking with respect to the first branch of play.
Similarly, in my dissertation, I found that people were operating at two levels
simultaneously. Yes, they were attempting to solve the problem at hand. In addition,
they were attempting to “make sense of” and “understand” the nature of the problem
at hand. Converging evidence from several sources including latencies to make a move,
error probabilities, verbal protocols, and transfer effects all indicated that within the
overall problem of moving from the current state to the goal state, people had to learn or
discover three basic properties of the situation that they found themselves in. These
discoveries about the nature of the problem space took long and highly variable times.
Deciding on what “move to make” was done relatively quickly and with a much smaller
variance.
In my work managing a project on the “Psychology of Aging”, (Fozard, Thomas &
Waugh, 1976; Thomas, Fozard, & Waugh, 1977) there were two findings of particular
interest to this workshop. First, healthy older male veterans, across a wide variety of
cognitive tasks, showed not only increased times to complete tasks, but also, in
comparison with younger healthy males, increased variance both within and between
subjects. Second, with many other variables partialed out, there was a significant
negative correlation between years of formal education and intra-subject variance. For
these subjects, veterans who were retired or came from a variety of white and blue collar
professions, the battery of experiments to which they were exposed (e.g., choice
reaction time, memory scanning, paired associate learning) represented novel tasks.
The subjects were “instructed” how to do the tasks and given a few practice trials.
However, it is highly unlikely, a priori, that everyone was immediately able to translate
the declarative knowledge of instruction into an optimally constructed procedure for
accomplishing the various tasks. How much of the observed age effect in variance was
due to biological factors, as opposed to generational factors (younger subjects, e.g.,
probably having more practice with more arbitrary tasks such as school requires and
having carried these out much more recently) is impossible to tell from those data.
Similarly, there are alternative explanations about the possible reasons for a correlation
between having more formal education and having a smaller variance in performance.

2. What both of these findings do question, however, is the traditional psychological
assumption that human performance in experiments is measuring something
fundamental about the “hardware” of the human system. In reality, subjects in
experiments are active participants who quickly attempt to make sense of the situation
and often must construct strategies, attempt to gain feedback about the effectiveness of
those strategies and then modify their strategies accordingly.
Perhaps three personal anecdotes can be useful in illustrating the point that subjects are
actively attempting to make sense of the experimental situation and thinking about how
to optimize their performance in that situation. The first two concern the author as an
experimental subject and the third as an experimenter. In our undergraduate
introductory psychology class, the instructor illustrated paired associate learning by
giving multiple learning trials of a series of ten paired associate CVC’s. After learning
all the associates on the first trial, I was amazed how many trials other students took. In
asking some of them about this later, it became clear that what seemed obvious to me;
viz., to make a story out of each paired associate, was not at all an obvious strategy to
others. So, e.g., the first pair was MOF-DAQ and I imagined being offered a strawberry
daiquiri by my friend Bob Hoerner. In graduate school, I was a subject in a
tachistoscopic experiment in which letters or words were presented followed by a
“mask” (a random pattern of dots) which is supposed to “erase” the retinal image.
My results were atypical because when the letter was presented, I immediately moved
my eyes away thus leaving the retinal image of the target and the mask in two different
places on the retina. As an undergraduate, one of my part time jobs was to teach space
science to sixth graders and another job involved being a research assistant to a
behavioral psychologist. In that latter context, I was running subjects in a large
“Skinner Box” in an experiment to determine whether or not applying a verbal label to
a discriminative stimulus (in this case, a large red circle) made a difference in
generalization gradients along various dimensions. Before one particular experimental
trial, one of the kids was simply waiting their turn in the anteroom and so I decided
instead of just having him sit there with folded hands, I would teach him about the
planets. After the experiment, when I debriefed this subject, to my complete amazement
and shock, he interpreted the experiment to be a test of how well he had learned the
material I presented on the solar system!! From my perspective, the little mini-lecture on
the solar system was simply a way to pass the time and impart some knowledge and
had nothing whatever to do with the operant conditioning experiment. From the
perspective of the subject, however, who came to the “University” to a “laboratory”
and then proceeded to see a lot of diagrams with circles and names on a chalkboard
followed by watching a series of red circles and pulling a lever for nickels followed by
being presented with other colors and sizes of circles and ellipses, pulling a lever and
not getting nickels, this was one single experience which he attempted to make some
coherent sense out of. He constructed a narrative, if you will, that included all the data.
When I joined IBM Research in 1973, my first set of studies involved evaluating an idea
for a new query language called, “Query By Example” in which users wrote queries
directly into a visual representation of a relational data base. For a variety of reasons, it
turned out that this language was remarkably easy to learn and use, in comparison with
other available alternatives, in the sense that experimental subjects did well at
translating English questions into Query By Example. There were some troubling
exceptions, however, mainly having to do with the directness of that translation process,
on a word by word basis. For example, one column in the sample data base was
labeled “Year of Hire.” If the following English question were presented to users:

3. “Print a list of all the employees hired after 1970,” the results were quite good.
However, if the English question were put in this form, “Print a list of all the employees
with less than three years experience,” the results were much less encouraging.
Similarly, given an English question such as, “Print a list of all the items that are both
large and red,” the queries tended to be quite accurate. However, when presented with
“Print a list of all the large items and all the red items,” the results were much less
encouraging. (In the latter case, subjects tended to use the AND operator in the query
language when an OR operation was called for). In a follow-up experiment, subjects
(college students) were not given English queries to “translate” into Query By
Example, but instead, were given a fairly complex data base reflecting various relations
in a typical college. Then, they were given a series of “issues” and asked to write their
own queries whose answers might shed some light on those issues. Then, they were to
translate their own English queries into Query By Example. By and large, students at
that time (@ 1975) were fairly clueless about the types of questions that could and could
not be reasonably answered by a computer system. For instance, in response to the
issue, “Many of the younger faculty feel that they are not paid enough relative to the
older faculty,” many students wrote the English question, “Are the younger faculty
being paid enough?” and then attempted to translate that into Query By Example.
Given the much more widespread use computers by students today and of googol in
particular, a similar study might yield quite different results now. However, the general
result remains. Successful use of a system requires more than simply understanding
the syntax of the system; it requires making sense of the situation and how to respond to
that situation (Thomas & Gould, 1975; Thomas, 1983).
During the 1970’s, we conducted a series of experiments on “the psychology of
design” (Thomas & Carroll, 1978; Carroll, Thomas & Malhotra,1979; Carroll, Thomas &
Malhotra,1980; Malhotra,Thomas, Carroll & Miller, 1980). Here too, it became clear
that “solving” a design problem, while difficult, was often not nearly so crucial as
finding and formulating problems. The best designers were able to formulate a problem,
attempt to solve it, and then, in the course of solving it, completely redefine and
reformulate the problem itself. In most of our formal education, however, this kind of
behavior is not only unnecessary; it is actively discouraged or penalized. To address
this issue in my own teaching during undergraduate statistics classes, I sometimes
posed the following quiz question --- which many of the students found inordinately
difficult --- “If I do three t-tests on Monday and four t-tests on Tuesday, how many t-
tests have I done in total?” The overall interpretation of the context of statistics for
many students is that it is a difficult, complex, and novel situation. Giving a problem in
that context that is easy, simple, and relies only on already acquired skills requires a
change of perspective and set. Reflection on this question lead to a more general
discussion on the importance of understanding the “deep” (and not just “superficial”)
nature of a situation before applying a particular statistical test.
More recent and current work relevant to sense-making .
Stories. From approximately 1999-2001, I managed a research project on the business
uses of stories and story-telling. Stories are a quintessential way for people to make
sense of complex situations. Stories can prove useful in cultural change, personal
change, sales, knowledge creation and sharing (Thomas, 1999). They can be highly
memorable and motivating. On the downside, once a person accepts as “true” a
particular narrative viewpoint of a complex situation, it can be difficult to persuade them
to consider alternative ways to make sense of a complex situation.

4. Pattern Languages. The term “Pattern Language” was first introduced by Alexander
(Alexander, et. als, 1977) in the field of architecture. Since, Pattern Languages have
been applied to such diverse fields as object-oriented programming, management, and
human-computer interaction. Patterns are named recurring abstract solutions to
recurring problems. A Pattern Language is a lattice of inter-related patterns that attempt
to provide coverage for the set of recurring problems in a given field. A person familiar
with the Pattern Language in a given field can use them, not only as a guide to solving
specific types of problems, but also as a sensitization device for finding and formulating
problems. In this sense, Pattern Languages provide one conceptual tool for sense-
making for use by individuals or communities. For the past several years, we have been
involved in working collaboratively to develop a socio-technical pattern language along
with tools to help construct, organize, find, and use patterns.
e-learning. In 2002-2003, I was responsible for the user experience for a “Dynamic
Assembly of Learning Objects” project (Farrell, Thomas, Rubin, Gordin, Katriel,
O’Donnell, Fuller, 2004; Farrell, Thomas, Dooley, Rubin, Levy, O’Donnell, Fuller,
2003; Farrell, Dooley, Thomas, Rubin & Levy, 2003). Learning Objects are much like
learning modules and also contain metadata that may specify topic, level of difficulty,
prerequisites, intended audience, rhetorical purpose, author, length, reading level and so
on. Our system enables users to build a kind of personalized mini-course relevant to
their specific learning goals, background and time constraints. Initial interviews indicated
a strong need for this kind of personalized but semi-automatically generated course. In
our case, we use metadata added by Subject Matter Experts in conjunction with a
pedagogically motivated ontology to help select and organize the material. A series of
field studies and an experimental study indicated that the system helped considerably in
sense-making (Thomas & Farrell, 2004).
Business Consulting. Currently, I am working with IBM business consultants to build
tools to help them with what they do which is essentially to help their clients with
collective sense-making. The fastest growing business segment in IBM is services and
a large part of that is business consulting services. In many ways, the very existence of
business consultants gives support to the importance of sensemaking as well as its
apparent difficulty. After all, should it not be the case that the executives who are
running a company should know more about it than anyone else? If it were not
commonly done, we might think it very strange that highly competitive and highly paid
executives would pay outside consultants to help them “make sense” of their own
company and how it fits into a larger ecological scheme.
Business consultants can prove valuable precisely because they are able to see
patterns and use perspectives that are different from the ones that their clients have
grown accustomed to.
Making Sense of Sensemaking
Solving problems, in particular, well-defined problems, often requires a logical, step-by-
step approach. Much of our educational process values, trains, and rewards such a
process. In real life, such approaches can prove useful in solving engineering problems,
writing computer programs, or implementing accounting practices. Such approaches,
however, are almost completely useless, in my opinion, for discovering engineering
problems to be solved, understanding what computer programs are worth writing, or for

5. deciding whether an accounting practice is ethical. Problem finding and problem
formulation are much more akin to what are generally considered perceptual rather than
conceptual processes. Doing a good job in problem finding and formulation requires
taking multiple perspectives, being able to distinguish figure from ground, being able to
see patterns, and being able to relate the present situation to relevant past experiences.
I believe that technological aids have primarily, but not exclusively, focused on aiding
people in solving well-defined problems. However, I believe that technological aids can
be designed to help people with sensemaking. The key approaches here are to allow
multiple and flexible representations of situations, to bring to bear multiple perspectives,
and to remind people of potentially relevant experiences.
References:
Alexander, C. A., Ishikawa, S., Silverstein, M., Jacobson, M. Fiksdahl-King, I., and
Angel, S. A Pattern Language. New York: Oxford Press, 1977.
Carroll, J., Thomas, J.C. and Malhotra, A. (1980). Presentation and representation in
design problem solving. British Journal of Psychology/,71 (1), pp. 143-155.
Carroll, J., Thomas, J.C. and Malhotra, A. (1979). A clinical-experimental analysis of
design problem solving. Design Studies, 1 (2), pp. 84-92.
DeGroot, A. D. (1965). Thought and choice in chess. The Hague: Mouton.
Farrell, R., Thomas, J. Rubin, B., Gordin, D., Katriel, A., O’Donnell, R., Fuller, E., and
Rolando, S. Personalized just-in-time dynamic assembly of learning objects. E-learning
2004. November, 2004.
Farrell, R., Thomas, J., Dooley, S., Rubin, W., Levy, S., O’Donnell, R., Fuller, E.
Learner-driven assembly of Web-based courseware. E-learn 2003 , Phoenix, Arizona,
Nov. 7-11, 2003.
Farrell, R., Dooley, S., Thomas, J., Rubin, B. And Levy, S. Implementing and extending
Learning Object Metadata for learning-based assembly of computer-based training.
Learning Technology Newsletter, Vol.5, 1, January, 2003, 14-16.
Fozard, J. L., Thomas, J. C., and Waugh, N. C. (1976). Effects of age and frequency of
stimulus repetitions on two-choice reaction time. Journal of Gerontology, 31, (5), pp.
556-563.
Malhotra, A., Thomas, J.C. Carroll, J. M., and Miller, L. A., (1980). Cognitive processes
in design. International Journal of Man-Machine Studies, 12, pp. 119-140.
Thomas, J.C. (1974). An analysis of behavior in the hobbits-orcs problem. Cognitive
Psychology 6 , pp. 257-269. Thomas, J.C. & Gould, J.D., (1975), A psychological study
of Query By Example, Proceedings of AFIPS, 1974 National Computer Conference,
Arlington, VA: AFIPS Press, 44, 439-445.
Thomas, J. C., Fozard, J. L. and Waugh, N. C. (1977). Age-related differences in naming
latency. American Journal of Psychology, 90(30), pp. 499-509.

6. Thomas, J.C. (1978). A design-interpretation analysis of natural English. International
Journal of Man-Machine Studies, 10, pp. 651-668.
Thomas, J.C. and Carroll, J. (1978). The psychological study of design. Design Studies,
1 (1), pp. 5-11.
Thomas, J.C. (1983). Psychological issues in the design of data-base query languages.
In M. Sime and M. Fitter (Eds.), Designing for human-computer communication..
London: Academic Press.
Thomas, J. C. (1999) Narrative technology and the new millennium. Knowledge
Management Journal, 2(9), 14-17.
Thomas, J. & Farrell, R. (2004). An experimental investigation of the effectiveness of
individualized web-based learning based on the dynamic assembly of learning objects.
IBM Research Report, 2004.