120 years ago the emergent field of experimental psychology became embroiled in debates as to whether plateaus in performance are real (or not) and if so whether they were due to periods in which league-stepping methods (originally defined as a hierarchy of habits that enabled experts to step leagues while novices were ``bustling over furlongs or inches'') were being acquired (or not). 20 years ago both the human-computer interaction and cognitive science communities were seized with concerns over performance plateaus (i.e., extended periods of stable suboptimal performance) from experts. I briefly review this history with the aim of drawing distinctions between performance asymptotes and performance plateaus, and argue that remediating one is the domain of design while remediating the other is the domain of training.

1. Asymptotes, Plateaus, and Limits to Human Performance
Wayne D. Gray
Cognitive Science Department, Rensselaer Polytechnic Institute
Presentation to the IBM Cognitive Systems Institute – 2015.Feb.05
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 1 / 30

2. THANKS TO THESE SUPPORTERS!
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 2 / 30

3. The Cognitive Science of The Little Engine that Could
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 3 / 30
A view of human systems (aka people) as attempting to optimize
performance under constraints. Where these constraints come from:
Bounds on our innate cognitive capacities.
Limits to our acquired skill and knowledge.
The structure of the external task environment in which we
operate.
The goals we are trying to achieve.
If this human system is chugging along, like the Little Engine that
Could, at some given performance ceiling – we can then ask whether
this limit this due:
to external factors that can be altered,
to internal factors that can be altered, or
to factors that cannot be altered.

4. PERFORMANCE GENERALLY IMPROVES WITH
PRACTICE. . . BUT WHY?
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 4 / 30
W. L. Bryan
With practice, performance whether it is with
telegraphy, typing, software programs,
arithmetic, programming, mnemonics, or
video games generally improves.
But experts are not simply faster than
novices; rather, they develop a hierarchy of
habits than enable them to “step leagues”
while novices are “bustling over furlongs or
inches”.
CITATION
Bryan, W. L. & Harter, N. (1897). Studies in the physiology and psychology of
the telegraphic language. Psychological Review, 4(1), 27–53
Bryan, W. L. & Harter, N. (1899). Studies on the telegraphic language: the
acquisition of a hierarchy of habits. Psychological Review, 6(4), 345–375

5. PERFORMANCE GENERALLY IMPROVES WITH
PRACTICE . . . EXCEPT WHEN IT DOESN’T!
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 5 / 30
Hypothesized that telegraphic expertise consisted of a hierarchy
of habits.
Plateaus were periods in which elements at one level of the
hierarchy were being combined so as to be used at a higher level.
dots and dashes → letters → words → phrases
But . . .

6. THESE PLATEAUS COULD LAST A LONG LONG TIME
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 6 / 30
E. L. Thorndike
“I venture to prophesy that the thousand
bookkeepers in, say, the grocery stores of New
York who have each had a thousand hours of
practice at addition, are still, on the average,
adding less than two-thirds as rapidly as they
could, and making twice as many errors as they
would at their limit.”
“It appears likely that the majority of teachers make no gain in
efficiency after their third year of service, but I am confident that
the majority of such teachers could teach very much better than
they do.”
“It seems to me therefore that mental training in schools, in
industry and in morals is characterized, over and over and over
again, by spurious limits – by levels or plateaus of efficiency
which could be surpassed.”
CITATION
Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology of
Learning. NYC: Teachers College, Columbia University

7. PLAN FOR THIS TALK
Skip the remainder of the first 90 years (1897 – 1987) of scientific
research on expert performance.
Jump to 1987 to Carroll & Rosson’s Paradox of the Active User and
Ericsson’s (1993) Deliberate Practice and the view shared by both
that expert is not good enough.
Three types of performance asymptotes and one type of plateau.
Summarize everything in time for questions!
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 7 / 30

8. Outline
1 Mere Expertise is Not Good Enough
2 Plateaus and Asymptotes
3 Outside the Lab: Good → Better → Best!
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 8 / 30

9. OUTSIDE THE LAB – THE PARADOX OF THE ACTIVE
USER
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 9 / 30
Jack Carroll Mary Beth Rosson
In 1987, Carroll and Rosson coined the term,
Paradox of the Active User, to refer to the
“suboptimal use of office productivity software”
by people who use the systems daily across the
course of weeks, months, and years.
Carroll and Rosson, who at that time worked for
the IBM Watson Research Center, shared the
HCI community’s concern that the expected
productivity gains of the computer revolution
were not occurring.

10. MERE EXPERTISE IS NOT GOOD ENOUGH
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 10 / 30
Anders Ericsson
A few years later, based on his studies of human
expertise, Ericsson (1993) concluded that, “the
belief that a sufficient amount of experience or
practice leads to maximal performance appears
incorrect”.
CITATION
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of
deliberate practice in the acquisition of expert performance. Psychological
Review, 100(3), 363–406

11. LIMITS TO EXPERTISE
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 11 / 30
After years of lurking in the background, the
plateau had returned to front and center.

12. RESOLVING THE PARADOX OF THE ACTIVE USER
Suboptimal performance can be amazingly stable! Fu and Gray (2004).
What is optimal?
Is it optimal to learn 100 different commands and procedures that
each do one thing very well (fast, precisely, . . . )?
Is it optimal to learn one command and procedure that can be
tweaked into doing 100 different things? (slowly, approximately, . . . )
Breaking a habit – the case of hunt & peck versus touchtyping
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 12 / 30
CITATION
Fu, W.-T. & Gray, W. D. (2004). Resolving the paradox of the active user:
Stable suboptimal performance in interactive tasks. Cognitive Science,
28(6), 901–935

13. Outline
1 Mere Expertise is Not Good Enough
2 Plateaus and Asymptotes
3 Outside the Lab: Good → Better → Best!
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 13 / 30

14. PLATEAUS VERSUS ASYMPTOTES: POLE VAULTING
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 14 / 30
A history of technological innovations as the composition of the
pole changed from ash (wood), to bamboo (wood), to
fiberglass/carbon.
Each technology, enabled pole vaulters to break new records
Followed by invention of new methods that resulted in new
rounds of record breaking as those methods were adopted and
adapted by athletes.
Asymptote → New technology → New methods → Asymptote

15. PLATEAUS VERSUS ASYMPTOTES: HIGH JUMPING
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 15 / 30
A history of innovations in methods.
Method – Fosbury Flop could have been invented earlier.
Plateau → New Method → New Plateau
The Scissors and Straddle technique for high jumping.
The Fosbury Flop technique for high jumping.

16. PLATEAUS AND ASYMPTOTES
How do plateaus and asymptotes come about??
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 16 / 30

17. ASYMPTOTE DUE TO ARTIFACT DESIGN
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 17 / 30
Crossman’s (1959) study of cigar rollers in Cuba. Plot shows a continued
increase in performance over a two year period (estimated as 3 million
cigars) and then a flattening of the curve.
Newell and Rosenbloom (1981, p. 7) attribute this flattening to a “known
lower bound for the performance time” in this task; namely, the “cycle time
of the machine.”

18. ASYMPTOTE DUE TO SYSTEM DESIGN
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 18 / 30
A field trial of two workstations for Telephone Operators
(Gray, John, & Atwood, 1993).
Expected call times to decrease across the 4-mon trial.
But after 2-mon worktimes stabilized with times per call
slower than for the old workstation
Slow enough to increase annual operating costs by $6.2
million (in 2014 dollars).
Diagnosis:
Designers believed call time driven by the # of keys-per-call.
Predicted savings of 4.1 s in mean item per call for annual savings of $24m.
BUT cognitive modeling showed that old workstation enabled Operators to
interleave keypresses, chats with customer, and wait time for external
databases.
Conclusion:
Based on the models, Operators were becoming more expert at the new
workstation but asymptotes due to systems design prevented these gains in
expertise from yielding performance increments.

19. ASYMPTOTE DUE TO MEASUREMENT METHOD
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 19 / 30
Example from Space Fortress in which a measurement method was introduced
about 1994 that has affected many of the conclusions reached by researchers ever
since.
To simplify the story . . . changed scoring so that it included 4 component scores
and one overall score. These measures can be shown to be (1) not independent of
each other and (2) two of these measures asymptote even as skilled performance
increases.

20. ASYMPTOTE OR PLATEAU? THE CASE OF DIGIT SPAN
The Digit Span Task – An important part of the Wechsler Adult
Intelligence Scale (WAIS) IQ test (and others).
Digits (0-9) are read at the rate of 1 per sec.
Followed immediately by ordered recall.
If all digits were recalled correctly, the length of the next run of digits
was increased by 1.
If all are not correct, the next run is decreased by 1.
The population norm is 7 ± 2.
Is this an asymptote due to limitations built into the human brain? or
Is this a plateau due to massive stable suboptimal performance on the
part of the entire human population?
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 20 / 30

21. ASYMPTOTE OR PLATEAU? THE CASE OF DIGIT SPAN
Well? . . . This is an IQ test item!! Therefore it MUST be measuring an
individual difference variable that differs between humans but is stable, or
asymptotic, for any given individual.
Right . . . ?
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 21 / 30

22. IS HUMAN DIGIT SPAN A POPULATION ASYMPTOTE
OR PLATEAU? – PLATEAU!!
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 22 / 30
The difference between a plateau and asymptote is made clear by the
existence of extreme experts with a known history of transcending the
plateau.
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
SFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSF
10
20
30
40
50
60
70
80
0 10 20 30 40 50
Practice (in 5 day blocks)
DigitSpan
Perhaps the human asymptote is 80 ± 2 instead. . . ????

23. Strategy Plateaus and the Paradox of the Active User.
Issue: The difference between a plateau and an asymptote may be
hard to determine.
Asymptotes may reflect a problem that can be fixed by design
(whether artifact design or system design); however, plateaus due to
strategy-induced suboptimality may arise when the strategies deployed
do not enable utility maximization in the task environment.
Overcoming such strategy-induced suboptimality is usually very
difficult.
Example of transfer from visually-guided to touch typing.
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 23 / 30

24. Outline
1 Mere Expertise is Not Good Enough
2 Plateaus and Asymptotes
3 Outside the Lab: Good → Better → Best!
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 24 / 30

25. Real World Tasks?
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 25 / 30

26. Real World Tasks?
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 26 / 30

27. Real World Tasks?
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 26 / 30

28. Outside the Lab – Resolving the Paradox of the Active User
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 27 / 30
If “most people and professionals reach a stable performance
asymptote within a limited time period” (Ericsson, 2004), then
practice does not make perfect.
Unless, as a society, we can be content with “stable suboptimal
performance plateaus” then something more is needed. We
suggest that this “something more” is research into the
acquisition of expertise in mundane (i.e., everyday) task
environments.

29. Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 28 / 30
Thank You!!

30. REFERENCES I
Bryan, W. L. & Harter, N. (1897). Studies in the physiology and psychology
of the telegraphic language. Psychological Review, 4(1), 27–53.
Bryan, W. L. & Harter, N. (1899). Studies on the telegraphic language: the
acquisition of a hierarchy of habits. Psychological Review, 6(4),
345–375.
Crossman, E. R. F. W. (1959). A theory of the acquisition of speed-skill.
Ergonomics, 2(2), 153–166.
Ericsson, K. A. (2004). Deliberate practice and the acquisition and
maintenance of expert performance in medicine and related domains.
Academic Medicine, 79(10, S), S70–S81.
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of
deliberate practice in the acquisition of expert performance.
Psychological Review, 100(3), 363–406.
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 29 / 30

31. REFERENCES II
Fu, W.-T. & Gray, W. D. (2004). Resolving the paradox of the active user:
Stable suboptimal performance in interactive tasks. Cognitive Science,
28(6), 901–935.
Gopher, D., Weil, M., & Bareket, T. (1994). Transfer of skill from a
computer game trainer to flight. Human Factors, 36(3), 387–405.
Gray, W. D., John, B. E., & Atwood, M. E. (1993). Project Ernestine:
Validating a GOMS analysis for predicting and explaining real-world
performance. Human-Computer Interaction, 8(3), 237–309.
Newell, A. & Rosenbloom, P. S. (1981). Mechanisms of skill acquisition and
the law of practice. In J. R. Anderson (Ed.), Cognitive skills and their
acquisition (pp. 1–55). Hillsdale, NJ: Lawrence Erlbaum Associates.
Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology of
Learning. NYC: Teachers College, Columbia University.
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 30 / 30