FORR is a cognitive architecture that allows agents to develop expertise through experience. It uses multiple knowledge representations and heuristics called "Advisors" to make decisions. As an agent solves problems, it learns which Advisors are most useful and how to best combine their recommendations. This allows FORR-based agents to improve their performance and adapt their decision-making process. FORR has been successfully applied to develop agents that can play games, navigate environments, solve constraint satisfaction problems, and collaborate as teams.

1. FORR
A Cognitive Architecture for Expertise
Susan L. Epstein
The Graduate Center and Hunter College of
The City University of New York

2. Executive summary
• FORR (FOr the Right Reasons) is an architecture
• FORR-based systems develop expertise
• FORR-based systems learn quickly from problem solving experience
• FORR-based systems are built from
§ World knowledge (descriptives)
§ Good reasons for making decisions (Advisors)
• FORR-based systems can restructure their decision process
• FORR confirmed cognitively plausible on human subjects
2Background • FORR • Applications

3. People, agents and expertise
• People are our best model of intelligent agents
§ Some human approaches work well on really hard problems
§ Their methods are robust to imperfect data
§ They pursue multiple goals
• If an agent is to collaborate with people, it is necessary to understand
human decision processes
• A cognitively plausible agent simulates significant human characteristics
• Expert does things faster and better than the rest of us [D’Andrade 1990]
3Background • FORR • Applications

4. Characteristics of human experts
• They work in a domain (set of related problem classes)
• They satisfice = make good enough decisions
• They entertain multiple decision-making heuristics [Ratterman
& Epstein 1995]
• They access multiple representations
• They do situation-based reasoning [Klein & Calderwood 1991]
• Human experts are made, not born
4
Learning is the hallmark of human intelligence
Background • FORR • Applications

5. Agent architecture
• Postulates general principles
• System shell for diverse domains
• Requirements for cognitive plausibility
§ Display reasonable behavior
• Make obvious decisions
• Avoid obvious errors
• Solve easy problems quickly
§ Balance accuracy and speed
§ Be robust to error
§ Tolerate and reason with inconsistent, incomplete, noisy data
§ Learn
5Background • FORR • Applications
Do forever
Sense the world
Select an action
Execute that action

6. Fundamental issues for a learning architecture
• What is there to learn?
• From whom to learn?
• When to learn?
• How to learn?
• How to use learned knowledge to make decisions?
• How to manage reality and noise?
6Background • FORR • Applications

7. Cornerstones of FORR’s pragmatic approach
• Expertise is learned, that is, it develops with experience
• Easy questions should have fast (reactive) answers
• Satisfice = make good enough decisions in a simplified model of a complex
world (and recover if need be)
• Exploit synergy inherent in multiplicity
§ Multiple domain-dependent representations
§ Multiple domain-dependent heuristics for decision making
§ Multiple learning methods
• Maintain flexibility
§ Decouple data, learning methods, and decision methods
§ Restructure its own decision-making process
• Transparency: explain decisions
FORR's building blocks are descriptives and Advisors
7Background • FORR • Applications

8. Multiple representations
• Descriptive = a shared data object
§ Value provided on demand
§ Defined with functions that determine how and when to update it
§ Value may be learned
• Although a descriptive has a single representation, many descriptives can
represent the same world state
• Examples:
X-O-blank
empty/occupied
lines on the board
8Background • FORR • Applications

9. Multiple ways to use knowledge
• Operationalization = how to use a data object
• Although a descriptive has a single representation, it can be
operationalized in many ways
Ways to reason about the empty/occupied squares
Calculate possible actions
Predict opponent's move
Ways to reason about the lines
Report a result
Finish a winning line
Block your opponent’s winning line
Create a fork
Plan a win on a specific line
9Background • FORR • Applications

10. An Advisor operationalizes descriptives
• Implements a class-independent, action-selection rationale
• Limitedly-rational (resource-limited) procedure
• Input: state of the world + descriptives + possible actions
• Output: comments whose strengths express intensity of support or
opposition to individual actions (or sets of actions)
• Domain-specific
< Advisor, action, strength>
Advisor
current state possible actions
relevant descriptives
10Background • FORR • Applications

11. Often, Advisors disagree
11
O
X X
O Panic
(prevent
immediate
loss)
Worried
(prevent long-
range loss)
Victory
(win!)
And rely on learned descriptives
• Good openings
• Endgame play
• Strategies
…
Background • FORR • Applications

12. More about Advisors
• Advisors have different properties
§ Some are always right
§ Some need more time to decide
§ Some would like to make a sequence of decisions, not just one
• Comments are opinions from the perspective of the Advisor's rationale
§ On a single action
do x x is better than y
don’t do z do x or y
x is a 10, y is an 8, but z is a –3
§ On an (unordered or fully or partially ordered) set of actions
do x and y do p and then q
do p and then do q and r
12Background • FORR • Applications

13. FORR (FOr the Right Reasons)
• Premise: synergy among domain-specific rationales solves problems
• Descriptives isolate representation from reasoning
• Advisor hierarchy
§ Tier 1: correct, quick, pre-sequenced
§ Tier 2: reactive plan rationales
§ Tier 3: voting among heuristics based on
their comment strengths and learned weights
13
<AdvisorA, action2, 10>
<AdvisorA, action4, 8>
<AdvisorA, action7, 6>
<AdvisorB, action2, 7>
<AdvisorB, action3, 9>
<AdvisorC, action1, 9>
<AdvisorC, action2, 7>
<AdvisorC, action3, 9>
<AdvisorC, action7, 9>
…
Voting
For Advisor i and action j
argmax
j
diwicij∑
Background • FORR • Applications

14. The FORR decision cycle
14
take actionyes
Tier 1: Reaction from
perfect knowledge
Victory T-11 T-1n…
Decision?
no
Background • FORR • Applications
state actions
descriptives

15. The FORR decision cycle
15
take actionyes
Tier 1: Reaction from
perfect knowledge
Victory T-11 T-1n…
Decision?
begin planyes
Tier 2: Plans triggered by
situation recognition
no
T-21 T-22 T-2m…
Decision?
Background • FORR • Applications
state actions
descriptives

16. The FORR decision cycle
16
take actionyes
Tier 1: Reaction from
perfect knowledge
Victory T-11 T-1n…
Decision?
begin planyes
no
T-32T-31 T-3k…
…Tier 3: Heuristic reactions
Voting take action
Tier 2: Plans triggered by
situation recognition
no
T-21 T-22 T-2m…
Decision?
Background • FORR • Applications
state actions
descriptives

17. How to develop a problem solver
17
• Specialize FORR with domain knowledge
§ Problem classes
§ Advisors
§ Descriptives with learning methods
• To solve a class of problems robustly, FORR learns
§ Descriptives’ values
§ Rationales’ relative utility
§ New Advisors
§ How to reorganize tier 3
Domain
knowledge
FORR
FORR-based
problem solver
Learned
problem solver
Problem class
Experience
WARNING: problem solving
often provides noisy data
Background • FORR • Applications

18. FORR-based single agents
18
• Hoyle learned to play 19 two-person, perfect-information, finite-board
games as well or better than human / machine expert [Epstein, 2001]
• Ariadne learned to navigate efficiently in grid worlds, despite perceptual
limitations and no map [Epstein, 1995]
• ACE learned to solve constraint satisfaction problems and rediscovered
the Brélaz heuristic [ Epstein & Freuder, 2005]
• SemaFORR: controls an autonomous search-and-rescue robot [Epstein,
Schneider, Ozgelen, Munoz, Costantino, Sklar & Parsons, 2012]
Background • FORR • Applications

19. Lessons learned
19
• Reactive plans work well
• Elimination of inaccurate heuristics produces substantial speedup
• Lazy descriptive computation also provides speedup
• Self-awareness supports transparency
• Advisor weights may have problem-stage context
• Weight learning has subtle pitfalls (example extraction)
• Autonomous restructuring must balance accuracy against risk
• Sometimes it is more efficient not to reason at all
Background • FORR • Applications

20. FORR-based collaborating agents
20
• Co-FORR: 5 collaborating agents for 2D park design [Epstein, 1998]
• FORRSooth: learned to conduct a spoken dialogue with a library patron
who orders books [Epstein, Passonneau, Gordon, & Ligorio, 2012]
• SemaFORR: controls autonomous search-and-rescue robot team [Epstein,
Aroor, Evanusa, Sklar & Parsons, 2015]
Each new domain poses new challenges whose solution strengthens FORR
Background • FORR • Applications

21. FORR-based results
21
• PhD theses
§ Shih on learning multiple behavior sequences, 2000
§ Lock on learning multiple plans from behavior sequences, 2003
§ Petrovic on weight learning for multiple Advisors, 2008
§ Ligorio on learning to select attributes, 2011
§ Li on representation and exploitation of multiple complex relationships, 2011
§ Yun on parallelization of multiple solvers, 2013
§ Osisek on application of multiple relationships in recommendation (in progress)
§ Aroor on reactive planning for multiple robots (in progress)
• Applications to bioinformatics (with Dr. Lei Xie)
§ Protein-protein interaction networks
§ Virtual drug screening
Background • FORR • Applications

22. Take home message
To develop expertise
FORR learns to harness the synergy of
multiplicity in representation and reasoning
22Background • FORR • Applications

23. Acknowledgements
We gratefully acknowledge the support of
The National Science Foundation
CUNY’s High Performance Computing Center
Continued thanks to my collaborators
Gene Freuder Rebecca Passonneau
Rick Wallace Lei Xie
Elizabeth Sklar Simon Parsons
and a host of undergraduate and graduate students
with whom I continue to learn
23

24. Selected references
• Epstein, S. L. 2001. Learning to Play Expertly: A Tutorial on Hoyle. In
Machine Learning in Game Playing
• Epstein, S. L. 1998. Pragmatic Navigation: Reactivity, Heuristics, and
Search. Artificial Intelligence, 100 (1-2): 275-322.
• Epstein, S. L., E. C. Freuder and M. Wallace 2005. Learning to Support
Constraint Programmers. Computational Intelligence 21(4): 337-371.
• Epstein, S. L., R. J. Passonneau, T. Ligorio and J. Gordon. 2012. Data
Mining to Support Human-Machine Dialogue for Autonomous Agents. In
Proceedings of Agents and Data Mining Interaction (ADMI2011).
• Epstein, S.L., Aroor, A., Evanusa, M., Sklar, E.I., Simon, S. 2015. Navigation
with Learned Spatial Affordances. In Proceedings of CogSci 2015.
http://www.cs.hunter.cuny.edu/~epstein/
24