Wlodek Zadronzny's presentation for Cognitive Systems Institute Group Speaker Series March 12, 2015.

1. IBM Watson in the
Classroom
Wlodek Zadrozny (UNC Charlotte/formerly IBM Research)
Sean Gallagher (UNC Charlotte)
Watson Polymath Ideas
Valeria de Paiva (Nuance)
Lawrence S. Moss (Indiana University)
WatsonSim development
Walid Shalaby
Adarsh Avadhani and others

2. Background papers

3. Similar work on Watson in the classroom
RPI
Columbia U.
UT Austin
CMU(?)
???

4. “Watson became possibly the first nonhuman
millionaire by besting its human competition”
Arguably, this event started the era of “cognitive
computing”

5. The Jeopardy! Challenge: Solved in 2011
No replication of the solution as of 2015
Broad/Open
Domain
Complex
Language
High
Precision
Accurate
Confidence
High
Speed
$600
In cell division, mitosis
splits the nucleus &
cytokinesis splits this
liquid cushioning the
nucleus
$200
If you're standing, it's the
direction you should
look to check out the
wainscoting.
$2000
Of the 4 countries in the
world that the U.S. does
not have diplomatic
relations with, the one
that’s farthest north
$1000
The first person
mentioned by name in
‘The Man in the Iron
Mask’ is this hero of a
previous book by the
same author.
Based on a slide from IBM

6. Two Research Challenges:
Replication of IBM Watson performance
Understanding why Watson works.
Watson heterogeneity
is perfect for
introducing students to
IR and NLP
Opportunity for MS and
Undergrad Student
Research
Data Sets are
Available
on
J-Archive
Deeper Research
Questions:
We know how Watson
works, but we don’t know
why it works
Watson Architecture
was described in
details
in
IBM J. R&D +
patents

7. Watson in the classroom: Spring 2014
 Motivation: teaching computer science using solved challenges
 Content: “Semantic Technologies in IBM Watson”
(provided by IBM)
 Students (20): Upper level undergrad(1/3), MS(2/3), 1PhD
 Since we didn’t have any code we decided to build a Watson simulator
Students followed the idea of IBM Watson architecture,
simplifying whenever possible, e.g. no UIMA
 Used as a way to learn:
Information Retrieval (Bing, Google, Lucene, Indri)
Elements of machine learning (using Weka, logistic regression, )
Elements of NLP: why NLU is difficult, POS tagging,
parsing (with OpenNLP)
Data preparation: regular expressions, polite data crawling, etc.
…

8. Teaching Objective: Learning IR and NLP
All students should learn all the technologies involved.
Grading should be based on the degree of mastery.

9. The Resulting System
as of Spring 2014
Shaded components were more complete.

10. WatsonSim Components Changed

11. Watsonsim Accuracy
- Bing, Lucene and Indri as search
- Wikipedia, Wikiquotes, and Shakespeare as sources
- Using n-gram scores, parse tree comparisons, LAT matching
- SVM based score aggregation

12. Integrating contributions of different
teams was often challenging
- We logged over 3500 runs, recording accuracies
- The peak is around 26.6% top accuracy
- 36.6% for the top three candidates

13. Current Status: Individual Study
 Started the Watson MOOC
 Reading Watson papers
 Extending WatsonSim as a practicum for individual studies
 Adding new scorers
 Adding question analyzers and classifiers
 Adding new sources
 Code available on github:
https://github.com/SeanTater/uncc2014watsonsim

14. Plans
 Cognitive computing class
 Use as a vehicle for individual study
and undergrad/MS research
 Research (proposal) around the “why” question

15. Two Research Challenges:
Replication of IBM Watson performance
Understanding why Watson works.
Deeper Research
Questions:
We know how Watson
works, but we don’t know
why it works
Main points:
1. Despite IBM Watson’s
success, we don’t know
why it works?
2. Figuring it out can be an
open collaborative project:
Polymath style
?

16. Why should the NLP researchers care?
 We have a mismatch between the academic/scientific
theory of meaning and NLU and what technical
experience seems to be telling us
 Inadequate theories might be limiting our ability to
make progress in NLP
 There are interesting research questions, given Watson
unorthodox approach to QA

17. A challenge in representing meaning
of text?
Schuetze 2013 argues that
“meaning is heterogeneous and
that semantic theory will always
consist of distinct modules that
are governed by different
principles”
 We don’t have a formal
theory that would support this
view
 A formal model of Watson
might be a starting point
The(?) meaning is constructed in multiple steps

18. Some questions for a formal model
of Watson?
 What is the role of search in computing meaning?
In Watson, formally speaking, it constrains the entities that end
up in the correct discourse model.
Can any collection of text be reorganized that way? Dynamically?
 Semantics of the deferred type evaluation/deferred meaning
computation?
 Formal model of scoring?
Meaning through interaction? Or intersection of constraints?
 Is it Watson-like model fundamental? Or an engineering feat?

19. Other sample research topics
 Role of formal semantics and theorem proving
“the overall performance of QA systems is directly
related to the depth of NLP resources”
“the prover boosts the performance of the QA system on TREC
questions by 30%” D.Moldovan in 2003
Similar results reported by MacCartney and Manning 2009
 Could interactive theorem provers, e.g. Coq and HOL, be
adapted to improve performance of QA systems, and NLP systems
in general? (Same question for automated provers).
 Role of natural logics

20. Conclusions
 Formal model of Watson might help in building more
realistic theories of natural language understanding
 Polymath model might be useful in creating a formal theory/model of
Watson
 More realistic theories might help in building better NLP
systems
 Experiments with Watson-models are possible
 Now, experiments with IBM Watson in the ‘cloud’ are possible
 Watson can be the basis for teaching NLP