Talk for the women+@DCS Sheffield University, UK Title: Natural Language Inference for Humans Valeria de Paiva, Topos Institute, Berkeley, USA Abstract: One hears much about the incredible results of recent neural nets methods in NLP. In particular much has been made of the results on the Natural Language Inference task using the huge new corpora SNLI, MultiNLI, SciTail, etc, constructed since 2015. Wanting to join in the fun, we decided to check the results on the corpus SICK (Sentences Involving Compositional Knowledge), which is two orders of magnitude smaller than SLNI and presumably easier to deal with. We discovered that there were many results that did not agree with our intuitions. As a result, we have written so far five papers on the subject (with another one submitted to COLING2020). I want to show you a potted summary of this work, to explain why we think this work is not near completion yet and how we're planning to tackle it. This is work with Katerina Kalouli, Livy Real, Annebeth Buis and Martha Palmer. The papers are Explaining Simple Natural Language Inference. Proceedings of the 13th Linguistic Annotation Workshop (LAW 2019), 01 August 2019. ACL 2019, WordNet for “Easy” Textual Inferences. Proceedings of the Globalex Workshop, associated with LREC 2018 Graph Knowledge Representations for SICK. informal Proc of 5th Workshop on Natural Language and Computer Science, Oxford, UK, 08 July 2018 Textual Inference: getting logic from humans. Proc of the 12th International Conference on Computational Semantics (IWCS), 22 September 2017 Correcting Contradictions. Proc of Computing Natural Language Inference Workshop (CONLI 2017) @IWCS 2017

1. 1/23
Introduction
NL Inference
Women in Computer Science
Natural Language Inference for Humans
Valeria de Paiva
Women+@DCS Sheffield
July 2020
Valeria de Paiva Women+@DCS

2. 2/23
Introduction
NL Inference
Women in Computer Science
Thanks, Aline!
Valeria de Paiva Women+@DCS

3. 3/23
Introduction
NL Inference
Women in Computer Science
Personal stories
I’m a logician, a proof-theorist, a computational semanticist and a
category theorist.
I work in industry in Silicon Valley, have done so for the last 20
years, applying the purest of pure mathematics, in surprising ways.
Valeria de Paiva Women+@DCS

4. 4/23
Introduction
NL Inference
Women in Computer Science
Personal stories
Valeria de Paiva Women+@DCS

5. 5/23
Introduction
NL Inference
Women in Computer Science
PARC, XLE and Bridge
Valeria de Paiva Women+@DCS

6. 6/23
Introduction
NL Inference
Women in Computer Science
Powerset, Cuil and Nuance
Valeria de Paiva Women+@DCS

7. 7/23
Introduction
NL Inference
Women in Computer Science
Natural Language Inference (NLI)
A shock when the work of almost a decade at PARC was out
of reach when I left in 2008
I gave a talk at SRI proposing to redo it all, open source (de
Paiva 2010 Bridges)
Pleased to report that almost all of it is now available
open-source
Most work with/by Katerina Kalouli, PhD student at
Konstanz
Valeria de Paiva Women+@DCS

8. 8/23
Introduction
NL Inference
Women in Computer Science
Natural Language Inference: why?
In May 2016 Google announced Parsey McParseface, the
world’s most accurate parser1: 94% accuracy
In 2014 Marelli et al launched the SICK corpus at SemEval
2014: an easy (no named entities, no temporal phenomena,
limited vocabulary, etc..), linguist curated corpus to test
compositional knowledge
Can we use SyntaxNet to process SICK with off-the-shelf
tools such as WordNet and SUMO?
It’s complicated! Five papers and counting!
1
ai.googleblog.com/2016/0/announcing-syntaxnet-worlds-most.
html
Valeria de Paiva Women+@DCS

9. 9/23
Introduction
NL Inference
Women in Computer Science
Natural Language Inference: what?
Examples from SNLI dataset at Stanford
Valeria de Paiva Women+@DCS

10. 10/23
Introduction
NL Inference
Women in Computer Science
NLI for Humans
Easier to detect inference than to decide on “good”semantic
representations
Data-driven NLU need large, diverse, high-quality corpora
annotated to learn inference relations: entails, contradicts,
neutral
Can we trust the corpora we have?
Are they really learning logical inferences?
Are the findings on the big corpora available SNLI, MNLI,
SciTail, etc transferable and generalizable? (Plenty of recent
work showing no, systems learn biases of the corpora, cannot
be redeployed)
Valeria de Paiva Women+@DCS

11. 11/23
Introduction
NL Inference
Women in Computer Science
NLI for SICK
Explaining Simple
Natural Language
Inference ACL2019
Textual Inference:
getting logic from
humans IWCS2017
Correcting
Contradictions,
CONLI 2017
Graph Knowledge
Representations for
SICK, NLCS2018
WordNet for “Easy”
Textual Inferences
LREC2018
Valeria de Paiva Women+@DCS

12. 12/23
Introduction
NL Inference
Women in Computer Science
NLI for SICK
Are the annotations in SICK logical? Can we trust them?
Several problems: lack of guidelines on co-reference, how to
annotate contradictions, ungrammatical and non-sensical
sentences, noisy data, etc..
This meant contradictions in SICK are not symmetric and
they need to be
Contradictions require alignment between entities and events,
which need to be ”close enough”
how to decide when things are close enough?
Can we do simpler case where sentences are
”one-word-apart”using WordNet?
More guidelines necessary for SICK annotation?
Valeria de Paiva Women+@DCS

13. 13/23
Introduction
NL Inference
Women in Computer Science
NLI for SICK
https://logic-forall.blogspot.com/2020/03/
sick-dataset-in-these-trying-times.html
Valeria de Paiva Women+@DCS

14. 14/23
Introduction
NL Inference
Women in Computer Science
Are we there yet?
Manning: Computational Linguistics and Deep Learning, 2015
”NLP is kind of like a rabbit in the headlights of the Deep
Learning machine, waiting to be flattened.”
Hinton 2015: ”I will be disappointed if in five years’ time we do
not have something that can watch a YouTube video and tell a
story about what happened.”
[not totally flattened, yet]
Valeria de Paiva Women+@DCS

15. 15/23
Introduction
NL Inference
Women in Computer Science
Conclusions so far
Working for division of semantic labor between
symbolic/structural and distributional approaches
Have fledgling proposal GKR with strict separation of
conceptual and contextual structures
Also concrete proposal for injecting distributionality in GKR:
promising results (COLING submission)
Further Work: Still working to produce a ‘correct’ SICK
Working on annotations and theorem provers
test GKR with further datasets, further distributional
architectures
Valeria de Paiva Women+@DCS

16. 16/23
Introduction
NL Inference
Women in Computer Science
4th Workshop Women in Logic 2020
Valeria de Paiva Women+@DCS

17. 17/23
Introduction
NL Inference
Women in Computer Science
Women in Computer Science
I grew up believing most of the gender wars had been fought
by our grandmothers, suffragettes or not.
that the law allowed me to get into colleges and work places.
that I could always apply for scholarships and grants. I had
plenty of women teachers.
I thought my job was to work hard and show people I could
do the job as well as any man
I knew the numbers were bad both in Computing and in
Maths, but I thought they’re bad as usual, not particularly
bad.
That time would be on our side, that things were going to get
more equal as time went by
Valeria de Paiva Women+@DCS

18. 18/23
Introduction
NL Inference
Women in Computer Science
Women in Computer Science
Valeria de Paiva Women+@DCS

19. 19/23
Introduction
NL Inference
Women in Computer Science
Women in Computer Science
Valeria de Paiva Women+@DCS

20. 20/23
Introduction
NL Inference
Women in Computer Science
Women in Computer Science
When Nat Shankar asked me if I wanted to say a few words about
Logic in Computer Science, in its 30th birthday, I warned him that
he might not like the few words.
Then we launched the Workshop Women in Logic, the facebook
group Women in Logic and the blog.
Valeria de Paiva Women+@DCS

21. 21/23
Introduction
NL Inference
Women in Computer Science
Women in Computer Science
Workshops in Iceland, UK, Canada and this year Paris, France.
Funding for scholarships from SIGLOG, VCLA (Vienna Center for
Logic and Algorithms), and ILLC (institute for Language, Logic,
and Computation), Amsterdam, Netherlands.
Valeria de Paiva Women+@DCS

22. 22/23
Introduction
NL Inference
Women in Computer Science
Women in Computer Science Data
We have a spreadsheet of women logicians, editable by
everyone, since 2012.
A collection of spreadsheets checking numbers of female
Invited Speakers in many of the theoretical Computer Science
main conferences.
Careful work on number of women invited speakers for the
ASL meetings (thanks Johanna Franklin!)
Have a mailing list and many plans. Join us!
Valeria de Paiva Women+@DCS

23. 23/23
Introduction
NL Inference
Women in Computer Science
More information
GKR Demo:
http://lap0973.sprachwiss.uni-konstanz.de:
8080/sem.mapper/
GKR source code:
https://github.com/kkalouli/GKR_semantic_parser
Ask KAterina questions!
Play with it and tells us all the other things we haven’t done, yet!
Thanks!
Valeria de Paiva Women+@DCS