Presented at the Unshared Task at the 3rd Workshop on Argument Mining

1. Perspective Based Local
Agreement and Disagreement
in Online Debate
1
Chantal van Son, Tommaso Caselli, Antske Fokkens,
Isa Maks, Roser Morante, Lora Aroyo and Piek Vossen
Unshared Task at the 3rd Workshop on Argument Mining

2. Agreement and disagreement
• Local (dis)agreement:
• Global: overall stance towards debate topic
• Local: between posts, segments, sentences, or (micro-)propositions
• Perspective based (dis)agreement:
• Texts are collections of (micro-)propositions associated with perspective
values (e.g. sentiment, factuality)
• Agreement: a correspondence between one or more perspective values
of a (micro-)proposition
• Disagreement: a divergence between them
2

3. 3
Example (1)
A: I’m so happy that Trump might be elected as the next president of the US!
B: Yes he might, but do you really think that’s a good thing?
elected
as next
president
Trump

4. 4
Example (1)
A: I’m so happy that Trump might be elected as the next president of the US!
B: Yes he might, but do you really think that’s a good thing?
elected
as next
president
Trump
FACTUALITY:
AFFIRMATIVE
POSSIBLE
FUTURE
Person A Person B
SENTIMENT:
NEGATIVE
FACTUALITY:
AFFIRMATIVE
POSSIBLE
FUTURE
SENTIMENT:
POSITIVE
FACTUALITY: AGREEMENT SENTIMENT: DISAGREEMENT

5. 5
Example (2)
Player A: I think that Miss Scarlett was killed by Colonel Mustard with a rope.
Player B: I think that Miss Scarlett was killed by Colonel Mustard with a candlestick.
killed
by Colonel
Mustard
Miss
Scarlett
with a
rope

6. 6
Example (2)
Player A: I think that Miss Scarlett was killed by Colonel Mustard with a rope.
Player B: I think that Miss Scarlett was killed by Colonel Mustard with a candlestick.
AGREEMENT
killed
by Colonel
Mustard
Miss
Scarlett
with a
rope
AFFIRMATIVE
CERTAIN
NON-FUTURE
AFFIRMATIVE
CERTAIN
NON-FUTURE
Player A Player B

7. 7
Example (2)
Player A: I think that Miss Scarlett was killed by Colonel Mustard with a rope.
Player B: I think that Miss Scarlett was killed by Colonel Mustard with a candlestick.
killed
by Colonel
Mustard
Miss
Scarlett
with a
rope
AFFIRMATIVE
CERTAIN
NON-FUTURE
NEGATIVE
CERTAIN
NON-FUTURE
Player A Player B
DISAGREEMENT

8. Data sets
• Variant C: Editorial articles from Room for Debate
(website N.Y. Times)
Debate title: Birth Control on Demand
Debate description: Should it be provided by the government
to reduce teen pregnancies?
Article title: Publicly Funded Birth Control Is Crucial
• Variant D: Discussions containing comments on
these articles
8

9. Annotation procedure
1) identifying micro-propositions
2) detecting (dis)agreement
9

10. Annotation procedure
1) identifying relevant text
2) identifying micro-propositions
3) detecting (dis)agreement
10

11. Annotation procedure
1) identifying relevant text
2) identifying micro-propositions
3) detecting (dis)agreement
11

12. Pilot Annotation
A sentence is defined to be commented
upon if:
• the comment repeats or rephrases (part
of) a statement made in the sentence;
• the comment attacks or supports (part
of) a statement made in the sentence.
12
5 expert annotators
8 articles
62 comments
1.186 sentences
2 rounds of annotation
Identify those sentences in the editorial article
that are commented upon in the comment

13. Distribution of Annotations
13
sentences(n)
0
35
70
105
140
annotators (n)
1 2 3 4 5
43
32
47
83
139

14. Strong, unambiguous relations
• ARTICLE:
But allowing children and teens to regulate their
behavior like adults gives them room to naturally
modify their own habits.
• COMMENT:
I empathize with your argument that allows children
to regulate their behavior like adults.
14
annotated by 5 out of 5

15. Less strong, requires inference
• ARTICLE:
Giving poor young women easy access to birth
control is about exactly that - control.
• COMMENT:
This is a rational argument for how access to birth
control is a win-win for young women, their partners,
and the taxpaying public who might otherwise foot
the welfare bill.
15
annotated by 2 out of 5

16. Background information/anecdotes
• COMMENT:
Years ago, during a classroom visit, I observed a
small group of black and Latino high school boys
sitting at their desks looking into handheld mirrors.
They were tasked with answering the question, “What
do you see?”
One boy said, “I see an ugly face.”
Another said, “I see a big nose.’"
16
annotated by 0 out of 5
(for none of the comments)

17. Conclusion
• Provided a more precise definition of local (dis)agreement based on
perspectives, and proposed a corresponding task
• Pilot annotation on relevant sentence identification
• Annotator disagreement: undesirable or insightful?
• New perspective on ‘traditional’ argument mining (functional classification of
statements through analysis of interactions)
• Multiple annotators: allows for using different thresholds for selecting data
• Future work: tackling other two subtasks using a combination of
expert, semi-automatic and crowd annotation, include comment-
comment relations
17https://github.com/ChantalvanSon/UnsharedTask-ArgumentMining-2016
c.m.van.son@vu.nl

18. Thank you!
18

19. Distribution of Annotations
19

20. Example: disagreement
• ARTICLE:
The bullies have moved from the playground to
the mobile screen, and there is no escaping
harassment that essentially lives in your pocket.
• COMMENT:
Ms. Tynes: The bullies haven't moved from the
playground to the screen. They do both.
20

21. 21
moved
from the
playground
the bullies
to the screen
Example: disagreement

22. 22
moved
from the
playground
the bullies
to the screen
person X
AFFIRMATIVE
CERTAIN
NON-FUTURE
Example: disagreement

23. 23
moved
from the
playground
the bullies
to the screen
person X person Y
AFFIRMATIVE
CERTAIN
NON-FUTURE
NEGATIVE
CERTAIN
NON-FUTURE
Example: disagreement

24. 24
moved
from the
playground
the bullies
to the screen
person X person Y
AFFIRMATIVE
CERTAIN
NON-FUTURE
NEGATIVE
CERTAIN
NON-FUTURE
Example: disagreement
DISAGREEMENT