The document presents a methodology called CLOR (Criminal Law and Procedure Ontology and Rule) for constructing a knowledge graph to support automated legal reasoning for answering questions from a US bar exam. The methodology involves identifying relevant concepts, entities, classes and properties from sample exam questions and background legal texts. An evaluation of a CLOR knowledge graph achieved a precision of 0.5 and recall of 1 on 16 new exam questions, outperforming a baseline approach. Future work involves fully implementing the methodology and applying it to other areas of the bar exam.

1. ©2018 Fawei, Pan, Kollingbam and Wyner
A Methodology for a Criminal Law and
Procedure Ontology for Legal QuesEon
Answering
Biralatei J. Fawei1, Jeff Z. Pan1 , MarEn J. Kollingbaum1 and Adam Wyner2
1 2

2. ©2018 Fawei, Pan, Kollingbam and Wyner
Automated Legal Reasoning
• Legal professionals have to pass
the bar exam to pracEce law
• MulEstate Bar ExaminaEon (MBE)
– used in most states in US
– MulEple choice
– Large body of legal cases and
associated set of answers
[Credit: law.com]

3. ©2018 Fawei, Pan, Kollingbam and Wyner
Example 1: Answer Sentence SelecEon
Task
[Credit: A. K. John et al.]

4. ©2018 Fawei, Pan, Kollingbam and Wyner
Example 2: Deep Textual Entailment
Task
AXer being red from his job, Mel drank almost a quart
of vodka and decided to ride the bus home. While on
the bus, he saw a briefcase he mistakenly thought was
his own, and began struggling with the passenger
carrying the briefcase. Mel knocked the passenger to
the floor, took the briefcase, and fled. Mel was
arrested and charged with robbery. His mistake
negated the required specific intent.
Case:
1. Mel should be acqui^ed.
Answers:
[Fawei et al., 2016]

5. ©2018 Fawei, Pan, Kollingbam and Wyner
ExisEng Work
• ExisEng work
– ExisEng textual entailment tools can oXen deal with
1-2 steps of reasoning (deep textual entailment task)
– Pure machine/deep learning approaches lack
explanability (answer sentence selecEon task)
• Idea:
– Applying learning directly might be too big a jump
– We need a methodology for construcEng KG
• learning may be used in some of the steps
– Use KG for deep QA
5

6. ©2018 Fawei, Pan, Kollingbam and Wyner
Problem Statement
• Problem: automated legal reasoning based on textual
informaEon
– given a case descripEon C, some source informaEon S as
background legal knowledge, some answer A
– the task is
1. to construst KG(C), KG(S), q(A)
2. to check if KG(C) U KG(S) |= q(A) and give explanaEons
• Source of (textual) informaEon
– Corpus of bar exam quesEons
– Bar examinaEon curriculum
– Bar examinaEon course materials
– Bar examinaEon preparatory materials
– Law textbooks
6

7. ©2018 Fawei, Pan, Kollingbam and Wyner
7
Criminal Legal Ontology and Rule (CLOR)
Methodology

8. ©2018 Fawei, Pan, Kollingbam and Wyner
StarEng from Competency QuesEons
“Mel was arrested and charged with robbery”
“Mel should be acquiAed”
• Step 1: IdenEfy key dimensions of the competency quesEons
– What are the elements of robbery?
• Step 2: For each dimension, idenEfy necessary concrete
competency quesEons.
– Under what condiEons should X be acqui^ed?
– Under what condiEons should X be convicted?
8

9. ©2018 Fawei, Pan, Kollingbam and Wyner
IdenEfying Relevent Classes and EnEEes
• Step 4: Separate the relevant nouns from the irrelevant (from the
selected noun)
– Relevant nouns: Mel, Briefcase, Robbery, Threat, Defense,
Voluntary-acEon, ...
• Step 5: Separate the nouns classes from the instance nouns
– Noun classes: Passenger, Robbery, IntoxicaEon, Mistake,
Voluntary-acEon, ...
– Instance nouns: Mel, Vodka, Briefcase
• Step 6: Decide the types of the instance nouns
– Class instances: Person(Mel), Property(Briefcase), ...
9

10. ©2018 Fawei, Pan, Kollingbam and Wyner
Define Classes and ProperEes
• Steps 7 and 8: creaEng the class hierarchy, idenEfying relevant
parent and sibling classes
– class hierarchy: Mel,...
• Step 9: Separate definable classes from atomic ones
– definable class: should-be-acqui^ed, ...
• Step 18: define definable classes and properEes with rules
– definable property: should-be-acqui^ed-of, ...
10

11. ©2018 Fawei, Pan, Kollingbam and Wyner
EvaluaEon
• Our ontology was evaluated in three ways:
– An evaluaEon against our competency quesEons in the
development stage
– A task-based approach for assessing the performance of
the system with respect to benchmark
• CLOR built based on 12 quesEons
• tested against 16 Bar exam quesEons
• Precision : 0.5 (baseline TIE: 0.5)
• recall: 1 (baseline TIE: 0.5)
– An assessement using the Pellet reasoner and the OntOlogy
Pioall Scanner
11

12. ©2018 Fawei, Pan, Kollingbam and Wyner
EvaluaEon (2)

13. ©2018 Fawei, Pan, Kollingbam and Wyner
Conclusion & Future Work
• We have proposed a KG based approach to deep
textual entailment in US Bar exam
• We have present the CLOR methodology and showed
how legal reasoning could be done based on CLOR
• In future we will
– Implement the procedure
– Apply CLOR methodology only on sample
quesEons
– work on other branches of Bar Exam quesEons
13

14. ©2018 Fawei, Pan, Kollingbam and Wyner
Thank you!
jeff.z.pan@abdn.ac.uk
homepages.abdn.ac.uk/jeff.z.pan/pages/
@jpansw