The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Professors Daniel Martin Katz & Michael J. Bommarito - Illinois Tech Law / Univ of Michigan CSCS (Updated Version)
Ähnlich wie The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Professors Daniel Martin Katz & Michael J. Bommarito (Updated) (20)
Independent Call Girls Pune | 8005736733 Independent Escorts & Dating Escorts...
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Professors Daniel Martin Katz & Michael J. Bommarito (Updated)
1. The Three Forms
of (Legal) Prediction
professor daniel martin katz
home | Illinois tech - chicago kent
blog | ComputationalLegalStudies
corp | LexPredict
experts, crowds & algorithms
professor michael j bommarito
4. play “whack-a-mole”, reacting to
problems by creating fear and
friction within organizations and
the impression that there is a
legal risk around every corner.
Mediocre Lawyers
5. can help clients shape
(perhaps distort)
external perception of risk.
Merely Clever Lawyers
6. design systems that
balance risk and improve
transparency, helping clients
correctly price risk internally
Great Lawyers
15. Quantitative Legal Prediction
- or -
How I Learned to Stop Worrying and Start
Preparing for the Data Driven Future of the
Legal Services Industry
Professor Daniel Martin Katz
#LegalAnalyics
#LegalData #LegalPrediction
18.
Daniel Martin Katz, Joshua Gubler, Jon Zelner, Michael Bommarito, Eric Provins
& Eitan Ingall, Reproduction of Hierarchy? A Social Network Analysis of the American
Law Professoriate, 61 Journal of Legal Education 76 (2011)
23. Acyclic digraphs arise in many natural and artificial processes. Among the
broader set, dynamic citation networks represent a substantively important
form of acyclic digraphs. For example, the study of such networks includes the
spread of ideas through academic citations, the spread of innovation
through patent citations, and the development of precedent in common law
systems.
26. Daniel Martin Katz, The MIT
School of Law? A Perspective on
Legal Education in the 21st
Century, University of Illinois
Law Review 1431 (2014)
New York Times - August 1, 2014
Daniel Martin Katz, an associate professor with
expertise in big data and powerful computing and their
applications to legal studies. He hopes to give his
students a leg up in a job market that seems
increasingly bleak, and to help them become “T-
shaped,” by which he means having deep knowledge —
the downward swipe of the letter T — as well as a
broadened set of abilities. So providing them with
information on seemingly arcane subjects like data
analytics can be a career builder. “Analytics plus law
gets you into a niche,” he said.
42. Quantitative Legal Prediction
- or -
How I Learned to Stop Worrying and Start
Preparing for the Data Driven Future of the
Legal Services Industry
Professor Daniel Martin Katz
43.
44.
45.
46.
47.
48.
49.
50. Today we are going to
talk about one key
idea in prediction
54. For today we will apply
these approaches to the
decisions of the
Supreme Court of United States
55. Every year, law reviews, magazine and
newspaper articles, television and radio
time, conference panels, blog posts, and
tweets are devoted to questions such as:
How will the Court rule in particular cases?
56.
57. There are only 3 ways
to predict something
Experts
Crowds
Algorithms
59. Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political Science
Approaches to Predicting
Supreme Court Decision
Making
The Supreme Court
Forecasting Project:
84. “Software developers were asked on two
separate days to estimate the completion
time for a given task, the hours they
projected differed by 71%, on average.
When pathologists made two assessments of
the severity of biopsy results, the correlation
between their ratings was only .61 (out of a
perfect 1.0), indicating that they made
inconsistent diagnoses quite frequently.
Judgments made by different people are
even more likely to diverge.”
93. Our approach is a special version
of random forest
Black
Reed
Frankfurter
Douglas
Jackson
Burton
Clark
Minton
Warren
Harlan
Brennan
Whittaker
Stewart
White
Goldberg
Fortas
Marshall
Burger
Blackmun
Powell
Rehnquist
Stevens
OConnor
Scalia
Kennedy
Souter
Thomas
Ginsburg
Breyer
Roberts
Alito
Sotomayor
Kagan
1953 1963 1973 1983 1993 2003 2013
9-0 Reverse
8-1, 7-2, 6-3
19 19 19 19 19 20 20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
- Reverse
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
-
8-1, 7-2, 6-3
9-0
19 19 19 19 19 20 20
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2463244
http://arxiv.org/abs/1407.6333
available at
Revise and Resubmit @ PloS One
94.
95. we have developed an
algorithm that we call
{Marshall}+
random forest
96. Benchmarking
since 1953
+
Using only data
available prior to
the decision
Mean Court Direction [FE]
Mean Court Direction 10 [FE]
Mean Court Direction Issue [FE]
Mean Court Direction Issue 10 [FE]
Mean Court Direction Petitioner [FE]
Mean Court Direction Petitioner 10 [FE]
Mean Court Direction Respondent [FE]
Mean Court Direction Respondent 10 [FE]
Mean Court Direction Circuit Origin [FE]
Mean Court Direction Circuit Origin 10 [FE]
Mean Court Direction Circuit Source [FE]
Mean Court Direction Circuit Source 10 [FE]
Difference Justice Court Direction [FE]
Abs. Difference Justice Court Direction [FE]
Difference Justice Court Direction Issue [FE]
Abs. Difference Justice Court Direction Issue [FE]
Z Score Difference Justice Court Direction Issue [FE]
Difference Justice Court Direction Petitioner [FE]
Abs. Difference Justice Court Direction Petitioner [FE]
Difference Justice Court Direction Respondent [FE]
Abs. Difference Justice Court Direction Respondent [FE]
Z Score Justice Court Direction Difference [FE]
Justice Lower Court Direction Difference [FE]
Justice Lower Court Direction Abs. Difference [FE]
Justice Lower Court Direction Z Score [FE]
Z Score Justice Lower Court Direction Difference [FE]
Agreement of Justice with Majority [FE]
Agreement of Justice with Majority 10 [FE]
Difference Court and Lower Ct Direction [FE]
Abs. Difference Court and Lower Ct Direction [FE]
Z-Score Difference Court and Lower Ct Direction [FE]
Z-Score Abs. Difference Court and Lower Ct Direction [FE]
Justice [S]
Justice Gender [FE]
Is Chief [FE]
Party President [FE]
Natural Court [S]
Segal Cover Score [SC]
Year of Birth [FE]
Mean Lower Court Direction Circuit Source [FE]
Mean Lower Court Direction Circuit Source 10 [FE]
Mean Lower Court Direction Issue [FE]
Mean Lower Court Direction Issue 10 [FE]
Mean Lower Court Direction Petitioner [FE]
Mean Lower Court Direction Petitioner 10 [FE]
Mean Lower Court Direction Respondent [FE]
Mean Lower Court Direction Respondent 10 [FE]
Mean Justice Direction [FE]
Mean Justice Direction 10 [FE]
Mean Justice Direction Z Score [FE]
Mean Justice Direction Petitioner [FE]
Mean Justice Direction Petitioner 10 [FE]
Mean Justice Direction Respondent [FE]
Mean Justice Direction Respondent 10 [FE]
Mean Justice Direction for Circuit Origin [FE]
Mean Justice Direction for Circuit Origin 10 [FE]
Mean Justice Direction for Circuit Source [FE]
Mean Justice Direction for Circuit Source 10 [FE]
Mean Justice Direction by Issue [FE]
Mean Justice Direction by Issue 10 [FE]
Mean Justice Direction by Issue Z Score [FE]
Admin Action [S]
Case Origin [S]
Case Origin Circuit [S]
Case Source [S]
Case Source Circuit [S]
Law Type [S]
Lower Court Disposition Direction [S]
Lower Court Disposition [S]
Lower Court Disagreement [S]
Issue [S]
Issue Area [S]
Jurisdiction Manner [S]
Month Argument [FE]
Month Decision [FE]
Petitioner [S]
Petitioner Binned [FE]
Respondent [S]
Respondent Binned [FE]
Cert Reason [S]
Mean Agreement Level of Current Court [FE]
Std. Dev. of Agreement Level of Current Court [FE]
Mean Current Court Direction Circuit Origin [FE]
Std. Dev. Current Court Direction Circuit Origin [FE]
Mean Current Court Direction Circuit Source [FE]
Std. Dev. Current Court Direction Circuit Source [FE]
Mean Current Court Direction Issue [FE]
Z-Score Current Court Direction Issue [FE]
Std. Dev. Current Court Direction Issue [FE]
Mean Current Court Direction [FE]
Std. Dev. Current Court Direction [FE]
Mean Current Court Direction Petitioner [FE]
Std. Dev. Current Court Direction Petitioner [FE]
Mean Current Court Direction Respondent [FE]
Std. Dev. Current Court Direction Respondent [FE]
0.00781
0.00205
0.00283
0.00604
0.00764
0.00971
0.00793
TOTAL 0.04403
Justice and Court Background Information
Case Information
0.00978
0.00971
0.00845
0.00953
0.01015
0.01370
0.01190
0.01125
0.00706
0.01541
0.01469
0.00595
0.02014
0.01349
0.01406
0.01199
0.01490
0.01179
0.01408
TOTAL 0.22814
Overall Historic Supreme Court Trends
0.00988
0.01997
0.01546
0.00938
0.00863
0.00904
0.00875
0.00925
0.00791
0.00864
0.00951
0.01017
TOTAL 0.12663
Lower Court Trends
0.00962
0.01017
0.01334
0.00933
0.00949
0.00874
0.00973
0.00900
TOTAL 0.07946
0.00955
0.00936
0.00789
0.00850
0.00945
0.01021
0.01469
0.00832
0.01266
0.00918
0.00942
0.00863
0.00894
0.00882
0.00888
Current Supreme Court Trends
TOTAL 0.14456
Individual Supreme Court Justice Trends
0.01248
0.01530
0.00826
0.00732
0.01027
0.00724
0.01030
0.00792
0.00945
0.00891
0.00970
0.01881
0.00950
0.00771
TOTAL 0.14323
0.01210
0.00929
0.01167
0.00968
0.01055
0.00705
0.00708
0.00690
0.00699
0.01280
0.01922
0.02494
0.01126
0.00992
0.00866
0.01483
0.01522
0.01199
0.01217
0.01150
TOTAL 0.23391
Differences in Trends
101. Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political Science
Approaches to Predicting
Supreme Court Decision
Making
The Supreme Court
Forecasting Project:
102. Given Some Data:
(X1, Y1), ... , (Xn, Yn)
Now We Have a New Set of X’s
We Want to Predict the Y
103. Form a BinaryTree that
Minimizes the Error
in each leaf of the tree
CART
(Classification & RegressionTrees)
112. 1
0
1
1
1
0
0
0
0
0
1
1 1
1
0
0
1
1
1
1
0
01
0
Xi1
Xi2
0
1 2
1
2
Adapted from Example
By Mathematical Monk
If No - then we are in zone (a) ...
we tally the number of zeros and ones
Using Majority Rule do we assign a
classification to this rule this leaf
split 1
(a)
Xi1 > 1 ?
YesNo
(0,5)
Classify as 1
zone (a)
138. expert crowd algorithm
via back testing we can learn the
weights to apply for particular problems
ensemble method
learning problem is to discover when to use a given stream of intelligence
152. Myriad Genetics
“Myriad employs a number of proprietary
technologies that permit doctors and patients
to understand the genetic basis of human
disease and the role that genes play in the
onset, progression and treatment of disease.”
153. Myriad Genetics
“Myriad was the subject of scrutiny
after it became involved in a lengthy
lawsuit over its controversial patenting
practices” which including the
patenting of human gene sequences ....
173. lots of litigation decisions
are just a version of this basic idea
law = finance
174. this is a part of the
industry where you
need rigorous
#LegalUnderwriting
175. but lots of litigation decisions
are actually implicit litigation finance
(or self insurance)
#fin(legal)tech
176. however most implicit litigation
finance is not based upon
rigorous underwriting …
law =! finance
(but it will)
177. we expand on this theme in this presentation
http://computationallegalstudies.com/2015/10/fin-legal-tech-laws-future-from-finances-past-katz-bommartio/
186. Michael J. Bommarito II
@ mjbommar
computationallegalstudies.com
lexpredict.com
bommaritollc.com
university of michigan center for the study of complex systems@
187. Daniel Martin Katz
@ computational
computationallegalstudies.com
lexpredict.com
danielmartinkatz.com
illinois tech - chicago kent college of law@