Part 1 of this webinar series demystifies the underlying technology used in technology assisted review (and predictive coding).

1. Demys&fying
Technology
Assisted
Review
Everything
You
Need
to
Know
(But
Were
Afraid
to
Ask)
Sonya
L.
Sigler

2. Agenda
 State
of
the
Industry
 Studies
 Cases
 Why
Use
Technology
Assisted
Review
 Poten=al
Impact
of
TAR
on
Cases
 Technology
Assisted
Review
&
How
to
Use
It
 Key
Factors
in
Choosing
to
Use
TAR
 Mi=ga=ng
Risks:
A
Few
Success
Tips
 The
Big
Looming
Ques=ons
 Q
&
A
Demys&fying
Technology
Assisted
Review

3. Technology
Assisted
Review:
State
of
the
Industry
 Document
Review
cost
is
the
#1
challenge
for
e-­‐discovery
Rand
Study:
doc
review
comprises
73%
of
total
e-­‐discovery
costs
 Technology
Assisted
Review
consistently
outperforms
blind
keyword
culling
and,
in
many
cases,
human
review
 Court
Approved
TAR
for
use
in
e-­‐discovery:
Process
is
key,
but…
 Technology
Assisted
Review
(TAR)
addresses
this
problem
head-­‐on
Process
MaNers
3
Demys&fying
Technology
Assisted
Review

4. State
of
the
Industry:
Technology
Assisted
Review
is
the
Wave
Demys&fying
Technology
Assisted
Review

5. Fact
or
Myth?
Manual
review
by
humans
of
large
amounts
of
informa5on
is
as
accurate
and
complete
as
possible
-­‐
perhaps
even
perfect
-­‐
and
cons5tutes
the
gold
standard
by
which
all
searches
should
be
measured
This is “The reigning Myth of ‘perfect’ retrieval using traditional means”
Best
Prac5ces
Commentary
on
the
Use
of
Search
and
Informa5on
Retrieval
Methods
in
E-­‐Discovery
The
Sedona
Conference
Journal
(2007)
p.
199
Human beings retrieved less than 20% of the relevant documents when they
believed they were retrieving over 75%
An
Evalua5on
of
Retrieval
Effec5veness
for
a
Full-­‐Text
Document
Retrieval
System
Blair
&
Maron
(1985)
Demys&fying
Technology
Assisted
Review

6. What
Are
Courts
Saying
About
TAR?
Da
Silva
Moore
Global
Aerospace
Kleen
Products
Demys&fying
Technology
Assisted
Review

7. How
We
Talk
About
Technology
Assisted
Review
Linear
Review
Accelerated
Review
Email
Threading
Near
Duplicate
Detec=on
Automated
Review
Clustering
Per
Relevance
Ranking
Categoriza=on
(Supervised)
Document
Machine
Learning
Cost
Latent
Seman=c
Indexing
Sta=s=cal
probability
PaXern
Analysis
Sampling
Data
for
High
Precision
and
Recall
Rates
Organiza8on
Commitment
Demys&fying
Technology
Assisted
Review

8. It’s
Star=ng
to
Gain
Trac=on
in
e-­‐Discovery
100%
Scan
and
OCR
%
of
cases
Online
Review
50%
Transparant
KW
Search
Accelerated
Review
Automated
Review
0%
Mid
90's
'00
'05
'10
'15
Increasing
data
volumes,
increasing
cost
of
review
Demys&fying
Technology
Assisted
Review

9. Key
Differences
Between
Linear
Review
and
TAR
Demys&fying
Technology
Assisted
Review

10. When
to
Use
Automated
Review
 2
broad
applica=ons:
 Pre-­‐Keyword
Cull
-­‐
Combined
use
for
Search/Cull
and
Review
 Poten=ally
most
accurate
results
 More
costly
in
today’s
industry
 Post
Keyword
Cull
–
Itera&ve
Keyword
Cull,
then
TAR
Usage
 >
90%
of
today’s
use
cases
 S=ll
more
accurate
than
linear
review
(according
to
numerous
studies)
 Primary
goal:
reduce
review
cost
Demys&fying
Technology
Assisted
Review

11. Poten=al
Impact
of
Automated
Review
Senior
Precision
&
Involvement
Review
Strategy
Cost
Time
Recall
Upfront
Human
Resources
Fast
start,
Linear
Review
$$$
Low
Not
Req'd
Heavy
throughout
slow
finish
Accelerated
Med
start,
Medium
upfront,
$$
Med
Med
Review
med
finish
Medium
later
Automated
Slow
start,
Heavy
upfront,
$
Med+
High
Review
fast
finish
Light
later
 Massive
cost
savings
 Increased
speed
 Less
documents
to
review
 Faster
speeds
in
review
 Ability
to
hit
tough
deadlines
 Get
to
your
relevant
data
faster
Demys&fying
Technology
Assisted
Review

12. How
This
Plays
Out
Senior
Involvement
Review
Strategy
Docs
to
Review
Cost
of
Review*
Time
to
Review*
Upfront
Crea=on
of
review
Linear
Review
239,063
$358,594
30
work
days
guidelines
Accelerated
Review
239,063
$258,225
20
work
days
Add’l
~2
days
using
Rela=vity
Analy=cs
Automated
Review
76,109
$144,708
15
work
days
4-­‐7
days
at
outset
using
Equivio
Relevance
Details:
 212gb
processed;
85%
KW
Cull
Rate:
239,063
docs
promoted
for
review
 Cost
of
review:
$300/hr
for
Senior
Associate;
$75/hr
for
contract
aNy
 20
contract
aNys
u&lized
*
Includes
upfront
cost
and
&me
to
train
tools
Demys&fying
Technology
Assisted
Review

13. Types
of
TAR
Tools
Linguis=c
–
word
based
Sta=s=cal
-­‐
#s
based
Demys&fying
Technology
Assisted
Review

14. TAR
Components
1
OR
OR
Machine
Machine
Seed
Set
Categoriza&on
Categoriza&on
Seed
Set
2 Random
Sample
3 Quality
Control
Itera&on
Sampling
Demys&fying
Technology
Assisted
Review

15. Sample
Technology
Assisted
Review
Workflow
Random
Sample
Seed
Set
if
needed
Expert
reviews
sample
Non-­‐
Responsive
responsive
Model
learns
Repeat
un=l
stable
Model
predicts
Responsive
Non-­‐responsive
Model
categorizes
all
remaining
documents

16. How
Automated
Review
Works
@SFLData
You:
Define
case
issues
We:
Affirm
use
case
You:
Review
control
set
We:
Affirm
sta=s=cal
validity
You:
Review
training
sets
We:
Affirm
training
stability
Both:
Decide
on
cutoff
point
based
on
recall/precision
rates
You:
Perform
bult-­‐in
QC
We:
Test
the
Method,
Test
the
Rest
Demys&fying
Technology
Assisted
Review

17. Key
Factors
to
Consider
for
Using
TAR
 Type
of
Data
 Richness/Density
of
Data
 Amount/Volume
of
Data
 Timeline
Involved
 People
Demys&fying
Technology
Assisted
Review

18. Mi=ga=ng
Risks:
A
Few
Tips
for
Success
 Get
buy-­‐in
 Put
your
best
people
on
it
 Appreciate
Process
 Up-­‐front
=me
commitment
 Data
loads
Demys&fying
Technology
Assisted
Review

19. The
Big
Looming
Ques=ons
 What
is
the
difference
between
the
tools?
And
how
do
I
know
if
one
is
beXer
than
the
other?
 Do
I
really
need
to
have
a
senior
person
train
the
tool?
 How
do
I
make
a
decision
on
what
to
review
and
what
to
leave
behind?
 Do
I
need
to
be
open
with
Opposing
about
the
use
of
a
TAR
tool?
 Should
I
do
keyword/term
culling
in
advance?
Demys&fying
Technology
Assisted
Review

20. Your
Ques=ons
Post
your
ques=ons
in
the
chat
sec=on
now
and
let’s
discuss…
Demys&fying
Technology
Assisted
Review

21. Thank you!
Sonya
L.
Sigler
Vice
President,
Product
Strategy
&
Consul&ng
SFL
Data
415-­‐321-­‐8385
sonya@sfldata.com
www.sfldata.com
Next
Webinar:
Demys=fying
Technology
Assisted
Review:
Part
2:
A
Deeper
Dive
into
the
Technology
August
29th,
9:30-­‐10:30
am
PDT
Demys&fying
Technology
Assisted
Review