The Myths + Realities of Machine-Learning Cybersecurity

AI Truths and Myths
Dr. Chase Cunningham,
Principal Analyst Security and Risk
September 28, 2017

3© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Let’s Define AI…. Or At Least Be Real About It
AI (Today)=
Math, Patterns, Computations, Iterations

Artificial Intelligence As It Stands Today
AI =
Data + Machine Learning + Human Interaction

What AI is Not

Timeline of AI Failures
• Failure of machine translation1966
• Abandonment of connectionism1970
• DARPA's frustration with the Speech Understanding Research program at Carnegie Mellon University1971−75
• Large decrease in AI research in the United Kingdom in response to the Lighthill report1973
• DARPA's cutbacks to academic AI research in general1973−74
• Collapse of the Lisp machine market1987
• Cancellation of new spending on AI by the Strategic Computing Initiative1988
• Expert systems slowly reaching the bottom1993
• Quiet disappearance of the fifth-generation computer project's original goals1990s

Watson
• IBM’s Artificial Intelligence
computer system
• Capable of answering
questions in natural
language
• Competed against
champions on Jeopardy
and won

Watson’s Sources of Information
• Encyclopedias
• Dictionaries
• Thesauri
• Newswire articles
• Literary works
• Databases, taxonomies,
and ontologies
• Wikipedia articles
And more

How Watson Works
• Receives the clues (questions) as electronic texts
• It then divides these texts into different keywords and sentence
fragments and searches for statistically related phrases
• Quickly executes thousands of language analysis algorithms
• The more algorithms that find the same answer increase Watson’s
confidence of his answer and it calculates whether or not to make a
guess

What is Machine Learning?
Applications of algorithms that
• improve their performance
• at some task
• with experience

Building Blocks of AI
• Classification
• Compare unknown against larger
known dataset
• Clustering
• Find data points similar in nature
• Regression
• Measure statistical relationships
between variables based on
history or training set

Where Companies Want to Use AI
34% of companies
plan/are using AI to
mitigate security risks

What is Intelligence: The Turing Test
A machine can be described as a
thinking machine if it passes the
Turing Test.
i.e. If a human agent is engaged
in two isolated dialogues
(connected by teletype say); one
with a computer, and the other
with another human and the
human agent cannot reliably
identify which dialogue is with
the computer.

FORRESTER.COM
Thank you
Dr. Chase Cunningham
ccunningham@forrester.com

Separating
Myth
from
Reality
Machine
Learning
and
A.I.
in
Cybersecurity

18
Hey.
I’m
Stephan
Jou.
I
like
analytics.
• CTO
at
Interset
• Previously:
Cognos and
IBM’s
Business
Analytics
CTO

Office
• Big
data
analytics,
visualization,
cloud,
predictive

analytics,
data
mining,
neural
networks,
mobile,

dashboarding and
semantic
search
• M.Sc.
in
Computational
Neuroscience
and

Biomedical
Engineering,
and
a
dual
B.Sc.
in

Computer
Science
and
Human
Physiology,
all
from

the
University
of
Toronto

19
About
Interset
At
Interset,
we
catch
bad
guys
with
math.
• Data
science
and
machine
learning
on
big
data
analytics

technologies
• Cover
multiple
cybersecurity
use
cases
• Based
in
Ottawa,
Ontario,
Canada
• Award
winning
threat
detection
platform
• Successful
deployments
across
multiple
verticals
• Clients
include
US
Intelligence
Communities
And
a
leader
in
security
analytics.

20
Best
Practices
and
Real-‐Life
Examples
There
is
too
much
FUD,

confusion
and
snake
oil
out

there!
How
can
we
separate
myth
from
reality?
Q A
Construct
a
mathematical
proof
of
correctness!
Best
practices,
patterns,
and
lessons

from
actual
real-‐life
case
studies!

21
Case
Study
#1:
$20B
Manufacturer
X
2 Engineers
stole data
1 Year
$1 Million Spent
Large security
vendor failed to
find anything
2 Weeks
Easily
identified the 2
Engineers
Found 3
additional users
stealing data in
North America
Found 8
additional users
stealing data in
China

22
Lesson
#1:
The
Math
Matters
– Test
It
• Too
much
snake
oil
• The
math
matters
– but
the
use
case
matters

more!
• Don’t
rely
on
a
smoking
gun
Recommendations
• Agree
on
the
use
cases
in
advance
• Use
a
proof-‐of-‐concept
with
historical/existing
data
to
test
the
SA’s
math
• Engage
red
team
or
pen
testing
if
available
• Evaluate
the
results:
Do
they
support
the
use
cases?

23
Case
Study
#2:
Every
Interset
Customer
Millions
of
events

analyzed
with

machine
learning
Anomalies

discovered
by

data
science
High
quality

“most
wanted”

list
By
analyzing
the
intersection
of
data
from
users,
machines,
files,
projects,

servers,
sharing
behavior,
resource,
websites,
IP
Addresses
and
more

24
Lesson
#2:
Less
Alerts,
Not
More
• Solution
should
help
you
deal
with
less
alerts,
not
more alerts
• Solution
should
leverage
sound
statistical

methods
to
reduce
false
positives
and
noise
• Should
allow
you
to
do
more
with
the

limited
resources
you
have
Recommendations
Measure
and
quantify
the
amount
of
work
effort
involved
with
and
without
the

Security
Analytics
system

25
Case
Study
#3:
Defense
Contractor
High
Probability
Anomalous
Behavior
Models
• Detected
large
copies
to
the
portable
hard
drive,

at
an
unusual
time
of
day
• Bayesian
models
to
measure
and
detect
highly

improbable
events
High
Risk
File
Models
• Detected
high
risk
files,
including
PowerPoints
used
to
collect
large
amounts
of
inappropriate

content
• Risk
aggregation
based
on
suspicious
behaviors

and
unusual
derivative
movement

26
Lesson
#3:
Automated,
Measured
Responses
• Security
Analytics
system
should
allow
you

to
quantify risk,
not
just
a
binary
alert
• Consider
how
to
automate
responses
to

low,
medium,
high
and
extreme
risk

scenarios
• Where
does
security
analytics
fit
into
your

existing
runbook?
Recommendations
• Ensure
the
Security
Analytics
system
has
the
ability
to
output
a
risk
assessment

level
or
score,
not
just
a
binary
alert
• Ensure
the
Security
Analytics
system
can
integrate
with
downstream
systems
• Evaluate
the
solution
with
automated
response
systems
as
part
of
the
deployment

27
Case
Study
#4:
Healthcare
Records
and
Payment

Processing
• Profile:
6.5
billion
transactions
annually,
750+

customers,
500+
employees
• Team
of
7:
CISO,
1
security
architect
,
3
security

analysts,
2
network
security

• Analytics
surfaced
(for
example)
an
employee
who

attempted
to
move
“sensitive
data”
from
endpoint
to

personal
Dropbox
• Employee
was
arrested
and
prosecuted
using
incident

data
Focus
and
prioritized
incident
responses
Incident
alert
accuracy
increased
from
28%
to
92%
Incident
mitigation
coverage
doubled
from
70
per
week
to
140

28
Lesson
#4:
Meaningful
Metrics
• Hawthorne
Effect:
Whatever
gets
measured,
gets
optimized
Recommendations
• Define
meaningful
operational
metrics
(not
just
“false

positives”)
• Build
a
process
for
measuring
and
quantifying
over
time,
not

just
during
a
pilot
• Ensure
the
Security
Analytics
system
supports
a
feedback

process
to
adjust
the
analytics
to
support
your
target
metrics

29
What
Have
We
Learned?
Lessons
• The
Math
Matters
– Test
It
• Less
Alerts,
Not
More
• Automated,
Measured
Responses
• Meaningful
Metrics
Recommendations
• Agree
on
the
use
cases
in
advance
• Evaluate
results
with
and
without

security
analytics
system
• Assess
risk
level,
not
binary
alert
• Ensure
integrated
feedback
and

automated
response

The Myths + Realities of Machine-Learning Cybersecurity

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (19)

Ähnlich wie The Myths + Realities of Machine-Learning Cybersecurity

Ähnlich wie The Myths + Realities of Machine-Learning Cybersecurity (20)

Mehr von Interset

Mehr von Interset (7)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

The Myths + Realities of Machine-Learning Cybersecurity