Opportunistic Adversaries - On Imminent Threats to Learning-based Business Automation (presentation at SRII 2012)

IBM Research – Tokyo

Opportunistic Adversaries
– On Imminent Threats to
Learning-based Business Automation –

Michiaki Tatsubori, IBM Research – Tokyo
Shohei Hido, Preferred Infrastructure, Inc.

M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation


About This Talk

§ A business process with automated decision through
machine learning is useful & promising

§ The “opportunistic adversaries” – potential adversaries
exploiting its misclassification, which is inevitable
– A case study with loan exam automation

§ A reference design & implementation of counter
measures

2 M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation


Business Processes with Machine Learning
– a Promising Approach
An Example of Credit Card Fraud Detection
BPM ML
Order
Validation

Order parameter Transparent
Transparent
Fraud Decision
Decision
Detection Report parameter Service
Service
(e.g. exception)
Models
Models
Exception? Exception
Yes handling
No Induce models

No Yes
Order
accepted? Learning
Training parameter Learning
Service
Service
& decision record History
History
Process Repository
Repository
Review
Order
Order process histories
Order rejection histories


Potential Application: Loan Exam Processing



Potential Application: Insurance Claims Processing



Supervised Machine Learning is the Key Technology
D training = {( x1 , y1 ),..., (x n , yn )}

§ Machine learning for where x i ∈ V (V : feature - vector space)
and y j ∈ C (C : a set of class labels)
process automation:
– Learning from known Learning Data:
decisions for input
Approve
parameters
Distinction by
– Allowing automated a ground-truth function
(unknown)
decision for unknown input
parameters Models
Models

Ex. Insurance claim
processing, credit order
approval, etc. Reject



Inevitable Misclassifications are Compensated
by Other Benefits
Produce a function
§ Hard to avoid misclassifications h:x → y
– Tradeoffs between false where x ∈ V (V : feature - vector space)
positives versus false and y ∈ C (C : a set of class labels)
negatives Test Data:
§ Overall business models can Approve
compensate loss from
FP Distinction by
misclassifications with benefit a learned function
(probabilistic)
from automation:
– Less human workload
– Less careless misses
FN

Reject



Opportunistic Adversaries: Threats by Adversaries
Outsmarting Machinery Misjudgment
– Opportunistic adversaries Test Data:
scenario:
Approve
• A user detects the
misclassification by the FP
FP FP
FP
FP
system for certain input
parameters
• Attackers provide parameters
so that they resemble the
former input parameters FN
misclassified
Ex. A manual for “legally Reject
cheating insurance claims”



Conditions Where Opportunistic Adversaries Become
Threats

§ Threat: Damages from spreading adversaries which outsmart inevitable
false positives/negatives with ML, under the condition:
– Attacks intentionally forge inputs (integrity attack),
– Attacks start from a tiny false positive/negative case revealed to
potential attackers (exploratory and indiscriminate attack), and
– Unawareness of damages (stealthy attack)

§ Existing works didn’t address this situations or required impractical
amount of learning and test samples
– Transfer learning [Sugiyama 2006]
– Adversarial learning [Lowd 2005]
– Outlier detection [Hido 2008]



BPM & Abstract Decision Service + Anomaly Detection

BPM Decision Service
Order
Validation

Order parameter Transparent
Transparent
Fraud Decision
Decision
Detection Report parameter Service
Service
(e.g. exception) Models
Models
Exception? Exception
Yes handling
No
Rule
Rule
Repository
Repository
No Yes
Order
accepted?

History of
History of
Process Automated
Automated
Review Decisions
Order Decisions Input Frequencies
Input Frequencies



Reference Countermeasure Prototype Outline
§ Record timestamps of training and test Record timestamps of input data:
inputs
A1
§ Cluster training inputs to segmentalize Class A
the input space into subclasses A2
A3
§ Maintain frequency statistics about
per-subclass probabilities of training time
inputs for various times and timeframes Class B
and test inputs for recent times and
timeframes
§ Detect significant relative increases in
distribution Time series analysis
each subclass as anomaly to alert (telling for subclass probabilities
as an exception)
1 2 3 log t
– Sensing potential attacks
outsmarting the trained model Score :
– Giving a chance of human review Ps( test ) (l )
and model update q( x (test)
)=
E ( Pt ( training ) (l )) (σ ( Pt ( training) (l )) + 1)
k

where s = t k and l = g( xk )
(test) (test)



Architecture of Reference Implementation
Training Input Test Input

label
A B B A
Time
time
t1 t2 t3 t4 stamp
s1 s2 s3 s4
stamp
Classification Output

Classifier Classifier A B
B
Generator s1 s2 s3
classifier

Sub-classifier Time Series
Generator Analyzer
(Test Data)
sub-classifier
Time Series
Analyzer distribution
distribution
frequency
(Training Data)
1 2 3 log t
statistics
1 2 3 log t
(test data)
frequency Anomaly
statistics detector notify anomaly
(training data)



Preliminary Experimental Results
§ Observed effectiveness in an Learning Data:
Attack
experiment with spam filtering
– Experimented with
Spambase (mails with
some spams) in UCI data Test Data:
Clusters

– Used first 80% for training
and last 20% for testing
– Replaced 5% of testing
data with misclassified Clusters

Freq. Ratio / Std. Dist.
samples
– Observed they are Detected
detected as anomaly



Concluding Remarks

§ Defined “Opportunistic Adversaries”
as a threat to automated business Approve
processes with machine learning FP
FP FP
FP
FP
– Integrity, exploratory, indiscriminate,
and stealthy attacks

FN

§ A reference solution architecture Reject
proposed
– + anomaly detection in temporal input
space distribution statistics



Thank you!
Questions?

M Tatsubori & S Hido: Opportunistic Adversaries, SRII 2012, San Jose Jul 25, 2012 © 2012 IBM Corporation

Opportunistic Adversaries - On Imminent Threats to Learning-based Business Automation (presentation at SRII 2012)

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Opportunistic Adversaries - On Imminent Threats to Learning-based Business Automation (presentation at SRII 2012)

Ähnlich wie Opportunistic Adversaries - On Imminent Threats to Learning-based Business Automation (presentation at SRII 2012) (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Opportunistic Adversaries - On Imminent Threats to Learning-based Business Automation (presentation at SRII 2012)

Hinweis der Redaktion