I will first introduce adversarial machine learning, emerging research direction dealing with security aspects of machine learning. Then, I will explain poisoning and evasion attacks, followed by the description of transferability phenomena. Finally, I will talk about the proposed defenses against such types of attacks and their effectiveness.

1. Security of Machine
Learning
Jelena Milošević
TU Wien

2. My Work
Malware detection
[C&S-19, FPS-18, DASC-16,
SECRYPT-16, CCNC-16,
IWSMA-14]
Face recognition [arxiv-18]
Anomaly detection [IJCNN-18]
Failure prediction [CINC-14]
Machine Learning for Security
2

3. My Work
Malware detection
[C&S-19, FPS-18, DASC-16,
SECRYPT-16, CCNC-16,
IWSMA-14]
Face recognition [arxiv-18]
Anomaly detection [IJCNN-18]
Failure prediction [CINC-14]
Machine Learning for Security
Attacks against ML
[In submission]
Robust defences [FSS-18]
Explainable ML
3
Security of Machine Learning

4. My Work
Malware detection
[C&S-19, FPS-18, DASC-16,
SECRYPT-16, CCNC-16,
IWSMA-14]
Face recognition [arxiv-18]
Anomaly detection [IJCNN-18]
Failure prediction [CINC-14]
Machine Learning for Security
Attacks against ML
[In submission]
Robust defences [FSS-18]
Explainable ML
Mobile systems [C&S-19, FPS-18, HST-17, DASC-16, SECRYPT-16,
CCNC-16, IWSMA-14], Embedded systems [IJCNN-18, arxiv-18,
CINC-14,SECRYPT-14,SECRYPT-13], Communication Networks [WIP]
4
Security of Machine Learning
Application Domains

5. My Work
Malware detection
[C&S-19, FPS-18, DASC-16,
SECRYPT-16, CCNC-16,
IWSMA-14]
Face recognition [arxiv-18]
Anomaly detection [IJCNN-18]
Failure prediction [CINC-14]
Machine Learning for Security
Attacks against ML
[In submission]
Robust defences [FSS-18]
Explainable ML
Mobile systems [C&S-19, FPS-18, HST-17, DASC-16, SECRYPT-16,
CCNC-16, IWSMA-14], Embedded systems [IJCNN-18, arxiv-18,
CINC-14,SECRYPT-14,SECRYPT-13], Communication Networks [WIP]
5
Security of Machine Learning
Application Domains

6. Why Do We Need Secure Machine Learning?
● Machine learning is:
○ more and more deployed in practice
○ frequently used in automated manner
○ used to protect safety critical infrastructure

7. Why Do We Need Secure Machine Learning?
● Machine learning is:
○ more and more deployed in practice
○ frequently used in automated manner
○ used to protect safety critical infrastructure
● At the same time ML is:
○ same as any other software and it has vulnerabilities that can be exploited
○ shown to have some inherent algorithms weaknesses and blind spots
○ black box approach

8. Why Do We Need Secure Machine Learning?
● Machine learning is:
○ more and more deployed in practice
○ frequently used in automated manner
○ used to protect safety critical infrastructure
● At the same time ML is:
○ same as any other software and it has vulnerabilities that can be exploited
○ shown to have some inherent algorithms weaknesses and blind spots
○ black box approach
This Talk: Before relying on ML deployed in practice we need to understand
where and how these systems fail

9. Feature selection
Selection of the best
performing algorithm
Dataset
Training
9
Machine Learning - Main Steps of Deployment

10. Feature selection
Selection of the best
performing algorithm
Dataset
Training
10
Machine Learning - Main Steps of Deployment

11. Feature selection
Selection of the best
performing algorithm
Dataset
Training
11
Deployment
Machine Learning - Main Steps of Deployment

12. What Can Go Wrong with Machine Learning?
Confidentiality Integrity Availability
Training
Deployment
Model stealing
Model inversion
Membership
inference attack

13. What Can Go Wrong with Machine Learning?
Confidentiality Integrity Availability
Training Poisoning
Deployment
Model stealing
Model inversion
Membership
inference attack
Evasion

14. What Can Go Wrong with Machine Learning?
Confidentiality Integrity Availability
Training Poisoning Poisoning
Deployment
Model stealing
Model inversion
Membership
inference attack
Evasion
Increasing false
positives

15. What Can Go Wrong with Machine Learning?
Confidentiality Integrity Availability
Training Poisoning Poisoning
Deployment
Model stealing
Model inversion
Membership
inference attack
Evasion
Increasing false
positives

16. Poisoning Attack on Machine Learning
16
Feature selection
Selection of the best
performing algorithm
Dataset
Training
Deployment

17. Poisoning Attack on Machine Learning
17
Feature selection
Selection of the best
performing algorithm
Dataset
Training
Deployment
Poisoning attack!

18. Poisoning Attack on Machine Learning
18
Feature selection
Selection of the best
performing algorithm
Dataset
Training
Deployment
Poisoning attack!

19. Poisoning Attack Example
19
Source: https://github.com/oreilly-mlsec/book-resources/blob/master/chapter8/figures/8-02-indiscriminate-poisoning-translation.png

20. Feature selection
Selection of the best
performing algorithm
Dataset
Training
20
Deployment
Evasion Attack on Machine Learning

21. Evasion Attack on Machine Learning
21
Feature selection
Selection of the best
performing algorithm
Dataset
Training
Deployment
Evasion attack!

22. Evasion Attack on Machine Learning
22
Feature selection
Selection of the best
performing algorithm
Dataset
Training
Deployment
Evasion attack!
Adversarial Sample

23. Adversarial Samples on Face Recognition DNNs
Example of adversarial glasses
Sharif et al, Accessorize to a Crime: Real and stealthy attacks on state of the art face recognition, ACM CCS 2016

24. Adversarial Samples on Face Recognition DNNs
Example of adversarial glasses
Sharif et al, Accessorize to a Crime: Real and stealthy attacks on state of the art face recognition, ACM CCS 2016

25. Adversarial Samples in the Physical World
Athalye et al, Synthesizing Robust Adversarial Examples https://arxiv.org/pdf/1707.07397.pdf

26. Adversarial Samples in the Physical World
26
Human
Machine
Learning
System

27. Adversarial Samples in the Physical World
27
Human
Machine
Learning
System

28. Adversarial Samples in the Physical World
28
Human
Machine
Learning
System
Generation of adversarial samples can be automated! [Goodfellow et al., Papernot et al.]

29. Adversarial Samples in the Physical World
29
Human
Machine
Learning
System
Generation of adversarial samples can be automated! [Goodfellow et al., Papernot et al.]
Evasion attacks transfer between different machine learning techniques!
[Szegedy et al., Papernot et al.]

30. Automation of Evasion Attacks
Attacks Pros Cons
Fast Gradient Sign Method
(FGSM) [1]
Fastest speed
Low computation cost
Large perturbations
Jacobian Saliency Map Attack
(JSMA) [2]
Small perturbations High computational costs
Carlini Wagner (CW) [3] Minimum perturbations Slowest speed, high
computational cost
[1] Goodfellow et al, Explaining and Harnessing Adversarial Samples https://arxiv.org/abs/1412.6572
[2] Papernot et al, The Limitations of Deep Learning in Adversarial Settings https://arxiv.org/pdf/1511.07528.pdf
[3] Carlini et al, Towards Evaluating the Robustness of Neural Networks https://arxiv.org/pdf/1608.04644.pdf

31. Libraries to Generate Adversarial Samples
● Cleverhans
○ https://github.com/tensorflow/cleverhans
● SecML
○ https://gitlab.com/secml/secml
● IBM Adversarial Robustness Toolbox (ART)
○ https://developer.ibm.com/open/projects/adversarial-robustness-toolbox/
● Foolbox
○ https://github.com/bethgelab/foolbox

32. Transferability of Evasion Attacks
32
● Instances crafted to cause misclassifications on one machine learning system
often cause misclassifications on other machine learning systems, too.
● Adversarial samples transfer between same machine learning techniques.
● Adversarial samples transfer across different machine learning
techniques.

33. Effects of Transferability
Image source Papernot et al, Transferability in Machine Learning: from Phenomena to Black-Box Attacks using
Adversarial Samples https://arxiv.org/abs/1605.07277

34. Why Such Attacks Happen?
● Learning algorithms overemphasize importance of some features
○ Even small modifications of such features can cause large changes to the prediction
outcome [1]
● Different classifiers tend to find the same set of relevant features
○ Which facilitates attacks to transfer across the models [2]
[1] Simon-Gabriel at al, First-order Adversarial Vulnerability of Neural Networks and Input Dimension
https://arxiv.org/abs/1802.01421
[2] Demontis et al, Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion andPoisoning Attacks
https://arxiv.org/pdf/1809.02861.pdf

35. Can We Defend?

36. Defenses Against Poisoning Attacks
● Exclude instances with significant negative impact
○ Reject on Negative Impact defence
○ Avoid to assign regions far from training data marked as legitimate
● Data sanitization
○ remove poisoning samples from training data using outlier detection and data filtering

37. Defenses Against Evasion Attacks
● Ensemble
● Adversarial retraining
● Defensive distillation
● Dimensionality reduction
● Regularization
● Using fuzzy boundaries and membership score as a feature

38. Defenses Against Evasion Attacks
● Ensemble
● Adversarial retraining
● Defensive distillation
● Dimensionality reduction
● Regularization
● Using fuzzy boundaries and membership score as a feature
Recent work [1] has shown that all these defences can be evaded
under some circumstances
[1] Carlini et al, Adversarial Examples Are Not Easily Detected:Bypassing Ten Detection Methods
https://nicholas.carlini.com/papers/2017_aisec_breakingdetection.pdf

39. Defenses Against Evasion Attacks
● Ensemble
● Adversarial retraining
● Defensive distillation
● Dimensionality reduction
● Regularization
● Using fuzzy boundaries and membership score as a feature
Recent work [1] has shown that all these defences can be evaded
under some circumstances
Existing defences lack thorough security evaluation
[1] Carlini et al, Adversarial Examples Are Not Easily Detected:Bypassing Ten Detection Methods
https://nicholas.carlini.com/papers/2017_aisec_breakingdetection.pdf

40. Defenses Against Evasion Attacks
● Ensemble
● Adversarial retraining
● Defensive distillation
● Dimensionality reduction
● Regularization
● Using fuzzy boundaries and membership score as a feature
Recent work [1] has shown that all these defences can be evaded
under some circumstances
Existing defences lack thorough security evaluation
Adversarial examples are much more difficult to detect than
previously recognized
[1] Carlini et al, Adversarial Examples Are Not Easily Detected:Bypassing Ten Detection Methods
https://nicholas.carlini.com/papers/2017_aisec_breakingdetection.pdf

41. Other Vulnerabilities of Machine Learning
● Accountability
● Transparency
● Explainability
● Bias and fairness

42. Takeaways
● Security of ML is of growing concern
○ Machine learning is proven to be vulnerable to a set of attacks
○ Some of the attacks are transferable and can be automated

43. Takeaways
● Security of ML is of growing concern
○ Machine learning is proven to be vulnerable to a set of attacks
○ Some of the attacks are transferable and can be automated
● Active countermeasures in order to secure ML should be adopted
○ Some defenses are already proposed but they are still not effective enough
○ Lessons and best practices from security domain should be adapted

44. Takeaways
● Security of ML is of growing concern
○ Machine learning is proven to be vulnerable to a set of attacks
○ Some of the attacks are transferable and can be automated
● Active countermeasures in order to secure ML should be adopted
○ Some defenses are already proposed but they are still not effective enough
○ Lessons and best practices from security domain should be adapted
● Many open problems
○ Some examples are: better defenses, bias, privacy, lack of transparency
○ Future work needed

45. Takeaways
● Security of ML is of growing concern
○ Machine learning is proven to be vulnerable to a set of attacks
○ Some of the attacks are transferable and can be automated
● Active countermeasures in order to secure ML should be adopted
○ Some defenses are already proposed but they are still not effective enough
○ Lessons and best practices from security domain should be adapted
● Many open problems
○ Some examples are: better defenses, bias, privacy, lack of transparency
○ Future work needed
Jelena Milošević
jelena.milosevic@nt.tuwien.ac.at
https://www.linkedin.com/in/milosevicjelena
@DrMrLena
Questions?