Fighting fake registrations, phishing, spam and other types of abuse on the consumer web appears at first glance to be an application tailor-made for machine learning: you have lots of data and lots of features, and you are looking for a binary response (is it an attack or not) on each request. However, building machine learning systems to address these problems in practice turns out to be anything but a textbook process. In particular, you must answer such questions as: - How do we obtain quality labeled data? - How do we keep models from "forgetting the past"? - How do we test new models in adversarial environments? - How do we stop adversaries from learning our classifiers? In this talk I will explain how machine learning is typically used to solve abuse problems, discuss these and other challenges that arise, and describe some approaches that can be implemented to produce robust, scalable systems.

1. "You can't just turn the crank"
Machine learning for fighting abuse on the consumer web
David Freeman
Research Scientist/Engineer, Facebook Inc.
ScAINet 2018
Atlanta, GA USA, 11 May 2018

2. The consumer web

3. What do they try to do?
Malware
Payment
Fraud
Scraping
Click
Fraud
Phishing Spam Social
Engineering
Fake
Products
Scams
"Like"
FraudPromotion
Fraud
Identity
Theft
What do we see?
Fake
Reviews
Misinfor-
mation
Financial
Theft
Account
Resale
Fundamental question: Which requests are bad?
• Perfect for machine learning!

4. What could possibly go wrong?
Machine learning workflow
Label
Train
Validate
Launch
Measure
Profit!Lots!

5. How do we obtain labeled data?
(hint: not from your users)
Machine learning workflow
Label

6. • Human labeling of random samples.
• Labelers don't always know what they're looking for
• Labelers are inconsistent (with themselves and each other)
• Labelers get tired (esp. if most samples are good)
• Apply crowdsourcing best practices:
• Precise definitions, multiple labeling, ML-assisted sampling
• But will it scale?
Labeling: Gold standard
Objective measurement

7. • Find high-precision signals of badness
• Examples: unusual user-agent, malformed header
• DO NOT BLOCK ON THESE SIGNALS
• They are controlled by the adversary
• When the adversary adapts you will lose visibility
• Automatically generate signals using anomaly detection.
Labeling: Silver standard
Automatic labeling

8. • Use whatever you have!
• CS data, rules, other models
• Mitigate risks of blindness and feedback loops:
• Oversample manually labeled examples
• Oversample false positives and false negatives when retraining.
• Undersample positive examples from previous iterations of this model.
• Sample and label examples near the decision boundary
Labeling: Bronze standard
Be scrappy

9. • Users are terrible at
reporting.
• Product flows bias
reporting.
• Reports can be gamed.
• Reports can serve as an
directional measure.
Labeling: Iron standard
Have your users do the work

10. • Segment the problem
• e.g. status with link from country X
• Downsample intelligently
• if your distribution is lumpy, sample from all the lumps
• Learning the prior vs. focusing on the bad stuff
• no golden rule here -- you have to experiment
Assembling a training set
Labeling is just the beginning

11. {Training set 2
{
Model v2
Refreshing your data
Don't forget the past!
{Training set 1 {
Model v1
Mitigation:
• Keep old attacks around (exponential decay?)
• Keep old models around (raise thresholds?)
{
Training set N
{
Model vN

12. How do you know your model is ready to go?
Machine learning workflow
Train
Validate

13. • Labels aren't perfect
• Often miss on recall
• Models interact with each other
• Use offline P-R and ROC to stack-rank model candidates
Validating Performance
Don't trust offline replay Model B
FP
Model A

14. • Fundamental A/B testing assumption:
Experiment effects are independent of the cohorts chosen
The Perils of A/B Testing

15. The Perils of A/B Testing
A B
X
• Looks good so far....
Start with a small experiment

16. The Perils of A/B Testing
A B
X
• Did the adversary give up or iterate?
Roll it out to (almost) everyone — Option 1

17. The Perils of A/B Testing
A B
• Now your experiment is a vulnerability
Roll it out to (almost) everyone — Option 2

18. • Run new model online in "log-only" mode
• Evaluate performance where the new
model disagrees with the old one.
• ideally via sampling & labeling
• Push based on FP/FN tradeoff
Using Shadow Mode
Prod model
FP
New model

19. How do you figure out if it worked?
Machine learning workflow
Launch
Measure

20. True Positives Don't Matter
What's happening here?
Time
Precision

21. • Really want # of good users affected
• Solution: use one minus specificity (aka FPR)
True Positives Don't Matter
What's happening here?
Time
TP
Time
FP vs.
Time
FP
Time
TP
1
TN
FP + TN<latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit><latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit><latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit><latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit>

22. Not so fast!
Machine learning workflow
Profit!Adapt!

23. What not to Do (I)
Show the adversary what your limits are
Message 500 people
Message 400 people
Message 300 people
🛑
🛑
✅

24. • Introduce delay in blocking
response (and/or)
• Undo the damage without
telling the user.
What to do (I)
Don't give immediate feedback

25. "We don't want to be the ones solving the CAPTCHAs"
What not to Do (II)
Look for specific content to block

26. What to Do (II)
Focus on bad behavior, not only bad content

27. What to Do (III)
Use data the adversary doesn't know/control

28. Scoring at Entry Points
prevent access to accounts
Clustering, Anomaly Detection
prevent accounts from doing damage
User Reporting
find false negatives
Behavioral Analysis
detect bad activityIncreasing
speed
More
information
available
What to Do (IV)
Defense in depth

29. • Think about each step of the ML process.
• It's hard to build a good training set.
• Adversarial adaptation breaks many assumptions.
• Control the data & the response.
Take aways
Thanks to: Hervé Robert, Isaac Fullinwider, Henry Lu, Sagar Patel, Hongyang Li, Nektarios Leontiadis