In my presentation I will show you a couple of applications that use artificial intelligence in order to improve our security and how easily it is to use other AI to break it. You may like it or not, but natural language processing, deep learning, computer vision are being developed very rapidly and already have significant impact on your life, working behind the scenes of multiple services you use every day.
However, as a great man once said "with great power comes great responsibility", same with the AI - the risk of abuse appears. I will show you how to beat AI using rogue AI, how a crowd-sourced human intelligence can beat AI, or finally how a small, unnoticed by human change in the input data (constructed by AI of course) can severly impact the output of AI processing. I will focus on applications that improve our security not only in the cyber world (like CAPTCHA), but also in real life world (e.g. car safety systems).
Last, but not least, I will tell you how to prevent such abuses and why it is so important to understand how above-mentioned tools work.
3. www.securing.pl@drdr_zz www.securing.pl@drdr_zz
• Intentions were good (bring peace to the world)
„On the day when two army corps may mutually
annihilate each other in a second, probably all civilized
nations will recoil with horror and disband their troops.”
• The reality was different
„I intend to leave after my death a large fund for the
promotion of the peace idea, but I am skeptical as to its
results.”
• The result – Nobel Prize
A story from the 19th century
4. www.securing.pl@drdr_zz
Back to the 21st century
• Artificial Intelligence
• Solves (very efficiently) problems that were unsolvable.
• Will AI revolutionize IT?
8. www.securing.pl@drdr_zz
• Verifier must know which images present cars
• Simple solution:
• A big database of manually categorized images
• AI solution:
• Use AI solution to recognize objects on images
and categorize them
• AI ready to use solutions:
• Inception (GoogleNet)
• AlexNet
• ResNet
• VGG
AI behind reCAPTCHA
9. www.securing.pl@drdr_zz
• Use existing image recognition solutions to solve CAPTCHA puzzles
• Google Reverse Image Search, Clarifai, Alchemy, TDL, NeuralTalk, Caffe
• Target
• Google reCAPTCHA
• Facebook CAPTCHA
Rogue AI for reCAPTCHA
10. www.securing.pl@drdr_zz
• Number of collected CAPTCHA image puzzles
• 63 000 for Google reCAPTCHA
• 200 for Facebook CAPTCHA
• Results
• Google reCAPTCHA – 70% (19 seconds)
• Facebook CAPTCHA – 83%
• With 40.000+ CAPTCHAs per day per host
Rogue AI for reCAPTCHA
12. www.securing.pl@drdr_zz
• Goal
• Automatic support agent
• Uses AI to learn FAQ for new processes
• Natural Language Processing
• Experiment
• Tay (abbr. Thinking about you)
• A twitter account by Microsoft (@TayandYou)
• Designed to mimic the language patterns of
a 19-year-old American girl
AI Chat Bot
13. www.securing.pl@drdr_zz
• Learns from interacting with
human users of Twitter
• Threat
• Knowledge from untrusted source
• Anyone could teach Tay
What can go wrong?
Users posted incorrect and offensive tweets to Tay and made it…
AI Chat Bot
15. www.securing.pl@drdr_zz
• Taken down after 16 hours and 96 000 tweets
• Lesson learned
• Define the boundaries
• Do not allow untrusted source to teach your AI
• The next Tay – Zo
• Twitter, Facebook and Skype
• Does not talk about sensitive topics
Are you ready for a Nazi in your support team?
AI Chat Bot
17. www.securing.pl@drdr_zz
• Artificial Intelligence in Automotive
• Rain sensor
• AI recognizes rain drops on the windshield
• Lane recognition
• Autopilot keeps the car on the lane
• Attack:
• Funny – turn on the wipers
• Scary – make the car to change lane to the opposite
AI in automotive
20. www.securing.pl@drdr_zz
• Attack scenario
• Change the „input image” to fool AI.
• Challenges
• Find out how to change the image.
• Change the physical world.
„Most of the adversarial examples generated in digital domain are pixel level’s
change, so it’s hard to deploy them in physical world.”
AI in automotive
21. www.securing.pl@drdr_zz
• Easy to get if you have access to the AI internals.
Activation map
Learning Deep Features for Discriminative Localization, Zhou et al., MIT
24. www.securing.pl@drdr_zz
• What can you see?
How hard is it to generate malicious input?
Egyptian cat
78% (by alexnet)
Assault rifle
93% (by alexnet)
25. www.securing.pl@drdr_zz
How hard is it to generate malicious input?
Pixels modified Pixels modified
By more than 1%
Pixels modified
By more than 2%
31. www.securing.pl@drdr_zz
Design
• Threat modelling
• Consider rogue AI as
threat
• Define boundaries
AI security
Development
• No untrusted source
teaching your AI
• Generate malicious
inputs and teach your AI
Use
• No critical decisions
based on AI only
• Monitor outputs from
AI (be up to date)
• Control boundaries
Architecture
Assessment
System
Testing
System
Monitoring