Shakacon X (2018) Presentation

1. Putting the Bits Back
Together
Forensic investigations in AWS EC2

2. $ whoami
Brandon Sherman
Senior Cloud Security
Engineer at Twilio, Inc.
Krav Maga instructor
Has cloudy outlook
Certified Solutions
Architect, Professional

3. $ whoami
Brandon Sherman
Senior Cloud Security
Engineer at Twilio, Inc.
Krav Maga instructor
Has cloudy outlook
Certified Solutions
Architect, Professional
A picture, just in case
you’ve forgotten what I look
like already
Or are in the back
This is a better looking
photo than the live version
the people at the front get

4. ~/agenda
Why?
Background on AWS, EC2, and EBS
Research, Methodology, and Findings
Protecting your forensic data

5. – Twilio value
“Start with why.”

6. $ cd ~/why?
💩 happens
(citation needed)

7. ~/why?
Compromises happen
@TODO: fill in with the most recent breach

8. ~/why?
Prevention is ideal
Detection is a must
If something bad happens, how can we determine how
bad that something was?

9. ~/why?
Forensic Investigation
Investigate an event in the past by utilizing scientific
techniques
Post-event is not the time to be developing
techniques

10. ~/why?
Personally: What is hidden is much more interesting
than what is visible
Has an attacker…
…deleted log files?
…caused log files to roll over?
…used a dropper which erased itself?

11. $ cd ~/the_cloud
Super-high-level overview of the cloud
$ ln -s ~/the_cloud /clouds/aws

12. ~/the_cloud/
This is not meant to be a introductory course in AWS
Also not meant to be an advanced course in AWS
Amazon Web Services is comprised of many, many,
many, many, many, many, many services

14. ls ~/the_cloud/
Focusing today on
EC2: Elastic Cloud Compute
EBS: Elastic Block Storage
IAM: Identity and Access Management
S3: Simple Storage Service

16. $ cd ~/the_cloud/ec2
Elastic Cloud Compute
Virtual machines on-demand
Plus now, even a few bare-metal ones
Varying capabilities, CPU, RAM, etc.

17. ~/the_cloud/ec2
Click button, get server
Script with a loop, hit API, get lots of servers
Default is shared tenancy, but you can get various
degrees of dedicated hardware

18. ~/the_cloud/ec2
The starting point for an EC2 instance is an AMI
Amazon Machine Image
This contains the host OS, configuration, etc.
Possible to create an AMI from a running instance—
including configuration files, etc.

19. ~/the_cloud/ec2
Some instances have Instance Storage
Some instances don’t
Instance Storage is directly-connected disk
Fast
Ephemeral

20. $ cd ../ebs
Elastic Block Storage
Request a volume, specify its size, and where you want
to attach it
Kinda-sorta network attached storage that presents
locally
Various backing stores

21. ~/the_cloud/ebs
standard gp2 io1 st1 sc1
Backing
material
HDD SSD SSD HDD HDD
Sizes 1GB–1TB
1GB–
16TB
4GB–
16TB
500GB–
16TB
500GB–
16TB
Max
Throughput
(volume)
90MBps 160MBps 500MBps 500MBps 250MBps
Price
$0.05/
GB/month
$0.10/
GB/month
$0.125/
GB/month
$0.045/
GB/month
$0.025/
GB/month
AFR
0.1% –
0.2%
0.1% –
0.2%
0.1% –
0.2%
0.1% –
0.2%
0.1% –
0.2%

22. ~/the_cloud/ebs
EBS has an AFR of
<0.2%
Commodity drive AFR is
~4%

23. ~/the_cloud/ebs
EBS is 20x more
durable than regular
drives!
How? Magic!

24. ~/the_cloud/ebs
Recap:
Disk storage which…
…is network attached storage…
…of various capabilities…
…presenting as either NVMe or SATA…
…with dynamic sizes…
…but of static allocation…
…which fail 1/20th as often as a single drive
…and can be snapshotted via API call

25. $ cd ../iam
Identity and Access Management
aka permission controls in AWS
Relatively fine-grained; who can perform which API
calls on what resources
We’ll get into this more later— it ties into how to
protect the chain of custody

26. $ cd ../s3
EBS volumes can have a snapshot taken
Does not have to be when the disk is detached, but be
wary of inconsistencies when snapshotting an active
volume
Snapshots are stored as blobs in S3

27. ~/the_cloud/s3
Snapshots are immutable
AMIs are special EBS snapshots
“Blessed” to be usable as a boot volume
Instance root volumes can be EBS or Ephemeral
storage

28. Story Time!
Alice
Bob
The Third Party
The story, all names, characters, and incidents portrayed
in this production are fictitious. No identification with
actual persons (living or deceased), places, buildings,
and products is intended or should be inferred.

29. Alice and Bob communicate with each other via the
“WhatsMyGramBook” service
“WhatsMyGramBook” uses one server, which was
breached
Were their communications accessed? What did the
attacker do?

30. ~/the_cloud/questions
Question One: If a snapshot of an EBS volume is
taken, will that snapshot only contain in-use blocks, or
are deleted blocks also included?
Question Two: Does it matter what the original EBS
volume type is? Has Amazon changed their
implementation between versions?
Question Three: Does the instance type matter? Does
NVMe vs. SATA make a difference?

31. ~/research/process
1. Launch a selection of EC2 instances
2. Attach one of each EBS volume type to each class of instance
3. Write files
4. Delete files
5. Snapshot disks
6. Rehydrate snapshot to new disk
7. Look for files

37. WMGB sysadmin/founder/developer/person logs into
the AWS Console and triggers a snapshot of the EBS
volumes attached to their EC2 instance
Now to see what happened on that volume

38. ./photorec
PhotoRec is freely available software to look for deleted
files
https://www.cgsecurity.org/wiki/PhotoRec
Looks at the raw blocks of a disk and compares data
to known file signatures

39. ./scripts/forensics.rb
With each snapshot, rehydrate to an EBS volume
Attach the volume to an instance
Run PhotoRec and look for deleted files

40. $ pry forensics.rb
Launching new Investigate instance
Waiting for i-04074842f4a3a8c10 to enter running state...
Executing setup commands
Waiting for command to be in finished, currently is Success
Command execution successful! Output:
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3276800 inodes, 13107200 blocks
655360 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2162163712
400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
download: s3://forensics-software/testdisk-7.0/AUTHORS to ../../root/testdisk/AUTHORS
download: s3://forensics-software/testdisk-7.0/INFO to ../../root/testdisk/INFO
download: s3://forensics-software/testdisk-7.0/documentation.html to ../../root/testdisk/documentation.html
download: s3://forensics-software/testdisk-7.0/fidentify.8 to ../../root/testdisk/fidentify.8
download: s3://forensics-software/testdisk-7.0/icons/48x48/apps/qphotorec.png to ../../root/testdisk/icons/48x48/
apps/qphotorec.png
download: s3://forensics-software/testdisk-7.0/THANKS to ../../root/testdisk/THANKS
download: s3://forensics-software/testdisk-7.0/NEWS to ../../root/testdisk/NEWS
download: s3://forensics-software/testdisk-7.0/COPYING to ../../root/testdisk/COPYING
download: s3://forensics-software/testdisk-7.0/Android.mk to ../../root/testdisk/Android.mk
download: s3://forensics-software/testdisk-7.0/ChangeLog to ../../root/testdisk/ChangeLog
download: s3://forensics-software/testdisk-7.0/icons/photorec.ico to ../../root/testdisk/icons/photorec.ico

41. Results!

42. Lots of files are returned!
WhatsMyGramBook team is very happy they attended
this talk
If WhatsMyGramBook had their log files on the root
EBS volume, they should expect to see many files
recovered from the AMI
Can be an overwhelming amount of returned files

43. AMIs are snapshots
AMIs therefore contain deleted files
Using PhotoRec on an EBS volume that was the
startup volume can net large amounts of recovered
files

45. Fair enough— I’ve told
the end of the story
before showing you the
results
How do we know it’s
possible to recover all
these files?

46. Pass Match the Hash

47. ~/scripts/compare.sh
Frequently, more files are returned than originally
seeded to disk
This is due to PhotoRec guessing where files end and
new files begin
Especially with text files, which don’t have clearly
defined “magic numbers” and EOF markers

49. ~/scripts/compare_bytes.sh
Comparing the first n bytes is a reliable way to
determine how many of the original files are recovered

50. $ grep -v -e '^/' analysis
Examining 84 files in /recovery/
c5.large/gp2.1/
Recovered: 84
Matches:48/56
Examining 88 files in /recovery/
c5.large/io1.1/
Recovered: 88
Matches:52/56
Examining 37 files in /recovery/
c5.large/sc1.1/
Recovered: 37

51. ~/findings
Source instance type has no detectable effect
Recovery success varied based on source volume type
Best recovery rates:
Standard, gp2, io1
Less-good rates:
sc1, st1

52. ~/findings
Examining 84 files in /mnt/forensic_recovery/c5.large/gp2.1/
Matches: 48/56 (86%)
Examining 88 files in /mnt/forensic_recovery/c5.large/io1.1/
Matches: 52/56 (93%)
Examining 37 files in /mnt/forensic_recovery/c5.large/sc1.1/
Matches: 25/56 (47%)
Examining 11 files in /mnt/forensic_recovery/c5.large/st1.1/
Matches: 11/56 (20%)
Examining 86 files in /mnt/forensic_recovery/c5.large/standard.1/
Matches: 50/56 (89%)

53. ~/findings
Weird artifacts
Recovery of PDFs from sc1/st1 based drives resulted
in massive files
…but not other drive types

54. /dev/nvme1n1 493G 196G 272G 42% /mnt/forensic_recovery272G

55. $ find . -printf '%s %pn'|sort -nr|head -n 5
161053454336 ./m3.medium/st1.1/f209993256.pdf
24292724736 ./m5.large/st1.1/f428097024.pdf
21367603200 ./t2.nano/st1.1/f948187936.pdf
17745333 ./t2.nano/standard.1/f0475136.m4p
17745333 ./t2.nano/st1.1/f0393216.m4p

56. $ find . -printf '%s %pn'|sort -nr|head -n 5
161 GB ./m3.medium/st1.1/f209993256.pdf
24.30 GB ./m5.large/st1.1/f428097024.pdf
21.36 GB ./t2.nano/st1.1/f948187936.pdf
0.18 GB ./t2.nano/standard.1/f0475136.m4p
0.18 GB ./t2.nano/st1.1/f0393216.m4p

57. Not sure why— this could be a recovery issue of
detecting the EOF marker further away from the start of
the PDF
But it only appeared when the source was an sc1/st1
volume

59. Chain of Custody
Snapshots can be shared to other accounts
If an attacker is in your account, they could delete the
snapshots as you take them, causing data to be lost
Copy to another, secured, account to keep them safe

62. $ sort -u ~/talk
What does your threat model look like?
Do you need high-quality forensics?
Yes: Don’t use sc1/st1/ephemeral
No: Understand the limitations

63. $ sort -u ~/talk
Consider writing only to non-root EBS volumes
Eliminates the large number of recoverable files
deleted from the AMI
Quite possibly too much noise to see the signal of
deleted files

64. $ sort -u ~/talk
Use multiple accounts
A breach of a server could mean the breach of all your
stuff
The loss of an API key could mean the loss of all your
stuff
An account limits the blast radius
Keep your forensics out of that blast radius

65. Thank you!

66. Few quick things…
We’re hiring! If you are interested, or know someone who’s
interested:
https://boards.greenhouse.io/twilio
We have a bug bounty program!
https://bugcrowd.com/twilio
SMS ‘shakacon’ to 213.27.SHAKA for my business card!
213.27.74252

67. head -n 5 /dev/random/
questions
SMS ‘shakacon’ to 213.27.SHAKA for my business card!
213.277.4252