a modification of my original gluecon presentation for the AWS denver/boulder meetup

1. VPC - Flying Blind on a Rocket Cycle
Matthew Boeckman - VP of DevOps at Craftsy.com
@matthewboeckman
http://enginerds.craftsy.com

2. Who is Craftsy
● Instructor led training videos for passionate hobbyists
● #19 on Forbes’ Most Promising Companies 2014

3. Three things
Keep it simple
Get there now
Be flexible

4. *disclaimer

5. VPC - Why
VPC is mature network
topology for AWS

6. VPC - Why
Network ACL’s allow for true
edge blocking

7. VPC - Why
Instances can be members of
multiple Security Groups
SG membership can change
post-instance launch

8. VPC - Why
VPN connectivity enables
extension of your corporate or
facilities network to AWS

9. Our stack in ec2-classic

10. What we hate about ec2-classic
● inflexible security groups
● per-IP maintenance of SG’s across regions
● ALLOW TCP 22 FROM 0.0.0.0/0
● no edge
● no edge
● no edge
● no edge

11. Our stack in VPC

12. VPC - flying blind

13. netcat, tcpdump and patience

14. be the packet
host a
host b
SG
SG
ACL
ACL
out
out,in
out,in
out,in
in
out
out,in
out,in
out,in
in

15. flexible security groups!
Office tomcat SG - applies to all tomcat instances
environment specific tomcat SG - only applied for a
specific environment

16. LIMITS
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html

17. LIMITS
100 SG’s per VPC
50 Rules per SG (including inherited rules by
referencing other SG’s)
5 SG’s per network interface (VPC hosts can have
multiple network interfaces)
HARD LIMIT 250 rules per interface
40 ACL’s per subnet (in + out <= 40)

18. routing
Private subnets can only route traffic destined for the internet to a
NAT instance (eni-0…). Public subnets route to the IGW. Routes
can be automatically propagated from VPN connections.

19. NAT instances
HOW BIG?!
● we chose m1-medium… because…. it seems big enough?
sure.
● we were wrong
● upgrading NAT instances - easy, anytime, no interruption
(m3.xl)

20. Connecting to your HQ with a VPN
● AWS docs on this are perfect - check if your firewall is on the supported
list. If so, one click configuration for your firewall
● A VPN connection - includes two tunnels, connected to two different IP’s
at VPC. THESE UNDERGO MAINTENANCE - PRACTICE FAILOVER

21. Cross region VPN
http://aws.amazon.com/articles/5472675506466066
http://fortycloud.com/interconnecting-two-aws-vpc-regions/
AWS has no product offering here. You can easily VPN two VPC’s in
the same region but not, you know, in different regions.
Tunnel MTU size is a thing
iptables -t mangle -A FORWARD -o eth0 -p tcp -m tcp --tcp-flags
SYN,RST SYN -m tcpmss --mss 1361:1536 -j TCPMSS --set-mss
1360

22. seriously?

23. ACL’s ARE NOT STATEFUL
ALLOW tcp 80 src 10.85.0.0/16
ALLOW tcp 443 src 10.85.1.0/24
ALLOW tcp established any
DENY ALL

24. SNS, Redshift, Route53, RDS
SNS - has no legs in VPC. Systems subscribing to SNS topics from private
subnets need an HTTP proxy in a public subnet for SNS to reach them.
Redshift/RDS- has legs in VPC - migrate your redshift or rds instances to
VPC (yay!)
Route53 - no support for “views” in VPC.

25. reservations!
Instance reservations purchased in EC2 classic DO NOT
MAGICALLY MOVE TO VPC
Do. Not. Forget. This. Step.

26. migration time best time
- use AWS support or account teams
- start with subnets and basic nat, vpn
- dev environments, soak
- preprod, soak

27. cloned production

28. shut it down

29. thank you
QUESTIONS!
Matthew Boeckman
@matthewboeckman

30. thank you
QUESTIONS!
Matthew Boeckman
@matthewboeckman
http://enginerds.craftsy.com
(deck will be there)

31. thank you
QUESTIONS!
Matthew Boeckman
@matthewboeckman
http://enginerds.craftsy.com
(deck will be there)

32. public subnet routing
In VPC you define a route table, and associate it with subnet(s). This
example sends all traffic for the defined CIDR block to local, and all
other traffic (0.0.0.0/0) to the internet gateway (igw). Traffic can
also be routed to Elastic Network Interfaces (eni) or VPN gateways
(vgw)

33. Lasciate ogne speranza, voi ch'intrate
tcp4 0 0 192.168.1.24.59021 www.pandora.com.http
ESTABLISHED
TCP connections originate on the sending side from ports 1025-
65535, they arrive at the port you specified (http 80, https 443, dns
53, etc). Return to your request comes back from that socket with
some exceptions (ftp).
ACL’s inbound/outbound are the inverse of your thinking. OUT from
your ec2 instance is IN to the ACL, then OUT to the destination host.
Destination hosts’ reply is IN to the ACL (yep!) and OUT to your ec2
instance.
My advice? Punt. Leave ACL’s as ALLOW ALL, and add DENY
statements against specific CIDR blocks or IP addys. Security Groups
are stateful. Use them.