Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’
nicht hilft - Verbindungsprobleme in AWS lösen
Steffen Gebert (@StGebert)
Wolfgang Schäfer (@wo_wue)
AWS Community Day DACH in Dresden
19.10.2022
2
This is Our Architecture
3
VPC left
10.1.0.0/16
VPC right
10.2.0.0/16
AWS Transit
Gateway
EC2 instance
client Amazon API
Gateway
VPC Endpoint
Transit Gateway
Attachment
Transit Gateway
Attachment
Problem
Problem
Problem
Problem
Problem
VPC
Reachability
Analyzer
VPC
Reachability
Analyzer
VPC
Reachability
Analyzer
VPC
Reachability
Analyzer
VPC
Reachability
Analyzer
VPC
Reachability
Analyzer
VPC
Reachability
Analyzer
VPC
Reachability
Analyzer
Fixing
Connectivity
VPC
Reachability
Analyzer
VPC
Reachability
Analyzer
Connectivity
Test
Metrics
This is Our Architecture
22
VPC left
10.1.0.0/16
VPC right
10.2.0.0/16
AWS Transit
Gateway
EC2 instance
client Amazon API
Gateway
VPC Endpoint
Transit Gateway
Attachment
Transit Gateway
Attachment
Metrics
Metrics
Metrics
Transit GW
Metrics
Transit GW
§ Per TGW and per
TGW Attachments
§ In and out bytes and
packets
§ Blackhole and NoRoute
metrics
Metrics
Transit GW
§ Per TGW and per
TGW Attachments
§ In and out bytes and
packets
§ Blackhole and NoRoute
metrics
§ Custom Dashboard
Metrics
Transit GW
§ Automatic Dashboard
“VPC Transit Gateway”
Metrics
Transit GW
Flow Logs
30
Flow Logs
§ VPC Flow Logs
§ TGW Flow Logs
new
32
CloudWatch Logs Insights
33
CloudWatch Logs Insights
34
Reachability Analyzer zu Flow Logs
35
Reachability Analyzer zu Flow Logs
36
Reachability Analyzer zu Flow Logs
37
Flow Logs – Additional Destinations
§ S3 and Kinesis Firehose
§ Use cases
• Continuous monitoring
• Retrospective analysis
Packet Capture
Wireshark
§ tcpdump running on
client instance
§ Streamed through SSH or
SSM connection
Wireshark
§ tcpdump running on
client instance
§ Streamed through SSH or
SSM connection
§ Comfortably displayed on
local computer
Wireshark
§ tcpdump running on
client instance
§ Streamed through SSH or
SSM connection
§ Comfortably displayed on
local computer
§ Filter out own traffic!
Wireshark
§ tcpdump running on
client instance
§ Streamed through SSH
connection
§ Comfortably displayed on
local computer
§ Filter out own traffic!
VPC left
10.1.0.0/16
VPC right
10.2.0.0/16
AWS Transit
Gateway
EC2 instance
client Amazon API
Gateway
VPC Endpoint
Transit Gateway
Attachment
Transit Gateway
Attachment
Transit Gateway
Attachment
VPC capture
10.99.0.0/16
EC2 instance
capture-receiver
VPC left
10.1.0.0/16
VPC right
10.2.0.0/16
AWS Transit
Gateway
EC2 instance
client Amazon API
Gateway
VPC Endpoint
Transit Gateway
Attachment
Transit Gateway
Attachment
VPC Traffic Mirroring
43
source
target
filter
session
VPC Traffic
Mirroring
§ Packets duplicated by Nitro
§ Accounts to packet/sec
limits of EC2 instance
§ Requires connectivity from
source to target
§ Only for EC2 instances
VPC Traffic
Mirroring
§ Packets duplicated by Nitro
§ Accounts to packet/sec
limits of EC2 instance
§ Requires connectivity from
source to target
§ Only for EC2 instances
VPC Traffic
Mirroring
§ Packets duplicated by Nitro
§ Accounts to packet/sec
limits of EC2 instance
§ Requires connectivity from
source to target
§ Only for EC2 instances
VPC Traffic
Mirroring
§ Packets duplicated by Nitro
§ Accounts to packet/sec
limits of EC2 instance
§ Requires connectivity from
source to target
§ Only for EC2 instances
VPC Traffic
Mirroring
§ Packets duplicated by Nitro
§ Accounts to packet/sec
limits of EC2 instance
§ Requires connectivity from
source to target
§ Only for EC2 instances
VPC Traffic
Mirroring
§ Packets duplicated by Nitro
§ Accounts to packet/sec
limits of EC2 instance
§ Requires connectivity from
source to target
§ Only for EC2 instances
VPC Traffic
Mirroring
§ Packets duplicated by Nitro
§ Accounts to packet/sec
limits of EC2 instance
§ Requires connectivity from
source to target
§ Only for EC2 instances
VPC Traffic
Mirroring
§ Packets duplicated by Nitro
§ Accounts to packet/sec
limits of EC2 instance
§ Requires connectivity from
source to target
§ Only for EC2 instances
VPC Traffic
Mirroring
§ Capturing now on target
instance
§ Packets received in
VXLAN encapsulation
That’s fun!
N O B O D Y E V E R D O I N G T H I S
53
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
Can it be
easier?
§ Aidan Steele’s projects
§ flowdogshark (GWLB)
§ vpcshark *
§ More concept studies than
for production
https://github.com/aidansteele/flowdog
https://github.com/aidansteele/vpcshark (* not yet publicly released)
When nothing helps…
Ask your AWS Account Team
THANKS Karl!
EMnify IoT Communication Cloud
67
User
Interface
EMN IFY IOT
COMMU N ICATION
CLOU D
API
Data
Event
Stream
CELLU LAR
N ETW OR K
Customer
IoT Applications
Customer
Operations Team
EMNI F Y SI M
CU STOMER
IOT DEVICE
Dr. Steffen Gebert Wolfgang Schäfer
68
Your Trouble Shooters
§ Director Technology, Infrastructure
§ @StGebert
§ Senior Core Network Engineer
§ @wo_wue
69
Learn from our mistakes!
§ IaC definition of the setup used in this talk
• Terraform
• incl. Reachability Analyzer and Traffic Mirroring
§ github.com/EMnify/
One More
Try
Oh.. Layer 8 issues J
Agenda
1. Problem Scenario
2. VPC Reachability Analyzer
3. Metrics
4. Flow Logs
5. Packet capture
6. About us
7. Your questions, please!

Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungsprobleme in AWS lösen

  • 1.
    Wenn selbst ‘erlaubeallen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungsprobleme in AWS lösen Steffen Gebert (@StGebert) Wolfgang Schäfer (@wo_wue) AWS Community Day DACH in Dresden 19.10.2022
  • 2.
  • 3.
    This is OurArchitecture 3 VPC left 10.1.0.0/16 VPC right 10.2.0.0/16 AWS Transit Gateway EC2 instance client Amazon API Gateway VPC Endpoint Transit Gateway Attachment Transit Gateway Attachment
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
    This is OurArchitecture 22 VPC left 10.1.0.0/16 VPC right 10.2.0.0/16 AWS Transit Gateway EC2 instance client Amazon API Gateway VPC Endpoint Transit Gateway Attachment Transit Gateway Attachment
  • 23.
  • 24.
  • 25.
  • 26.
    Metrics Transit GW § PerTGW and per TGW Attachments § In and out bytes and packets § Blackhole and NoRoute metrics
  • 27.
    Metrics Transit GW § PerTGW and per TGW Attachments § In and out bytes and packets § Blackhole and NoRoute metrics
  • 28.
  • 29.
    § Automatic Dashboard “VPCTransit Gateway” Metrics Transit GW
  • 30.
  • 31.
    Flow Logs § VPCFlow Logs § TGW Flow Logs new
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
    37 Flow Logs –Additional Destinations § S3 and Kinesis Firehose § Use cases • Continuous monitoring • Retrospective analysis
  • 38.
  • 39.
    Wireshark § tcpdump runningon client instance § Streamed through SSH or SSM connection
  • 40.
    Wireshark § tcpdump runningon client instance § Streamed through SSH or SSM connection § Comfortably displayed on local computer
  • 41.
    Wireshark § tcpdump runningon client instance § Streamed through SSH or SSM connection § Comfortably displayed on local computer § Filter out own traffic!
  • 42.
    Wireshark § tcpdump runningon client instance § Streamed through SSH connection § Comfortably displayed on local computer § Filter out own traffic!
  • 43.
    VPC left 10.1.0.0/16 VPC right 10.2.0.0/16 AWSTransit Gateway EC2 instance client Amazon API Gateway VPC Endpoint Transit Gateway Attachment Transit Gateway Attachment Transit Gateway Attachment VPC capture 10.99.0.0/16 EC2 instance capture-receiver VPC left 10.1.0.0/16 VPC right 10.2.0.0/16 AWS Transit Gateway EC2 instance client Amazon API Gateway VPC Endpoint Transit Gateway Attachment Transit Gateway Attachment VPC Traffic Mirroring 43 source target filter session
  • 44.
    VPC Traffic Mirroring § Packetsduplicated by Nitro § Accounts to packet/sec limits of EC2 instance § Requires connectivity from source to target § Only for EC2 instances
  • 45.
    VPC Traffic Mirroring § Packetsduplicated by Nitro § Accounts to packet/sec limits of EC2 instance § Requires connectivity from source to target § Only for EC2 instances
  • 46.
    VPC Traffic Mirroring § Packetsduplicated by Nitro § Accounts to packet/sec limits of EC2 instance § Requires connectivity from source to target § Only for EC2 instances
  • 47.
    VPC Traffic Mirroring § Packetsduplicated by Nitro § Accounts to packet/sec limits of EC2 instance § Requires connectivity from source to target § Only for EC2 instances
  • 48.
    VPC Traffic Mirroring § Packetsduplicated by Nitro § Accounts to packet/sec limits of EC2 instance § Requires connectivity from source to target § Only for EC2 instances
  • 49.
    VPC Traffic Mirroring § Packetsduplicated by Nitro § Accounts to packet/sec limits of EC2 instance § Requires connectivity from source to target § Only for EC2 instances
  • 50.
    VPC Traffic Mirroring § Packetsduplicated by Nitro § Accounts to packet/sec limits of EC2 instance § Requires connectivity from source to target § Only for EC2 instances
  • 51.
    VPC Traffic Mirroring § Packetsduplicated by Nitro § Accounts to packet/sec limits of EC2 instance § Requires connectivity from source to target § Only for EC2 instances
  • 52.
    VPC Traffic Mirroring § Capturingnow on target instance § Packets received in VXLAN encapsulation
  • 53.
    That’s fun! N OB O D Y E V E R D O I N G T H I S 53
  • 54.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 55.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 56.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 57.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 58.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 59.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 60.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 61.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 62.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 63.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 64.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 65.
    Can it be easier? §Aidan Steele’s projects § flowdogshark (GWLB) § vpcshark * § More concept studies than for production https://github.com/aidansteele/flowdog https://github.com/aidansteele/vpcshark (* not yet publicly released)
  • 66.
    When nothing helps… Askyour AWS Account Team THANKS Karl!
  • 67.
    EMnify IoT CommunicationCloud 67 User Interface EMN IFY IOT COMMU N ICATION CLOU D API Data Event Stream CELLU LAR N ETW OR K Customer IoT Applications Customer Operations Team EMNI F Y SI M CU STOMER IOT DEVICE
  • 68.
    Dr. Steffen GebertWolfgang Schäfer 68 Your Trouble Shooters § Director Technology, Infrastructure § @StGebert § Senior Core Network Engineer § @wo_wue
  • 69.
    69 Learn from ourmistakes! § IaC definition of the setup used in this talk • Terraform • incl. Reachability Analyzer and Traffic Mirroring § github.com/EMnify/
  • 70.
  • 71.
    Agenda 1. Problem Scenario 2.VPC Reachability Analyzer 3. Metrics 4. Flow Logs 5. Packet capture 6. About us 7. Your questions, please!