Explains the basics of IPsec: why IPsec, main IPsec protocols (Authentication Header or AH/Encapsulating Security Payload or ESP), modes (tunnel/transport) and ciphers (MD5/AES). Explains how IPv4 packets are being transformed with IPsec protocols, what are the issues with NAT and what is NAT traversal. At the very end of the presentation there is a real life example for secure communication between two Linux hosts (using ip xfrm).

1. IPsec Basics
Andriy Berestovskyy
2016
( ц ) А н д р
і й Б е р е с
т о в с ь к и
й
networking hour
TCP
UDP
NAT
IPsec
IPv4
IPv6
internet
protocolsAH
ESP
authentication
authorization
accounting
encapsulation
security
BGP
OSPF
ICMP
ACLSNAT
tunnel
PPPoE
GRE
ARP
discovery
NDP
OSI
broadcast
multicast
IGMP
PIM
MAC
DHCP
DNS
fragmentation
semihalf
berestovskyy

2. IPsec?
2

3. Internet Protocol Security (IPsec) — protocol suite for secure
IP communications that works by authenticating and encrypting
each IP packet of a communication session.
— Wikipedia
3

4. p
Why IPsec?
4
Host A Host B
GW 1
Unprotected
Network
GW 2
p
?
Examples?

5. p
p
p
IPsec Rationale
5
Host A Host B
GW 1
Unprotected
Network
GW 2
p p
p
p p
Some
Protection
Unprotected Protected

6. Basic IPsec Terms
Protocol: AH/ESP
Authentication Header — just authenticate
Encapsulating Security Payload — authenticate (optionally) and encrypt
Mode:Transport/Tunnel
Transport — encapsulates only IP payload (data)
Tunnel — encapsulates an entire IP packet (VPN)
Cipher: MD5/SHA-1/3DES/AES...
Hashes — integrity check values for authentication (MD5, SHA-1)
Encryption — using a secret key (3DES, Blowfish, AES, etc)
6
Key
exchange?

7. Secret Key
Key Exchange: IKE/Manual
Internet Key Exchange — secure key exchange mechanism
Manual — out of band key exchange mechanism
Mode: Main/Aggressive
Main — six packets to establish a secure association
Aggressive — three packets, but less secure
7
How to protect
IP packets?

8. IPv4 Packet
Protocol Numbers:
1 — Internet Control Message Protocol (ICMP)
4 — IPv4 Encapsulation (IPv4)
6 — Transmission Control Protocol (TCP)
17 — User Datagram Protocol (UDP)
41 — IPv6 Encapsulation (IPv6)
47 — Generic Routing Encapsulation (GRE)
50 — Encapsulating Security Payload (ESP)
51 — Authentication Header (AH)
...
http://www.iana.org/assignments/protocol-numbers
8
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = TCP IP Checksum
Source IP
Destination IP
TCP Segment
...
0 32
Protected by
IP Checksum

9. Authentication Header
Next Hdr — next protocol number
AH Length — 4-octet units, minus 2, multiple of 8
SPI — security association index on receiving host
Seq Number — strictly increasing number
ICV — authentication hash(es)
9
Next Hdr AH Length Reserved
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value (ICV)
...
0 32
What for?

10. AH in Transport Mode
10
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = AH IP Checksum
Original Source IP
Original Destination IP
Original TCP Segment
...
0 32
Next = TCP AH Length Reserved
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = TCP IP Checksum
Source IP
Destination IP
TCP Segment
...
0 32
Protected by
AH
Protected by
IP Checksum
Why not?

11. AH in Tunnel Mode
11
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = TCP IP Checksum
Source IP
Destination IP
TCP Segment
...
0 32
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = AH IP Checksum
Tunnel Source IP
Tunnel Destination IP
0 32
Next = IP AH Length Reserved
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Original IP Packet
...
Protected by
AH
Protected by
IP Checksum

12. HMAC* Authentication
12
IP Packet
Secret Key 0x363636363636...
XORed Key
0x5c5c5c5c5c5c...
Hash Function
XORed Key Intermediate Hash
Hash Function
Integrity Check Value
* Hash Message Authentication Code

13. Encapsulating Security Payload
SPI — security association index on receiving host
Seq Number — strictly increasing number (replay attack)
Padding — extends payload to cipher block size
Next Hdr — next protocol number
ICV — authentication hash(es)
13
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value (ICV)
...
0 32
Encrypted Payload
...
Padding
(0-255 bytes) Next HdrPad Len

14. ESP in Transport Mode
14
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = TCP IP Checksum
Source IP
Destination IP
TCP Segment
...
0 32
Protected
Protected by
IP Checksum
Encrypted and
Protected
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = ESP IP Checksum
Original Source IP
Original Destination IP
Encrypted TCP Segment
...
0 32
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Padding
(0-255 bytes) Pad Length Next = TCP

15. ESP in Tunnel Mode
15
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = TCP IP Checksum
Source IP
Destination IP
TCP Segment
...
0 32
Protected
Protected by
IP Checksum
Encrypted and
Protected
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = ESP IP Checksum
Tunnel Source IP
Tunnel Destination IP
Encrypted IP Packet
...
0 32
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Padding
(0-255 bytes) Pad Length Next = IP

16. AES* Encryption
16
Round Key 0
* Advanced Encryption Standard
Plain Data
Substitute Bytes
Shift Rows
(Mix Columns)
Encrypted Data
Round Key N
Initial Round
Encryption Rounds
(14 for AES-256)
Note: no Mix Columns at last round

17. IPsec + NAT
Any issues?
17

18. AH - does not work
IPsec + NAT Issue
18
ESP - just one session
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = AH IP Checksum
Source IP
Destination IP
Original TCP Segment
...
0 32
Next = TCP AH Length Reserved
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = ESP IP Checksum
Source IP
Destination IP
Encrypted TCP Segment
...
0 32
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Padding
(0-255 bytes) Pad Length Next = TCP
Why?Why?

19. IPsec + NAT
19
AH - does not work ESP - just one session
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = AH IP Checksum
Translated Source IP
Original Destination IP
Original TCP Segment
...
0 32
Next = TCP AH Length Reserved
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = ESP IP Checksum
Translated Source IP
Original Destination IP
Encrypted TCP Segment
...
0 32
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Padding
(0-255 bytes) Pad Length Next = TCP
ouch!
no ports
:(

20. ESP NAT Traversal
20
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto = ESP IP Checksum
Translated Source IP
Original Destination IP
Encrypted TCP Segment
...
0 32
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Padding
(0-255 bytes) Pad Length Next = TCP
Ver IHL DSCP Total LengthE
ID Fl Fragment Offset
TTL Proto=UDP IP Checksum
Translated Source IP
Original Destination IP
Encrypted TCP Segment
...
0 32
Security Parameters Index (SPI)
Sequence Number
Integrity Check Value
(MD5 or SHA-1)
Padding
(0-255 bytes) Pad Length Next = TCP
Destination Port = 4500
UDP Checksum
Random Source Port
Length

21. p
p
p
Real World Example
21
Host A Host B
GW 1
Unprotected
Network
GW 2
p p
Security
Association
Security
Policy p
p p
Security
Params Idx
Protected by
IP Checksum
Encrypted by
ESP

22. Security Association Database
ip xfrm state add src <SRC-IP> dst <DST-IP> proto esp
[auth <ALGO> <KEY>] enc <CIPHER> <KEY>
# xxd -p -l 16 /dev/urandom
a4adda25cba555587d29b22135e3c174
# ip xfrm state add src 172.16.1.3 dst 172.16.1.4 proto esp
spi 0x1 auth md5 MyPass
enc aes 0xa4adda25cba555587d29b22135e3c174
# ip xfrm state add src 172.16.1.4 dst 172.16.1.3 proto esp
spi 0x2 auth md5 MyPass
enc aes 0xa4adda25cba555587d29b22135e3c174
22
# ip xfrm state show
src 172.16.1.4 dst 172.16.1.3
proto esp spi 0x00000002 reqid 0 mode transport
replay-window 0
auth-trunc hmac(md5) 0x4d7950617373 96
enc cbc(aes) 0xa4ad...
sel src 0.0.0.0/0 dst 0.0.0.0/0
src 172.16.1.3 dst 172.16.1.4
proto esp spi 0x00000001 reqid 0 mode transport
replay-window 0
auth-trunc hmac(md5) 0x4d7950617373 96
enc cbc(aes) 0xa4ad...
sel src 0.0.0.0/0 dst 0.0.0.0/0

23. Security Policy Database
ip xfrm policy add <SELECTOR> dir <DIR> tmpl <TEMPLATE>
# ip xfrm policy add dev eth1 dir out tmpl proto esp
23
# ip xfrm policy show
src 0.0.0.0/0 dst 0.0.0.0/0 dev eth1
dir out priority 0
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 0 mode transport

24. Checklist
1. IPsec?
2. Transport vs tunnel?
3. AH?
4. ESP?
5. AH vs ESP?
6. HMAC?
7. Security Policy?
8. Security Association?
9. NAT traversal?
10. Linux tool?
24

25. References
1. RFC 4301. Security Architecture for the Internet Protocol.
2. RFC 4302. IP Authentication Header.
3. RFC 4303. IP Encapsulating Security Payload (ESP).
4. RFC 4306. Internet Key Exchange (IKEv2) Protocol.
5. An Illustrated Guide to IPsec: http://unixwiz.net/techtips/iguide-ipsec.html
6. IANA Protocol Numbers: http://www.iana.org/assignments/protocol-numbers
25