Beyond TCP: The evolution of Internet transport protocols

Beyond TCP: The evolu0on of
Internet transport protocols
Olivier Bonaventure
UCL

h2p://inl.info.ucl.ac.be
Paris, Polytechnique, Jan, 2016

Agenda
•  Internet transport protocols
– TCP
– SCTP
•  MulKpath TCP
– Basic principles
– Use cases
•  What's next ?
– QUIC

The origins of TCP
Source : h2p://spectrum.ieee.org/compuKng/soRware/the-strange-birth-and-long-life-of-unix

The Unix pipe model
echo wc
1234 abbsbbbs

The TCP bytestream model
Client Server
ABCDEF...111232
0988989 ... XYZZ
IP:1.2.3.4 IP:4.5.6.7

CongesKon collapse
JACOBSON, V. CongesKon avoidance and control. In Proceedings of SIGCOMM ’88 (Stanford,
CA, Aug. 1988), ACM.

Performance issues
•  TCP considered to be too complex by many
– SoRware implementaKon cannot cope with
increasing network bandwidth
•  For high performance, transport should be
implemented in hardware
– Transputers
– Simpler transport protocols

More limitaKons of TCP
•  Issues with the TCP pipe model
– Only supports a single bytestream
•  Some applicaKons need several streams with prioriKes
– No support for messages
– ConnecKons are a2ached to one IP address on
client and one IP address on server
•  No failover even if hosts have mulKple interfaces
•  No support for mobility
•  No load balancing for mulKhomed hosts

SCTP in two slides
•  Modern transport protocol
–  Cleaner connecKon establishment
•  Four-way handshake to counter SYN ﬂooding a2acks
–  Cleaner protocol
•  Flexible TLV packet format that is easy to extend
•  SelecKve acknowledgements from the start
–  Richer semanKcs
•  Messages, mulKple streams, unreliable delivery
•  Advanced API to replace socket API
–  Failover support
•  ConnecKon can move from one IP address to another one

SCTP connecKon establishment
INIT,Itag=1234
INIT-ACK,cookie,ITag=5678
COOKIE-Echo,Vtag=5678, cookie
COOKIE-ACK,Vtag=1234
Encrypt state in cookie,
Does not store it
Decrypts cookie,
Recover info to create state

What went wrong with SCTP ?
•  Replacing a transport protocol
Physical
Datalink
Network
TCP
Application
SCTP
ApplicaKons must be
rewri<en with new API
IP protocol=132
For SCTP packets

Deploying SCTP
•  ApplicaKons developers will invest in SCTP as
soon as SCTP is implemented on
– Clients
– Servers

The Internet architecture
that we explain to our students
Physical
Datalink
Network
Transport
Application
O. Bonaventure, Computer networking : Principles, Protocols and PracKce, open ebook, h2p://inl.info.ucl.ac.be/cnp3
Physical
Physical
Datalink
Physical
Datalink
Network

SCTP deployment
Physical
Datalink
Network
Transport
Application
Physical
Datalink
Network
Transport
Application
Physical
Datalink
Network
Physical
Datalink
TCP SCTP
SCTP SCTP

In reality
– almost as many middleboxes as routers
– various types of middleboxes are deployed
Sherry, JusKne, et al. "Making middleboxes someone else's problem: Network processing as a cloud service."
Proceedings of the ACM SIGCOMM 2012 conference. ACM, 2012.

Internet devices according to Cisco
h2p://www.cisco.com/web/about/ac50/ac47/2.html
Web Security
Appliance
NAC Appliance
ACE XML
Gateway
Streamer
VPN Concentrator
SSL
Terminator
Cisco IOS Firewall
IP Telephony
Router
PIX Firewall
Right and LeR
Voice
Gateway VVVV
Content
Engine
NAT

Middleboxes in the architecture
•  In the oﬃcial architecture, they do not exist
•  In reality...
Physical
Datalink
Network
Transport
Application
Physical
Datalink
Network
Transport
Application
Physical
Datalink
Network
TCP
Physical
Datalink
Network
Transport
Application

TCP segments processed by a router
Source port Destination port
Checksum Urgent pointer
THL Reserved Flags
Acknowledgment number
Sequence number
Window
Ver IHL ToS Total length
ChecksumTTL Protocol
Flags Frag. Offset
Source IP address
Identiﬁcation
Destination IP address
Payload
Options
THL Reserved Flags
Sequence number
Window
Flags Frag. Offset
Source IP address
Identiﬁcation
Payload
Options
IP
TCP

TCP segments processed by a NAT
THL Reserved Flags
Sequence number
Window
Flags Frag. Offset
Source IP address
Identiﬁcation
Payload
Options
THL Reserved Flags
Sequence number
Window
Flags Frag. Offset
Source IP address
Identiﬁcation
Payload
Options

TCP segments processed by a NAT (2)
•  acKve mode Rp behind a NAT
220 ProFTPD 1.3.3d Server (BELNET FTPD Server) [193.190.67.15]
Rp_login: user `<null>' pass `<null>' host `Rp.belnet.be'
Name (Rp.belnet.be:obo): anonymous
---> USER anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
---> PASS XXXX
---> PORT 192,168,0,7,195,120
200 PORT command successful
---> LIST
150 Opening ASCII mode data connecKon for ﬁle list
lrw-r--r-- 1 Rp Rp 6 Jun 1 2011 pub -> mirror
226 Transfer complete

TCP segments processed by an ALG
running on a NAT
THL Reserved Flags
Sequence number
Window
Flags Frag. Offset
Source IP address
Identiﬁcation
Payload
Options
THL Reserved Flags
Sequence number
Window
Flags Frag. Offset
Source IP address
Identiﬁcation
Payload
Options

© O. Bonaventure, 2011
How transparent is the Internet ?
•  25th September 2010
to 30th April 2011
•  142 access networks
•  24 countries
•  Sent speciﬁc TCP
segments from client
to a server in Japan

Honda, Michio, et al. "Is it s=ll possible to extend TCP?" Proceedings of the 2011 ACM
SIGCOMM conference on Internet measurement conference. ACM, 2011.

End-to-end transparency today
THL Reserved Flags
Sequence number
Window
Flags Frag. Offset
Source IP address
Identification
Payload
Options
THL Reserved Flags
Sequence number
Window
Flags Frag. Offset
Source IP address
Identification
Payload
Options
Middleboxes don't change
the Protocol field, but
some discard packets with a
Protocol field different than
TCP or UDP

TCP ConnecKon establishment
•  Three-way handshake
SYN,seq=1234,OpKons
SYN+ACK,ack=1235,seq=5678,OpKons
ACK,seq=1235,ack=5679

Data transfer
seq=1234,"abcd"
ACK,ack=1238,win=4
seq=1238,"efgh"
ACK,ack=1242,win=0

ConnecKon release
seq=1234,"abcd"
RST

ConnecKon release
seq=1234,"abcd"
ACK,ack=1239
FIN,ack=350
seq=345,"ijkl"
FIN, seq=1238
FIN,seq=349

MulKpath TCP
•  How can we eﬃciently use the mulKple
interfaces that are available on today's hosts?

Design objecKves
•  MulKpath TCP is an evolu=on of TCP

•  Design objecKves
– Support unmodiﬁed applicaKons
– Work over today’s networks (IPv4 and IPv6)
– Works in all networks where regular TCP works

The Mul=path TCP bytestream model
33
Client Server
ABCDEF...111232
0988989 ... XYZZ
IP:1.2.3.4
IP:4.5.6.7
IP:2.3.4.5 IP:6.7.8.9
BCD A

The MulKpath TCP protocol
•  Control plane
– How to manage a MulKpath TCP connecKon that
uses several paths ?
•  Data plane
– How to transport data ?

•  CongesKon control
– How to control congesKon over mulKple paths ?

A naïve MulKpath TCP
SYN+ACK+OpKon
ACK
seq=123, "abc"
seq=126, "def"
SYN+OpKon

A naïve MulKpath TCP
In today's Internet ?
SYN+OpKon
SYN+ACK+OpKon
ACK
seq=123, "abc"
seq=126, "def"
There is no
corresponding
TCP connecKon

Design decision

– A Mul=path TCP connec=on is composed of one or
more regular TCP subflows that are combined

•  Each host maintains state that glues the TCP subflows
that compose a MulKpath TCP connecKon together

•  Each TCP subflow is sent over a single path and appears
like a regular TCP connecKon along this path

MulKpath TCP and the architecture
Physical
Datalink
Network
Transport
Application MulKpath TCP
TCP1
socket
TCP2 TCPn ...
Application
A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, “Architectural guidelines for mulKpath TCP
development", RFC6182 2011.
No modiﬁcaKon
to ease deployment
MulKple subﬂows
to cope with
middleboxes

A regular TCP connecKon
•  What is a regular TCP connecKon ?
– It starts with a three-way handshake
•  SYN segments may contain special opKons
– All data segments are sent in sequence
•  There is no gap in the sequence numbers
– It is terminated by using FIN or RST

MulKpath TCP
SYN+OpKon
SYN+ACK+OpKon
ACK
SYN+OtherOpKon
SYN+ACK+OtherOpKon
ACK

How to combine two TCP subﬂows ?
SYN+OpKon
SYN+ACK+OpKon
ACK
SYN+OtherOpKon
SYN+ACK+OtherOpKon
ACK
How to link with
blue subﬂow ?

TCP 101
IdenKﬁcaKon of a TCP connecKon
Four tuple
– IPsource
– IPdest
– Portsource
– Portdest

All TCP segments
contain the four
tuple
THL Reserved Flags
Sequence number
Window
Flags Frag. Offset
Source IP address
Identiﬁcation
Payload
Options
IP
TCP

How to link TCP subﬂows ?
SYN, Portsrc=1234,Portdst=80+OpKon
SYN+ACK[...]
ACK
SYN, Portsrc=1235,Portdst=80
+OpKon[link Portsrc=1234,Portdst=80]
A NAT could change
addresses and
port numbers

How to link TCP subﬂows ?
+OpKon[Token=5678]
SYN+ACK+OpKon[Token=6543]
ACK
+OpKon[Token=6543]
MyToken=5678
YourToken=6543
MyToken=6543
YourToken=5678

TCP subflows
•  Which subflows can be associated to a
MulKpath TCP connecKon ?

– At least one of the elements of the four-tuple
needs to differ between two subflows
•  Local IP address
•  Remote IP address
•  Local port
•  Remote port

TCP subflows in pracKce
•  MulKpath TCP supports subflow agility
– Client/server can add subflows at any Kme
– Client/server can remove subflows at any Kme

How to transfer data ?
seq=123,"a"
seq=124,"b"
seq=125,"c"
seq=126,"d"
ack=124
ack=126
ack=125
ack=127

How to transfer data
in today's Internet ?
seq=123,"a"
seq=124,"b"
seq=125,"c"
ack=124
ack=126
ack=125
Gap in sequence numbering space
Some DPI will not allow this !

MulKpath TCP Data transfer
•  Two levels of sequence numbers

MulKpath TCP
TCP1
socket
TCP2
MulKpath TCP
TCP1
socket
TCP2
ABCDEF
Data sequence #
TCP1 sequence #
TCP2 sequence #

MulKpath TCP
Data transfer
Dseq=0,seq=123,"a"
DSeq=1, seq=456,"b"
DSeq=2, seq=124,"c"
DAck=1,ack=124
DAck=3, ack=125
DAck=2,ack=457

MulKpath TCP
How to deal with losses ?
•  Data losses over one TCP subﬂow
– Fast retransmit and Kmeout as in regular TCP
Dseq=0,seq=123,"a"
DAck=1,ack=124
Dseq=0,seq=123,"a"
DAck=1,ack=124

MulKpath TCP
•  What happens when a TCP subﬂow fails ?
Dseq=0,seq=123,"a"
DSeq=1, seq=456,"b"
DAck=0,ack=457
Dseq=0,seq=457,"a"
DAck=2,ack=458

Retransmission heurisKcs
•  HeurisKcs used by current Linux implementaKon

–  Fast retransmit is performed on the same subflow
as the original transmission
–  Upon Kmeout expiraKon, reevaluate whether the
segment could be retransmi2ed over another subflow

–  Upon loss of a subflow, all the unacknowledged data
are retransmi2ed on other subflows

Flow control
•  How should the window-based flow control
be performed ?

– Independant windows on each TCP subflow

– A single window that is shared among all TCP
subflows

Independant windows
Dseq=0,seq=123,"a"
DSeq=1, seq=456,"b"
DAck=2,ack=457,win=100
Dseq=2,seq=457,"c"

Independant windows
possible problem
•  Impossible to retransmit, window is already full on green
subﬂow
Dseq=0,seq=123,"a"
DSeq=1, seq=456,"b"

A single window shared by all subﬂows
Dseq=0,seq=123,"a"
DSeq=1, seq=456,"b"
Dseq=2,seq=457,"c"

A single window shared by all subﬂows
Impact of middleboxes
Dseq=0,seq=123,"a"
DSeq=1, seq=456,"b"

MulKpath TCP Windows
•  MulKpath TCP maintains one window per MulKpath
TCP connecKon

–  Window is relaKve to the last acked data (Data Ack)
–  Window is shared among all subflows
•  It's up to the implementaKon to decide how the window is shared
–  Window is transmi2ed inside the window field of the
regular TCP header
–  If middleboxes change window field,
•  use largest window received at MPTCP-level
•  use received window over each subflow to cope with the flow
control imposed by the middlebox

MulKpath TCP buﬀers
MulKpath TCP
TCP1
socket
TCP2
Scheduler
Transmit queues,
process only regular
TCP header
Reorder queue,
processes only
TCP header
MPTCP-level,
resequencing possible
send(...)
recv(...)

Sending MulKpath TCP informaKon
•  How to exchange the MulKpath TCP speciﬁc
informaKon between two hosts ?

•  OpKon 1
–  Use TLVs to encode data and control informaKon
inside payload of subﬂows

•  Op0on 2
–  Use TCP opKons to encode all MulKpath TCP
informaKon
OpKon 1 : Michael Scharf, Thomas-Rolf Banniza , MCTCP: A Mul=path Transport Shim Layer, GLOBECOM 2011

MulKpath TCP with only opKons
•  Advantages
–  Normal way of extending
TCP

–  Should be able to go
through middleboxes or
fallback

•  Drawbacks
–  limited size of the TCP
opKons, notably inside
SYN

–  What happens when
middleboxes drop TCP
opKons in data segments

MulKpath TCP using TLV
•  Advantages
–  MulKpath TCP could
start as regular TCP and
move to MulKpath only
when needed

–  Could be implemented
as a library in userspace
–  TLVs can be easily
extended
•  Drawbacks
–  TCP segments contain
TLVs including the data
and not only the data
•  problem for middleboxes,
DPI, ..

–  Middleboxes become
more diﬃcult

Michael Scharf, Thomas-Rolf Banniza , MCTCP: A Mul=path Transport Shim Layer, GLOBECOM 2011

Is it safe to use TCP opKons ?
•  Known opKon (TS) in Data segments
XD6BHM
Honda, Michio, et al. "Is it sKll possible to extend TCP?." Proceedings of the 2011 ACM

Is it safe to use TCP opKons ?
•  Unknown opKon in Data segments
XD6BHM

MulKpath TCP opKons
•  TCP opKon format

•  IniKal design
– One opKon kind for each purpose
(e.g. Data Sequence number)

•  Final design
– A single variable-length MulKpath TCP opKon
Kind Length OpKon-speciﬁc data

MulKpath TCP opKon
•  A single opKon type
– to minimise the risk of having one opKon
accepted by middleboxes in SYN segments and
rejected in segments carrying data
SubtypeKind Length
Subtype speciﬁc data
(variable length)

Data sequence numbers
and TCP segments
•  How to transport Data sequence numbers ?
– Same soluKon as for TCP
•  Data sequence number in TCP opKon is the Data
sequence number of the ﬁrst byte of the segment
THL Reserved Flags
Sequence number
Window
Payload
Datasequence number

Which middleboxes change
TCP sequence numbers ?
•  Some ﬁrewalls change TCP sequence numbers
in SYN segments to ensure randomness
– ﬁx for old windows95 bug

•  Transparent proxies terminate TCP
connecKons

Middlebox interference
•  Data segments
Data,seq=12,"ab"
Data,seq=14,"cd"
Data,seq=12,"abcd"
Such a middlebox could also be the network adapter of the
server that uses LRO to improve performance.

Segment coalescing

and middleboxes
seq=123,Dseq=0, "a"
seq=456, DSeq=1, "b"
seq=124, DSeq=2,"c" seq=123, DSeq=2, "ac"
copies one opKon
in coalesced
segment
buﬀers small
segments
seq=123, DSeq=0, "ac"

and middleboxes
seq=123,Dseq=0, "ab"
DSeq=0, seq=123,"a"
DSeq=0, seq=124,"b" Middlebox only
understands
regular TCP

A "middlebox" that both splits and
coalesces TCP segments

and middleboxes
•  How to avoid desynchronisaKon between the
bytestream and data sequence numbers ?

•  SoluKon
– MulKpath TCP opKon carries mapping between
Data sequence numbers and (diﬀerence between
ini=al and current) subﬂow sequence numbers
•  mapping covers a part of the bytestream (length)

MulKpath TCP
Data transfer
seq=123,DSS[0->123,len=1],"a"
seq=456, DSS[1->456,len=1],"b"
seq=124, DSS[2->124,len=1],"c"
DAck=1,ack=124
DAck=3, ack=125
DAck=2,ack=457

and middleboxes
seq=123,DSS[0->123,len=1], "a"
seq=456, DSS[1->456, len=1], "b"
seq=124,
DSS[2->124, len=1],"c"
seq=123,
DSS[0->123, len=1],
"ac"
DAck=2,ack=125
DSeq=0,ack=457
seq=125, DSS[2->125, len=1],"c"

and middleboxes
seq=123,DSS[0->123,len=1], "a"
seq=456, DSS[1->456, len=1], "b"
seq=124,
DSS[2->124, len=1],"c"
seq=123,
DSS[2->124, len=1],
"ac"
DAck=0,ack=125
seq=125, DSS[0->125, len=1],"a"
DAck=3,ack=126

MulKpath TCP and middleboxes
•  With the DSS mapping, MulKpath TCP can
cope with middleboxes that
– combine segments
– split segments

•  Are they the most annoying middleboxes for
MulKpath TCP ?
– Unfortunately not

TCP sequence number
and middleboxes

The worst middlebox
•  Is this an academic exercise or reality ?
seq=123,
DSS[1->123, len=2],
"aXXXb"
DAck=3,ack=125
seq=125,
DSS[3->125, len=2],
"cd"
seq=123,
DSS[1->123,len=2],
"ab"
DAck=3,ack=128
seq=128,
DSS[3->125, len=2],
"cd"

The worst middlebox
•  Is unfortunately very old...
– Any ALG for a NAT

220 ProFTPD 1.3.3d Server (BELNET FTPD Server) [193.190.67.15]
Rp_login: user `<null>' pass `<null>' host `Rp.belnet.be'
Name (Rp.belnet.be:obo): anonymous
---> USER anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
---> PASS XXXX
---> PORT 192,168,0,7,195,120
200 PORT command successful
---> LIST
150 Opening ASCII mode data connecKon for ﬁle list
lrw-r--r-- 1 Rp Rp 6 Jun 1 2011 pub -> mirror
226 Transfer complete

Coping with the worst middlebox
•  What should MulKpath TCP do in the
presence of such a worst middlebox ?
– Do nothing and ignore the middlebox
•  but then the bytestream and the applicaKon would be
broken and this problem will be diﬃcult to debug by
network administrators
– Detect the presence of the middlebox
•  and fallback to regular TCP (i.e. use a single path and
nothing fancy)

MulKpath TCP MUST work in all networks where
regular TCP works.

DetecKng the worst middlebox ?
•  How can MulKpath TCP detect a middlebox
that modiﬁes the bytestream and inserts/
removes bytes ?

– Various soluKons were explored

– In the end, MulKpath TCP chose to include its own
checksum to detect inserKon/deleKon of bytes

The worst middlebox
seq=123,
DSS[1->123, len=2,Inv],
"aXXXb"
seq=123,
DSS[1->123,len=2,V],
"ab"
RST, last DSeq=0 RST, last DSeq=0
seq=456, DSS[1->456, len=2,V], "ab"
DAck=3,ack=458

MulKpath TCP
•  What should be the length of the data
sequence numbers ?

– 32 bits
•  compact and compaKble with TCP
•  wrap around problem at highspeed requires PAWS

– 64 bits
•  wrap around is not an issue for most transfers today
•  takes more space inside each segment

MulKpath TCP
•  Data sequence numbers and Data
acknowledgements

– Maintained inside implementaKon as 64 bits ﬁeld

– ImplementaKons can, as an opKmisaKon, only
transmit the lower 32 bits of the data sequence
and acknowledgements

Data Sequence Signal opKon
CumulaKve Data ack
A = Data ACK present
a = Data ACK is 8 octets
M = mapping present
m = DSN is 8
Length of mapping, can extend
beyond this segment
Computed over data covered by
enKre mapping + pseudo header

The MulKpath TCP protocol
•  Control plane
–  How to manage a MulKpath TCP connecKon that uses
several paths ?
•  Data plane
–  How to transport data ?

–  How to control congesKon over mulKple paths ?
–  CongesKon windows on subﬂows MUST be coupled to
ensure that TCP remains fair with regular TCP

AIMD in TCP
•  CongesKon control mechanism
–  Each host maintains a conges=on window (cwnd)
–  No congesKon
•  CongesKon avoidance (addi0ve increase)
–  increase cwnd by one segment every round-trip-Kme
–  CongesKon
•  TCP detects congesKon by detecKng losses
•  Mild congesKon (fast retransmit – mul0plica0ve decrease)
–  cwnd=cwnd/2 and restart congesKon avoidance
•  Severe congesKon (Kmeout)
–  cwnd=1, set slow-start-threshold and restart slow-start

EvoluKon of the congesKon window
Cwnd Fast retransmit
Threshold
Threshold
Slow-start
exponential increase of cwnd
Congestion avoidance
linear increase of cwnd
Fast retransmit
Time

CongesKon control for MulKpath TCP
•  Simple approach
– independant congesKon windows
Threshold
Threshold
Threshold

Independant congesKon windows
•  Problem
12Mbps

Coupled congesKon control
•  CongesKon windows are coupled
– congesKon window growth cannot be faster than
TCP with a single ﬂow
– Coupled congesKon control aims at moving traﬃc
away from congested path

MulKpath TCP use cases
The beast

TCP on servers
•  How to increase server bandwidth ?
•  Load balancing techniques
– packet per packet
– per ﬂow load balancing
•  each TCP connecKon is mapped onto one interface

Increasing server bandwidth
with MulKpath TCP

•  Load balancing with MulKpath TCP
–  CongesKon control eﬃciently uses the two links
for each MPTCP connecKon
–  AutomaKc failover in case of failures

How fast can MulKpath TCP go ?
h2p://linux.slashdot.org/story/13/03/23/0054252/a-50-gbps-connecKon-with-mulKpath-tcp

How fast can MulKpath TCP go ?

Datacenters evolve
•  TraditionalTopologies are tree-
based
–  Poor performance
–  Not fault tolerant
•  Shift towards multipath topologies:
FatTree, BCube,VL2,
Cisco, EC2
…
C. Raiciu, et al. “Improving datacenter performance and robustness with mulKpath TCP,” ACM
SIGCOMM 2011.

Fat Tree Topology [Fares et al., 2008; Clos, 1953]
K=4
1Gbps
1Gbps
AggregaKon
Switches
K Pods with
K Switches
each
Racks of
servers

Fat Tree Topology [Fares et al., 2008; Clos, 1953]
K=4
AggregaKon
Switches
K Pods with
K Switches
each
Racks of
servers
SIGCOMM 2011.

TCP in FAT tree networks
Cost of collissions
SIGCOMM 2011.
0
200
400
600
800
1000
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
Throughput(Mb/s)
Rank of Flow
MPTCP
Optimal Throughput
TCP Flow Throughput

How to get rid of these collisions ?
•  Consider TCP performance as an opKmisaKon
problem

SIGCOMM 2011.
The MulKpath TCP way
Two subflows
differ by their
source port
ECMP balances
the subflows
over different
paths

MPTCP be2er uKlizes the FatTree network
0
200
400
600
800
1000
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
Throughput(Mb/s)
Rank of Flow
MPTCP
Optimal Throughput
TCP Flow Throughput
SIGCOMM 2011.
See also G. Detal, et al. , Revisi=ng Flow-Based Load Balancing: Stateless Path Selec=on in Data Center
Networks, Computer Networks, April 2013 for extensions to ECMP for MPTCP

How many subﬂows does MulKpath
TCP need ?
Total Throughput
0
10
20
30
40
50
60
70
80
90
100
RLB 2 3 4 5 6 7 8
Throughput(%ofoptimal)
Multipath TCP
TCP
SIGCOMM 2011.

Can we improve MulKpath TCP ?
•  Two subﬂows may follow similar paths

Improving ECMP
•  ECMP's hash
–  good load balancing
–  impossible to predict result

•  CFLB
–  replaces hash with
block cipher
–  hosts can select paths
for MulKpath TCP subﬂows
provided they know datacenter topology
G. Detal, Ch. Paasch, S. van der Linden, P. Mérindol, G. Avoine, O. Bonaventure, Revisi=ng Flow-Based Load
Balancing: Stateless Path Selec=on in Data Center Networks, to appear in Computer Networks

MulKpath TCP with CFLB in Fat-Tree
G. Detal, Ch. Paasch, S. van der Linden, P. Mérindol, G. Avoine, O. Bonaventure, Revisi=ng Flow-Based Load
Balancing: Stateless Path Selec=on in Data Center Networks, to appear in Computer Networks

MulKpath TCP on EC2
•  Amazon EC2: infrastructure as a service
–  We can borrow virtual machines by the hour
–  These run in Amazon data centers worldwide
–  We can boot our own kernel
•  A few availability zones have mulKpath topologies
–  2-8 paths available between hosts not on the same
machine or in the same rack
–  Available via ECMP

Amazon EC2 Experiment
•  40 medium CPU instances running MPTCP
•  During 12 hours, we sequenKally ran all-to-all
iperf cycling through:
– TCP
– MPTCP (2 and 4 subﬂows)

MPTCP improves performance on EC2
Same
Rack
0
100
200
300
400
500
600
700
800
900
1000
0 500 1000 1500 2000 2500 3000
Throughput(Mb/s)
Flow Rank
TCP
MPTCP, 4 subflows
MPTCP, 2 subflows
SIGCOMM 2011.

MoKvaKon
•  One device, many IP-enabled interfaces

MPTCP over WiFi/3G
8Mbps, 20ms
2Mbps, 150ms

TCP over WiFi/3G
C. Raiciu, et al. “How hard can it be? designing and implemenKng a deployable mulKpath TCP,” NSDI'12:
Proceedings of the 9th USENIX conference on Networked Systems Design and ImplementaKon, 2012.

MPTCP over WiFi/3G
C. Raiciu, et al. “How hard can it be? designing and implemenKng a deployable mulKpath TCP,” NSDI'12:
Proceedings of the 9th USENIX conference on Networked Systems Design and ImplementaKon, 2012.

MPTCP over WiFi/3G
MulKpath TCP
increases
throughput

MPTCP over WiFi/3G
What happened
here?

Understanding the performance issue
8Mbps, 20ms
2Mbps, 150ms Window
B
A
CD
Window full !
No new data can be sent on WiFi path
A
Reinject segment on fast path
Halve conges0on window on slow subﬂow

Low latency for Siri
•  Long-lived TLS connecKons
WiFi
3G/LTE
Voice samples
Voice samples

High bandwidth on smartphones
•  Koreans want 800+ Mbps on smartphones
WiFi
4G/LTE
Multipath TCP Regular TCP
SOCKS

Hybrid Access Networks
DSL
4G/LTE
Multipath TCP Regular TCP
Hybrid Access
Gateway
TCP
TCP

Issues with the current stack
Physical
Datalink
IPv4/IPv6
TCP
HTTP1.1
ASCII diﬃcult to
parse, no priority
Unsecure
Wait for three way
handshake before
data transfer
Physical
Datalink
IPv4/IPv6
TCP
HTTP/2
TLS
Secure,
But adds more delay
Physical
Datalink
IPv4/IPv6
UDP
QUIC
First bytes
A_er 2 RTTs
First bytes
A_er 3-4 RTTs First bytes
A_er 0 RTT

QUIC in a nutshell
•  First connecKon a2empt
CHLO [SNI, VER]
CHLO[Token, Crypto info]
ServerName and Version
Rejected
REJ[Config, Token, CerKficate]
DATA[Encrypted]
SHLO[Config, Token, CerKficate]
DATA[Encrypted]

QUIC features
– Leverages TCP's long history (CUBIC)
•  Retransmissions
– Be2er than with regular TCP
– Each segment has a diﬀerent seqnum
•  Avoids retransmission ambiguiKes
•  SelecKve acknowledgements
– Cleaner than in TCP

QUIC usage at google
QUIC handshakes fail when RTTs are greater than 2.5 seconds or
when UDP is blocked
Source : J. Iyengar, QUIC Overview, IETF93, July 2015, Prague

QUIC Reducing delays
TCP TCP + TLS QUIC
(equivalent to TCP + TLS)
Source : J. Iyengar, QUIC Overview, IETF93, July 2015, Prague

Why running QUIC over UDP ?
•  Simplest transport protocol
–  Supported correctly by all operaKng systems
–  Supported correctly by all middleboxes

•  ApplicaKon can enKrely control everything
–  Same version of QUIC runs on all pla€orms
–  QUIC can be upgraded as frequently as the applicaKon
–  ApplicaKon developer does not need to coordinate
with IETF or anyone

How to cope with middleboxes ?
•  Very few middleboxes interfere with UDP
– Some middleboxes drop UDP segments
•  ApplicaKons will detect and fallback to TCP
– Some middleboxes rate limit UDP
•  ApplicaKons will detect and fallback to TCP
•  What about middleboxes opKmising QUIC/UDP
– Nightmare for google
– Everything in QUIC (payload and headers) is
encrypted

TFO : A Faster TCP
•  Simple idea : send data in SYN segments
– Modern version of T/TCP
SYN(Src=C,seq=x, HTTP GET)‫‏‬
HTTP GET
SYN+ACK(Dest=C,ack=x+1,seq=y, HTTP Resp)‫‏‬
ACK(Src=A,seq=x)‫‏‬

Internet transport layer
•  SKll lots of innovaKon for an old layer…
–  TCP extensions
•  IniKal window, TCP Fast Open, …
–  MulKpath TCP is ge‚ng deployed
•  RFC6824 was published in January 2013
–  But Middleboxes have ossiﬁed the Internet

•  Other protocols
–  QUIC
•  Pushed by google for web applicaKons
–  TCPINC
•  Support encrypKon inside transport layer
–  TLS 1.3
•  Faster handshake and lower delays

Beyond TCP: The evolution of Internet transport protocols

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Beyond TCP: The evolution of Internet transport protocols

Similar to Beyond TCP: The evolution of Internet transport protocols (20)

More from Olivier Bonaventure

More from Olivier Bonaventure (20)

Recently uploaded

Recently uploaded (20)

Beyond TCP: The evolution of Internet transport protocols