1. Zero-Copy Event-Driven
Servers with Netty
Daniel Bimschas
Institute of Telematics, University of Lübeck
http://www.itm.uni-luebeck.de
1

2. Content
• Basics
– Zero-Copy Techniques
– Event-Driven Architectures
• Introduction to Netty
– Buffers
– Codecs
– Pipelines & Handlers
• Netty Examples
– Discard
– HTTP Server
2

3. Zero-Copy Techniques
3

4. Traditional Approach
• copying and context switches between kernel
and user space à poor performance!
Socket
Read Buffer NIC Buffer
Fast Buffer
Slow Kernel Context
Application Context
Application
App Buffer
Slide Source: Distributed Systems Course 2011/2012 by Dennis Pfisterer, Institute of Telematics, University of Lübeck, Germany 4

5. Zero-Copy Approach
• Kernel handles the copy process via Direct Memory Access (DMA)
– No CPU load
– Lower load on bus system
– No copying between kernelspace and userspace
Socket
Read Buffer NIC Buffer
Fast Buffer
Kernel Context
Perfect!
Task Application Context
Application
App Buffer
Slide Source: Distributed Systems Course 2011/2012 by Dennis Pfisterer, Institute of Telematics, University of Lübeck, Germany 5

6. Simple Benchmark: Copy vs. Zero-Copy
Duration [ms]
Data [Mbyte]
Slide Source: Distributed Systems Course 2011/2012 by Dennis Pfisterer, Institute of Telematics, University of Lübeck, Germany 6

7. Zero-Copy Between Communication Layers
• Often copying is not necessary
– If data is not modi ed a slice can be passed
forward without copying to a different buffer
Ethernet IP TCP HTTP XML
Application
Ethernet IP TCP HTTP XML
Transport Ethernet IP TCP HTTP XML
Internet Ethernet IP TCP HTTP XML
Link Layer Ethernet IP TCP HTTP XML
Slide Source: Distributed Systems Course 2011/2012 by Dennis Pfisterer, Institute of Telematics, University of Lübeck, Germany 7

8. Zero-Copy Between Communication Layers
• Sometimes slices of multiple packages can be
combined to extract e.g., a payload that is split
over multiple packages
• Newly “created” buffer points to original buffers
à No copying necessary
Virtual
HTTP (Part 1) HTTP (Part 2)
Buffer
Received TCP HTTP (Part 1) TCP HTTP (Part 2)
Buffers
8

9. Event-Driven Architecture in Networking
9

10. Request Processing in Multi-Thread Servers
t1: Thread S1: ServerSocket Waits most of time db1: DataBase
socket = accept()
s2: Socket without doing
<<create>>(socket)
t2: Thread actual work!
run()
socket = accept()
waitForData()
bytes = read() <<create>>
d1: Decoder
decode(bytes)
waitForData()
bytes = read()
req = decode(bytes)
<<create>>
s1: Servlet
processRequest(req)
query(...)
response results
write(response)
= thread idle
10

11. Request Processing in Multi-Thread Servers
• Usually one thread per request
– Thread idle most of the time (e.g. waiting for I/O)
– Thread even more idle when network slow
– Number of simultaneous clients mostly limited by
maximum number of threads
• Thread construction is expensive
– Higher latency when constructing on request
– Can be improved using pools of Threads
(see Java‘s ExecutorService & Executors classes)
11

12. Request Processing in Event-Driven Servers
s1: Socket s2: Socket io1: NioWorker e1: ExecutorThread = request 1
dataAvailable() = request 2
bytes = read() handleEvent(s1, bytes)
<<create>>
dataAvailable() d1: Decoder
decode(bytes)
bytes = read() handleEvent(s2, bytes)
<<create>>
dataAvailable() d2: Decoder
decode(bytes)
bytes = read() handleEvent(s1, bytes)
req = decode(bytes)
resp = processRequest(bytes)
write(resp)
dataAvailable()
bytes = read() handleEvent(s2, bytes)
req = decode(bytes)
resp = processRequest(bytes)
write(resp)
Disclaimer: this slide may contain errors and is far away from real implementation code but should do good for illustrative purposes 12

13. Request Processing in Event-Driven Servers
• Calls to I/O functions of OS are non-blocking
• Heavy usage of zero-copy strategies
• Everything is an event
– Data available for reading
– Writing data
– Connection established / shut down
• Different requests share threads
• Work is split into small tasks
– Tasks are solved by worker threads from a pool
– Thread number and control decoupled from individual
connections / simultaneous requests
• Number of simultaneous clients can be very high
– Netty: 50.000 on commodity hardware!
13

14. Introduction to Netty
14

15. Introduction to Netty
• „The Netty project is an effort to provide an asynchronous
event-driven network application framework for rapid
development of maintainable high-performance protocol
servers & clients.“
Source: http://netty.io
• Good reasons to use Netty:
• Simpli es development
• Amazing performance
• Amazing documentation (Tutorials, JavaDocs), clean concepts
• Lots of documenting examples
• Unit testability for protocols
• Heavily used at e.g., twitter for performance critical applications
15

16. Netty – Feature Overview
16

17. Introduction to Netty - Buffers
• Netty uses a zero-copy strategy for efficiency
• Primitive byte[] are wrapped in a ChannelBuffer
• Simple read/write operations, e.g.:
– writeByte()
– writeLong()
– readByte()
– readLong()
– …
• Hides complexities such as byte order
• Uses low overhead index pointers for realization:
17

18. Introduction to Netty - Buffers
• Combine & slice ChannelBuffers without copying
any payload data by e.g.,
– ChannelBuffer.slice(int index, int length)
– ChannelBuffers.wrappedBuffer(ChannelBuffer... Buffers)
• Pseudo-Code Example:
requestPart1 = buffer1.slice(OFFSET_PAYLOAD,
buffer1.readableBytes() – OFFSET_PAYLOAD);
requestPart2 = buffer2.slice(OFFSET_PAYLOAD,
buffer2.readableBytes() – OFFSET_PAYLOAD);
request = ChannelBuffers.wrappedBuffer(requestPart1, requestPart2);
Virtual
HTTP (Part 1) HTTP (Part 2)
Buffer
Received TCP HTTP (Part 1) TCP HTTP (Part 2)
Buffers
18

19. Netty – Feature Overview
19

20. Introduction to Netty - Codes
• Many protocol encoders/decoders included
– Base64
– Zlib
– Framing/Deframing
– HTTP
– WebSockets
– Google Protocol Buffers
– Real-Time Streaming Protocol (RTSP)
– Java Object Serialization
– String
– (SSL/TLS)
20

21. Introduction to Netty - Codecs
• Abstract base classes for easy implementation
– OneToOneEncoder
– OneToOneDecoder
• Converts one Object (e.g. a ChannelBuffer) into another (e.g.
a HttpServletRequest)
– ReplayingDecoder
• The bytes necessary to decode an Object (e.g. a
HttpServletRequest) may be split over multiple data events
• Manual implementation forces to check and accumulate data
all the time
• ReplayingDecoder allows you to implement decoding
methods just like all required bytes were already received
21

22. Netty – Putting it all together
22

23. Introduction to Netty – Pipelines & Handlers
• Every socket is attached
to a ChannelPipeline
• It contains a stack of
handlers
– Protocol
Encoders / Decoders
– Security Layers
(SSL/TLS, Authentication)
– Application Logic
– ...
23

24. Introduction to Netty – Pipelines & Handlers
• One ChannelPipeline per
Connection
• Handlers can handle
– Downstream events
– Upstream events
– Or both
• Everything is an event
– Data available for reading
– Writing data
– Connection established /
shut down
– …
24

25. Netty – ChannelPipeline Example: HTTP(S) Client
Client Application • Applications based
read(httpResponse) write(httpRequest) on Netty are built as
Channel a stack
httpResponse httpRequest
• Application Logic
ChannelPipeline
HttpRequestDecoder HttpRequestEncoder
String String
sites on top of the
StringDecoder StringEncoder channel
ChannelBuffer ChannelBuffer
• Everything else
SSLDecoder SSLEncoder
(decoding,
ChannelBuffer ChannelBuffer
securing, ...) is done
OS Socket object
inside the pipeline
Disclaimer: this slide is imprecise, may contain errors and there’s no one-to-one implementation. It shows a logic conceptual view of the Netty pipeline. 25

26. Netty Examples
DISCARD Server
26

27. Netty Examples – DISCARD Server
http://tools.ietf.org/html/rfc863
Source: Netty project source code @ http://github.com/netty/netty
27

28. Netty Examples – DISCARD Server Bootstrap
Source: Netty project source code @ http://github.com/netty/netty
28

29. Netty Examples – DISCARD Server
Source: Netty project source code @ http://github.com/netty/netty
29

30. Netty Examples – Echo Server
Source: Netty project source code @ http://github.com/netty/netty
30

31. Netty Examples – Static HTTP File Server
Source: Netty project source code @ http://github.com/netty/netty
31

32. Netty Examples – Static HTTP File Server
Source: Netty project source code @ http://github.com/netty/netty
32

33. Netty Examples – Static HTTP File Server
...
Source: Netty project source code @ http://github.com/netty/netty
33

34. References
• Netty
– Project: http://netty.io
– Tutorial: http://netty.io/docs/
– JavaDoc: http://netty.io/docs/3.2.6.Final/api/
– Sources: http://netty.io/docs/3.2.6.Final/xref/
– Development: https://github.com/netty/netty
34