Stevens HFT Conference

1. Architectural Differences in High-Frequency Trading Systems
Jim Wang, CFA
Program in Financial Engineering
Stevens Institute of Technology
Hoboken, NJ, 07030, USA

2. Base Architecture
Stock Exchange
Trading Engine
UDPx5 TCP
GUI
TCP

3. Latency Differences
• Based on Trading Engine Location:
Desktop – Broker – Exchange.
• 10 millisecond or larger: Engine on your
desktop.
• Btw 1-10 millisecond: Engine collocated
with your broker.
• 1 millisecond or less: Engine collocated
with exchange matching engine.

4. Engine on Desktop: Architecture
Exchange Match Engine
Broker Server
Broker GUI API
desktop
internet

5. Engine on Desktop: Performance
• I,090027.015,3007,THC,SEL 200@4.800000G
• S,090027.054,odId=3007 clId=0 pId=1646391774
st=PreSubmitted fil=0 rem=200 afPx=0.000000
lfPx=0.000000 pId=0 whyHeld=
• I,094334.875,3087,RAD,CXL -500@1.100000
• S,094334.914,odId=3087 clId=0 pId=1646391853
st=Cancelled fil=0 rem=500 afPx=0.000000
lfPx=0.000000 pId=0 whyHeld=
• Order = 54-15 = 39ms; Cancel = 914-875 = 39ms

6. Engine on Desktop: Cost
• Market Data Fee: NASDAQ Level I, NYSE Level I
• US Securities & Commodities Non-Professional Bundle
$10
• Free to clients who generate at least USD 30 per month.
• Commission: <=300,000 Shares USD 0.0035.
• Commission: 300,001-3,000,000 Shares USD 0.002.
• Island Exchange Fees: Remove $0.0030; Add ($0.0025).
• ARCA Exchange Fees: Remove $0.0030; Add ($0.0021)

7. Broker Collocation: Architecture
Exchange Match Engine
Broker Server
GUI
desktop
internet
API
Broker
Data
Center

8. Broker Collocation: Performance
• D36262260,OCCG001A 20100623100422CCG001A
00034927FMN T0000010000000000999992 14.5600
0.0000YISLD YN00000NNN 0.000000000INET,1437,1438,0,0
• <36262265,PCCG001A 13197704 3492700000013197704
FMN T00000100 14.5600100422ISLDINET
• Order = 65-60 = 5ms;
• D36262698,XCCG001A 20100623100422CCG001A
13188804,AFE,2336,2368,0,0
• <36262702,PCCG001A 13188804 3482700000013188804
AFE B00000000 23.1300100412ISLDINET
• Cancel = 702-698 = 4ms

9. Broker Collocation: Cost
• Collocation Fee: > $200 per month
• Market Data Fee: Level I and II, Professional > $600 per
month
• Commission: USD 0.0005 – 0.001 based on volume
• Island Exchange Fees: Remove $0.0030; Add ($0.0025).
• ARCA Exchange Fees: Remove $0.0030; Add ($0.0021)

10. Exchange Collocation: Architecture
Exchange Match Engine
Broker Check
GUIdesktop
internet
API
Exchange
Data
Center
Sponsored Access Naked Access

11. Source of Latency
• Propagation latency: speed of light 5us/km,
Mahwah – Weehawken 40km.
• Transmission latency: high speed communication
link throughput rate 1-10 Gbps. 1us/1kb to
serialize and transport.
• Processing latency. dedicated CPU for critical
threads, kernel bypass, hardware acceleration.

12. Propagation Latency
• Collocation within 200m 1us.
• Dark Fiber: Optical fiber infrastructure that
is currently in place, but is not being used
(dark).
• End-to-end direct connectivity, WAN
elimination.
• Dedicated use of a wavelength (not the
cable), Wavelength-division Multiplexing.

13. Transmission Latency
• Cut-through switching: the switch starts
forwarding a frame before the whole frame
has been received.
• Kernel Bypass: application communicate
with NIC using special supplied lib without
using standard system call.
• RDMA (remote direct memory access):
one computer place information directory
into the memory of another.

14. Context Switching
Step 0 Step 1 Step 2 Step 3 Step 4
Read (user context)
Sys Call Kernel Context
Write (user context)
Sys Call Kernel Context

15. Processing Latency
• Parallel Problem: With 10k symbols, 6
major exchanges, relatively independent,
tasks be streamlined.
• Through Software optimization:
flexibility, take advantage of general
purpose CPU improvements over time.
• Through Hardware Acceleration:
specialized hardware, improve consistency
by reducing jitter.

16. Software Optimization
• Separation of high speed vs high
complexity. Latency sensitive task in
critical path, computation intensive tasks
offload to separate thread/process.
• memory caching: critical decision thread
pined to a dedicated CPU.
• inline vs function calls: C, C++, Java.

17. Hardware Acceleration
• Parallel processors (general purpose multi-
core CPU, tile array architecture, cell
broadband engine)
• FPGA (field programmable gate array,
specialized communication)
• GPGPU (general purpose computation on
graphic processing units for computational
intensive tasks)