Direct network I/O allows network controllers (NICs) to expose multiple instances of themselves, to be used by untrusted software without a trusted intermediary. Direct I/O thus frees researchers from legacy software, fueling studies that innovate in multitenant setups. Such studies, however, overwhelmingly ignore one serious problem: direct memory accesses (DMAs) of NICs disallow page faults, forcing systems to either pin entire address spaces to physical memory and thereby hinder memory utilization, or resort to APIs that pin/unpin memory buffers before/after they are DMAed, which complicates the programming model and hampers performance. We solve this problem by designing and implementing page fault support for InfiniBand and Ethernet NICs. A main challenge we tackle—unique to NICs—is handling receive DMAs that trigger page faults, leaving the NIC without memory to store the incoming data. We demonstrate that our solution provides all the benefits associated with “regular” virtual memory, notably (1) a simpler programming model that rids users from the need to pin, and (2) the ability to employ all the canonical memory optimizations, such as memory overcommitment and demand-paging based on actual use. We show that, as a result, benchmark performance improves by up to 1.9x. These are slides from a highlight talk at Compiler, Architecture and Tools Conference (CATC) 2017, For more information, see our ASPLOS'17 paper at https://dl.acm.org/citation.cfm?id=3037697.3037710

1. Mellanox Technologies
Technion - Israel Institute of Technology
Compiler, Architecture and Tools Conference (CATC)
Dec 4, 2017, Haifa, Israel
Ilya Lesokhin, Haggai Eran, Shachar Raindel, Guy Shapiro,
Liran Liss, Sagi Grimberg, Muli Ben-Yehuda, Nadav Amit, Dan Tsafrir
1

2. Direct IO
 Provide applications / Guest OS direct
access to HW
 Examples
 SRIOV and PCIe pass-through
 RDMA: Remote Direct Memory Access
 User-level packet processing (DPDK)
 Eliminate overheads:
 Context switch / Hypercall
 Memory copies
 Customization: users can make their
own tradeoffs
 Latency vs. throughput
 CPU vs memory
 Custom API
2*Peter et. al., Arrakis: The Operating System is the Control Plane, OSDI’14
*

3. IOuser and IOprovider
HW
HypervisorOS
VM
OS
Driver
APP
User
DriverIOuser
IOprovider
3

4. Address translation
 Direct I/O requires
address translation
 Provide the same
isolation needed in CPU
 Most devices do not
tolerate page faults
4
IOuser
Page faults are
allowed
Page faults are
not allowed

5. DirectIO mandates pinning
Static pinning:
 Hypervisors pin guest memory for PCI pass-through
 RDMA registration pins memory
 DPDK buffers are pinned
 Benefits:
 Easier to implement
 Predictable IOuser performance
5

6. The price of pinning
 No canonical memory optimizations, achieved using
virtual memory and page faults:
Demand paging Over
commitment
Page migration
Delayed allocation Swapping NUMA migration
Mmap-ed files Deduplication Compaction
Calloc with zero
page
Copy on write Transparent huge
pages
Increase startup
time
Increase
memory usage
Decrease
performance
6

7. Partial solutions
 Fine-grained pinning: pin before use, release after
 Overhead voids benefits of DirectIO
 Course-grained pinning: pin-down cache
 Common in high-performance RDMA setups
 Cons:
 Complicates programming model
 Untrusted IOuser makes policy decisions
 Copying data to/from a static pinned buffer
 Time wasted on copying
7

8. We need page faults!
 Allow devices to access non-pinned buffers
 Similarly to how page faults enable CPU virtual-memory
 Any valid IOuser buffer could be used
Have the cake and eat it:
 Provide the benefits of DirectIO
 Allow virtual memory based optimizations
 Keep memory management policy in the IOprovider
8

9. IO paging today
 State of the art IO page faults
 PCIe ATS/PRI (Page Request Interface)
 Intel VT-d SVM (Shared Virtual Memory)
 AMD PPR (Peripheral Page Request)
 NVIDIA Unified Memory
 IBM CAPI
 Target accelerators and GPUs:
 Execution is suspended until page fault is
resolved
 Blocking a thread
 Suspending a GPU kernel
 Networking is different…
9
Not widely deployed yet –
we used the NIC’s internal
IOMMU
Resolve
page
fault

10. The receive page fault challenge
 Incoming network data cannot be stopped!
 Buffer?
 Expensive and rarely used (e.g. 100 Gbps x 10 ms = 125 MB)
 Back-pressure the network?
 Adversely affect unrelated flows
 Drop?
 Retransmission timeout reduces performance
10
NIC Memory

11. RDMA receive page faults
 Network and transport
properties
 Lossless network
 Typically large retransmission
timeouts
 Provides reliable point-to-
point connections in HW

12. Solution: pause sender
 Page fault is detected on the
device, which implements
the transport
 We modified the
Connect-IB firmware to:
 Tell sender to pause
 Ask driver to resolve page
fault
 Drop subsequent packets
NPF
Dropped
traffic
RNR
delay
2
3
1
4
5
6
4
3
5
6
Initiator Responder
12

13. Ethernet receive page faults
 Network properties
 Lossy network
 Typically SW transport, at
IOuser
 Transport layer cannot
intervene during page fault
 Just dropping packets and
using TCP retransmission
doesn’t cut it!
13

14. Solution: backup ring
 Direct-IO performance
as long as there are no faults
 Fallback due to page
faults, similar to para-
virtualization:
 Packets are stored in a
pinned ring on the
IOprovider
 Copied to the IOuser,
after the PF is resolved
IOuser
IOprovider
NIC pf
14

15. 15

16. Evaluation
 Implementation:
 Page fault support
for RDMA in
Connect-IB
firmware
 Emulated Ethernet
backup-ring by
duplicating packets
to both rings
16

17. Periodic IOPF
Major/min
or doesn’t
matter
90% efficiency at
2-15 page fault
frequency
17

18.  Standard iSER initiator
(iSCSI Extensions for RDMA)
 Stock Linux kernel
 fio benchmark
 Random 64KB reads
 Modified iSER target
 Based on open-source
tgt target implementation
 Staging buffer: 1GB of 512 KB
chunks.
Storage evaluation
18
TGT target
Staging buffer
Linux
Page Cache
Initiator Initiator Initiator…
Compete
on physical
memory

19. Storage evaluation
19

20. HPC evaluation
 Added MPI page fault support
 Implemented in the MXM communication library
 Obsoleted ~10K’s of LOC of pin-down cache
 Application benchmarks
 IMB (Intel MPI benchmark) application suite
 B_eff (effective bandwidth) benchmark
20

21. HPC Evaluation
Intel MPI Benchmarks
B_eff effective bandwidth (MB/sec)
21

22. Conclusions
 With network page-
faults you can have:
 Direct IO performance
 Virtual memory
canonical optimizations
 Simplified
programming model
 Receive page faults pose
a unique challenge
 Future work
 Better integration of IO
page faults
 Dirty and access bits
 Eviction policy
22
Questions?