The document discusses using RDMA (Remote Direct Memory Access) for high performance networking in Python. It describes how traditional sockets involve multiple memory copies that reduce performance. RDMA allows data to be directly placed in the receiving application's memory without intermediate copies. The document outlines an implementation of RDMA sockets in Python, called RdmaSocket, that exposes an interface similar to regular sockets but uses RDMA under the hood. Examples are given showing how it can simplify high performance networking applications in Python.

1. import rdma: zero-copy networking with RDMA and Python Andy Grover @groveronline http://groveronline.com http://blogs.oracle.com/linuxnstuff

3. Python and performance

4. Issues I ran into

5. (Questions anytime.)

7. server: recvfrom() -> (“get data.tgz”)

8. server: data = open(“data.tgz”).read()

9. server: sock.sendto(client, “OK “ + data)

10. client: recvfrom() -> data

12. 2 on S, 2 on C

13. S: read from user buffer, write to kernel buffer by OS

14. S: read from kernel buffer by HW

15. C: write to kernel buffer by HW

16. C: read from kernel buffer, write to user buffer (OS)

18. Extra copy on each side consumes CPU

19. Also consumes 3x RAM bandwidth!

20. What do we do???

21. Direct Data Placement (RDMA)

23. Sender tells HW data buffer and key, Target HW uses key to place received data directly in user buffer

24. Done!

25. Way complicated

27. Supports RDMA

28. RHEL 5.4+ supports natively

30. Full disclosure: my day job

31. Guaranteed delivery of datagrams

32. Allows RDMA ops via sendmsg() and CMSGs

33. Hides complexity of IB Verbs

34. Still pretty complex!

36. Learning opportunity

37. Can it be done?

38. Pythonically?

40. Interpreted, but fits in per-CPU cache

41. Many CPU cores these days

42. Shared RAM

43. Cache misses on the data

44. Implementation Issues.

46. Buffers shared behind the scenes

48. sliceable etc.

52. Solution: external library, python-eunuchs

53. Native support RSN

55. Solution: forget inheritance, just implement socket object methods

57. ctypes used heavily

58. C Module solely to return an address

60. client: cookie = sock.get_mr(m)

61. client: sock.sendto(server. “get data.tgz, my cookie is <cookieval>”)

62. server: recvmsg() -> (“get data.tgz, my cookie is <cookieval>”

63. server: m = mmap(“data.tgz”)

64. server: sock.rdma_sendmsg(client, m, cookieval, length, token, “OK”)

65. client: recvmsg() -> “OK”

67. Copied “OK” instead of “OK”+8K, CPU and cache win

68. It worked!

70. Dogfood it -- simplify RDS utility apps

71. RDMA loves async: go Twisted

73. Thanks! http://github.com/agrover/python-rds