Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/25wa194. Maciej Maciejewski talks about the primary differences between persistent memory, storage devices, and regular DRAM. He presents how Persistent Memory is exposed to the OS with ACPI 6.0 extensions, and describes the resulting changes made upstream to the Linux kernel to provide direct access. He also describes an open source library, known as the NVM Library. Filmed at qconlondon.com. Maciej Maciejewski works at Intel Technology Poland, where he is dealing with software solutions focusing on the next-gen non-volatile memory. He has worked at ADVA Optical Networking as a Senior Software Manager for 8 years. He has lead architectural work on the EU project ONE, with focus on next generation converged IP-optical network management systems.

1. Maciej Maciejewski

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
persistent-memory-dram

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. HARDWARE
2

5. Persistent memory
3

6. How does it differ from DRAM
4

7. How fast is it?
5

8. OPERATINGSYSTEM
6

9. ACPI NFIT
7

10. E820
8

11. Whatcanwedowithit?
9

12. 10
Persistent
Memory
User
Space
Kernel
Space
Application
Load/
Store
MMU
Mapp
ings
NVDIMM Driver
temporary file
Standard
File API
PM-aware
File System
NVML

13. http://pmem.io
https://github.com/pmem/nvml/
NVML
11

14. Memory
12

15. Memory - libvmmalloc
13
libvmem
jemalloc
Application
PM
Kernel
Space
User
Space
malloc
vmmalloc
mmap
DRAM
vmem_pool_mallocvmem_pool_create
constructor
temporary file

16. Memory - libvmem
14
libvmem
jemalloc (3.6.0)
Application
Persistent Memory
Kernel
Space
User
Space
libc
(or other)
vmem_malloc
vmem
fallocate/mmap
malloc
DRAM
mmap/sbrk
vmem_create
vmem_pool_malloc
malloc
vmem_pool_create

17. Memory - memkind
15

18. Block storage
16

19. 17

20. Byte persistency
18

21. Why persistency is an issue?
19
CPU
L1
L2
L3
iMC
NVDIMM
CPU
L1
L2
iMC
NVDIMM
CLFLUSH
CLFLUSHOPT
CLWB
SFENCE
PCOMMIT
SFENCE

22. 20
Flushing to Persistence
open(…);
mmap(…);
strcpy(pmem, „Hello");
msync(pmem, 6, MS_SYNC);
pmem_persist(pmem, 6);
strcpy(pmem, „Hello, World!");
pmem_persist(pmem, 14);
Crossing the 8-Byte Store
Result?
1. „0000000000...”
2. „Hello, W000000...”
3. „00000000orld!0”
4. „Hello, 00000000”
5. „Hello, World!0”
crash

23. nvml Persistent Libraries
libpmem – Basic persistency handling
libpmemblk – Block access to pmem
libpmemlog - Log file on pmem (append-mostly)
libpmemobj - Transactional Object Store on pmem
21

24. libpmemobj
Object API
Transactional API
22

25. Applications
23

26. pm_invaders
24

27. 25

28. Redis key/value store
26

29. 31
0
5
10
15
20
25
30
32 64 128 256 512 1024 2048 4096 8192
seconds
object size
Startup time
RDB
AOF
PM

30. 32
0
1000
2000
3000
4000
5000
6000
7000
8000
32 64 128 256 512 1024 2048 4096 8192
DRAMallocations[MB]
objects size
DRAM usage
AOF
RDB
PM

31. 33
0
20000
40000
60000
80000
100000
120000
36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141
operations/sec
% of OS memory
Running out of DRAM
No Persist

32. 34
0
20000
40000
60000
80000
100000
120000
36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141
operations/sec
% of OS memory
Running out of DRAM
No Persist
RDB

33. 35
0
20000
40000
60000
80000
100000
120000
36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141
operations/sec
% of OS memory
Running out of DRAM
No Persist
RDB
AOF

34. 36
0
20000
40000
60000
80000
100000
120000
36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141
operations/sec
% of OS memory
Running out of DRAM
No Persist
RDB
PM
AOF

35. 37
0
10000
20000
30000
40000
50000
60000
70000
80000
32 64 128 256 512 1024 2048 4096 8192
operations/sec
objects size
CPU usage
PM 1X
AOF 1X

36. 38
0
10000
20000
30000
40000
50000
60000
70000
80000
32 64 128 256 512 1024 2048 4096 8192
operations/sec
objects size
CPU usage
PM 1X
PM 4X
AOF 1X
AOF 4X

37. 39
0
10000
20000
30000
40000
50000
60000
70000
80000
32 64 128 256 512 1024 2048 4096 8192
operations/sec
objects size
CPU usage
PM 1X
PM 4X
PM 6X
PM 10X
AOF 1X
AOF 4X
AOF 10X

38. Moving even more data
0
100
200
300
400
500
600
Time [s] Memory [MB]
GMRES Sparse Matrix solver
Original Matrix data in PM All in PM
40

39. Modifying app engine
for (int i=0; i<NUMBER_OF_ITERATIONS; i++) {
result = calculateThis(i, result);
}
41
i_pm = 0; //at first runtime of app only
...
...
for (i_pm; i_pm<NUMBER_OF_ITERATIONS; i_pm++) {
TX_BEGIN(pool) {
result_pm = calculateThis(i_pm, result_pm);
} TX_END
}
i_pm = 0;

40. Possibilities
Choices
Even more fun ...
42

42. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/persistent-
memory-dram