How does the latency of DDR4 DRAM compare to Intel Optane Persistent Memory when used in both App Direct and Memory Modes for In-Memory database access? This talk is about the latency benchmarks that I performed by adding gettimeofday() calls around critical DB kernel operations. This talk covers the technology, cache hit ratios, lots of histograms and lessons learned.

1. Brought to you by
Optane PMem latency
for disk and in-memory DB
Doug Hood
Consulting Member of Technical Staff at

2. Latency of Optane PMem for databases
Can Intel Optane Persistent Memory improve database Latency?
■ Buffer cache reads
■ Log writes
■ Commits
Can Intel Optane Persistent Memory improve In-Memory database latency?
■ Reads and writes
■ Log writes
■ Commits
Should you use App Direct Mode or Memory Mode?

3. App Direct Mode, Memory Mode or Both?
App Direct Mode
■ Byte addressable memory
■ Persistent
■ Need to explicitly design/code for it
■ Need to create file systems for it
Memory Mode
■ Byte addressable memory
■ Non persistent
■ Treat it like DRAM
■ It just works
DRAM Optane Persistent Memory
App Direct Mode Memory Mode
Explicitly use DRAM & PMem as
needed based on your design
Just use ‘memory’
The ‘memory’ may be DRAM or PMem
depending on the working set size

4. App Direct Mode for disk based databases
Buffer Cache Reads
■ Design an automatic read hierarchy
■ Use App Direct Mode for the hotest data
■ Use NVMe storage for hot data
■ Use SSDs for the coldest data
■ Resulted in 10x faster reads
■ Enables consistent reads in 19 microseconds
Log Writes and Commits
■ 8 time faster than using the fastest NVMe

5. Optane PMem for in-memory databases
Intel Optane persistent memory is good, but it is not magic:
■ It has finite capacity (max 512 GB per module)
■ Only works with the latest Intel Xeon CPUs
■ Slower than DDR4 RAM
■ You still have to make hard design choices
■ Cost effective != free

6. App Direct Mode for in-memory databases
Pro
Instant DB load for multi TB database is nice
Sync writes to logs are fast & simple
Con
PMem is up to 3x slower than DDR4 RAM
Designing DDR4 caches for PMem = big redesign

7. Memory Mode for in-memory databases
Hot
Data
Cold
Data
Intel memory
controller
It just works!

8. How much PMem is really being used?
■ The latency of 100% PMem can be up to 3x slower than DRAM
■ Memory Mode always has a DRAM cache for PMem, but the size will vary
■ Vary the working set size to see the effect on performance
■ Need to check the CPU hardware events to verify what is actually
happening
DRAM = 384 GB
PMem = 1.5 TB
1:4 ratio of DRAM to
PMem
DRAM Optane Persistent Memory

9. Vary the working set size
Uniform with bounds
using drand48()
Zipfian various skew Poisson various skew

10. Tools used for these benchmarks

11. Memory Mode Latency for an IMDB
L3 Cache [0.1 us]
DDR4 RAM [1.07 us] Optane PMem [2.1 us]
Throughput
(TPS)

12. Analysis of Results
■ Working set within DRAM, ie < 384 GB
■ The Memory Mode read results were the same as the DRAM results
■ The Memory Mode write results were within 5% of the DRAM results
■ Working set within PMem, ie 25% - 100% of PMem
■ The DRAM cache hit ratio only slightly improved the performance
■ This is not what I expected, but a good outcome
■ The read scalability was not affected by the PMem
■ The write scalability was minimally affected by PMem

13. Workload Schema
create table vpn_users (
vpn_id integer not null,
vpn_nb integer not null,
directory_nb char(10) not null,
last_calling_party char(10) not null,
desc char(100) not null,
myjson char(8300),
primary key (vpn_id, vpn_nb)
);
Insert 1.4 TB of data [160M rows]

14. SQL Workload
select
directory_nb,
last_calling_party,
descr
from vpn_users
where vpn_id = ?
and vpn_nb= ?;
update vpn_users
set last_calling_party = ?
where vpn_id = ?
and vpn_nb = ?;
commit;
Lookup by PK Update by PK

15. Tools used for these benchmarks
■ gettimeofday()
■ Intel icc C++ complier [+ gcc]
■ Intel VTune Profiler
■ Intel Microarchitecture Code Named Cascade Lake-X Events
■ ipmctl https://github.com/intel/ipmctl
■ https://man7.org/linux/man-pages/man2/perf_event_open.2.html
■ Process Counter Monitor https://github.com/opcm/pcm
■ Prometheus & Grafana
■ A database with buffer cache (Oracle Exadata X8M)
■ An In-Memory database (Oracle TimesTen 18.1)
■ tptbm.c https://github.com/oracle/oracle-timesten-samples

16. Brought to you by
Doug Hood
douglas.hood@oracle.com
@ScalableDBDoug