16. Another Alternative
• Requires ultra high-speed/low-
latency network like Fiber-Channel
or Infiniband
• Latency is still at ~ 100microsec
• Expensive, complex and not
supported by cloud vendorsNew non-empty Instance
Redundant
Flash Array
Failed Instance
18. Redis on Flash (RoF)
• Flash used as a RAM extender (not as a persistent storage)
• In RAM: dict & keys & hot values
• On Flash: cold values
• Values are either in RAM or on Flash
• Multi-shard; multi-threaded; asynchronous
• Pluggable storage engine (RocksDB and IBM’s ark)
• 100% compatible with the OSS Redis
24. Customer Perspective
• Keep the speed of Redis
• Maintain all Redis commands - simple and complex
• Flexible configuration across regions
• Any # of regions, single/multi-az/rack, any combination of nodes/shards
• Fully-managed service
30. Our Approach to Master-Master
• Behaves like local Redis, performs like local Redis
• Based on CRDT (Conflict-free Replication Data Type)
• Supports Strong Eventual Consistency
• No consensus protocol; no need for R + W > N
• Any two nodes that have received the same (unordered) set of updates will be
in the same state
• More to come…
36. • Dealing with simple problems
• Writing code in shitty languages
• Using silly npm packages
• Not refactoring your poor old architecture because of politics
You are tired of…
45. • A marketplace for OSS and closed source modules
• All modules are certified by Redis Labs for these platforms:
Module Hub
1M+ instances 350K+ instances 100+ F500 Companies
www.redismodules.com
49. Adding a Module to Module Hub
Upload
Certification
Process
One monthA few clicks
50. Adding a Module to Module Hub
Upload Alpha
Certification
Process
One monthA few clicks Still not shown in
the marketplace.
Test by 10 selected
customers
51. Adding a Module to Module Hub
Upload Alpha Beta
Certification
Process
One month At least 1
month free
A few clicks Still not shown in
the marketplace.
Test by 10 selected
customers
52. Adding a Module to Module Hub
Upload Alpha Beta GACertification
Process
One month Still not shown in
the marketplace.
Test by 10 selected
customers
At least 1
month free
A few clicks
61. A new era of hardware (and what it might mean
for Redis)
A new era of data structures?
Talk outline
a
b
62. Human
computers
at Dryden by NACA (NASA) -
Dryden Flight Research Center
Photo Collection
http://www.dfrc.nasa.gov/Gallery/Photo/Places/HT
ML/E49-54.html. Licensed under Public Domain
via Commons -
https://commons.wikimedia.org/wiki/File:Human_c
omputers_-
_Dryden.jpg#/media/File:Human_computers_-
_Dryden.jpg
63. Computing on a human scale
63
10ns
70ns
10ms
10s
1:10s
116d
Registers
& L1-L3
File on
desk
Main
memory
Office filing
cabinet
HDDTrip to the
warehouse
65. 2-10m
Computing on a human scale
65
10s
1:10s
116d
File on
desk
Office filing
cabinet
Trip to the
warehouse
4x capacity
fireproof local
filing cabinets
23-40m
Phone
another office
(RDMA)
3h20m
Next-gen
warehouse
66. The new ~numbers everyone should know
66
Latency Bandwidth Capacity/IOPS
Register 0.25ns
L1 cache 1ns
L2 cache 3ns 8MB
L3 cache 11ns 45MB
DRAM 62ns 120GBs 6TB - 4 socket
NVRAM’ DIMM 620ns 60GBs 24TB - 4 socket
1-sided RDMA in Data Center 1.4us 100GbE ~700K IOPS
RPC in Data Center 2.4us 100GbE ~400K IOPS
NVRAM’ NVMe 12us 6GBs 16TB/disk,~2M/600K
NVRAM’ NVMf 90us 5GBs 16TB/disk,
~700/600K
67. NVMe Example: Redis on Flash
67
• 2M IOPS
• Sub-ms
latency
• 1:4
DRAM:NVM
split
(40x SATA
SSD)
68. NVMf Example: E8 Storage
68
$12M Series B led by Accel
• 10M IOPS in single appliance, accessed via RDMA
• 100us/50us read/write latency
• ...and this is with NAND Flash!
69. Data tiering in heterogeneous memory systems
DDR-DRAM NVM
Capacity per CPU 100s of GBs Terabytes
Read latency 1x 2x to 4x
Write bandwidth 1x ⅛ x to ¼ x
Estimated cost 5x 1x
Endurance 1016 106 to 108
Dulloor et al. 2016 (EuroSys)
70. What will it mean?
70
this!
All DRAM
(baseline)
All NVM
tiered memory,
with increasing
NVM proportion
Impact on cost?
Impact on
performance?
Impact on
performance/$?
71. Key-value store results - performance impact
Proportion of
DRAM
Performance
relative to all
DRAM
NVM only 1.15 - 1.45x
1/8 1.14 - 1.4x
1/4 1.06 - 1.13x
1/2 1.05x
beware!
72. Key-value store results - cost impact
Proportion of
DRAM
Performance/$
relative to all
DRAM
NVM only 3.45 - 4.5x
1/8 3.1x
1/4 2.3x
1/2 1.6x
73. And don’t forget about capacity!
Proportion of
DRAM
Performance
relative to all
DRAM
Performance/$
relative to all
DRAM
Capacity relative
to all DRAM
NVM only 1.15 - 1.45x 3.45 - 4.5x 4 - 10x
1/8 1.14 - 1.4x 3.1x 3.6 -8.9x
1/4 1.06 - 1.13x 2.3x 3.3 - 7.8x
1/2 1.05x 1.6x 2.5 - 5.5x
74. 74
Ultra low latency, PBs of data
• The RAMCloud storage system - Ousterhout et al. 2015
• Implementing linearizability at large scale and low latency - Lee
et al. 2015
1000+
Servers
PB+
Memory
RDMA RIFL
76. 76
No compromises
Distributed transactions with consistency, availability, and performance
Dragojević et al. 2015
“This paper demonstrates that new software in modern data
centers can eliminate the need to compromise. It describes the
transaction, replication, and recovery protocols in FaRM, a main
memory distributed computing platform. FaRM provides
distributed ACID transactions with strict serializability, high
availability, high throughput and low latency. These protocols
were designed from first principles to leverage two hardware
trends appearing in data centers: fast commodity networks
with RDMA and an inexpensive approach to providing non-
volatile DRAM.”
79. Redis is way more than just a ‘K-V store’...
b A new era of data structures?
80. Evolution of the humble data structure
80
Collection
classes
& stdlibs
?
single
thread/core
Concurrent
data types
multi-
thread/core
Distributed
data types
multi-server
Shared
concurrent
data types
shared-server
PACELC
81. 81
Libraries, Servers, & PACELC
App Instance App InstanceApp Instance
CRDT
Library
DS Server DS Server
Client Client Client Client Client
CRDT
Server
82. 82
CRDT Evolution
Operation Based
● prepare and effect
● smaller messages
● require reliable
exactly-once causal
broadcast
● membership
management
● no batching
State Based
● state shipping and
idempotent join
● larger messages
● support unreliable
communication
channels
● accommodate
batching
Delta-State Based
● local execution, ship
representation of the
effect & idempotently
join
● smaller messages
● support unreliable
communication
channels
● accommodate
batching
83. Data Structures! E.g. GCounter and PNCounter
{ replica_id value }
83
What are CRDTs made of?
{“A”->3, “B”->5}
HINCRBY gcounter A 1
{“A”->4, “B” -> 5}
HVALS gcounter… 4+5 = 9
{“A” -> 4, “B” -> 7}
{“A”->3, “B”->5}
HINCRBY gcounter B 1
HINCRBY gcounter B 1
{“A”->3, “B” -> 7}
{“A” -> 4, “B” -> 7}
delta B -> 7
delta A -> 4
A B
84. Causal Context X Dot Store
84
Causal ∂-CRDTs
version vector
(Redis hash) Dot.Event
(Redis set or hash)
Dot =
replica_id:sequence_no
INCR
+ a delta queue for propagation...
WATCH my_crdt:seq_no
next = INCR my_crdt:seq_no
MULTI
HSET my_crdt:ccontext replica_id next
SADD my_crdt:dotstore “replica_id:next:val”
RPUSH my_crdt:deltaq delta
EXEC
85. A new era of hardware (and what it might mean
for Redis)
A new era of data structures
Summary
a
b
91. Big Data with an IOT View
Volume
Velocity
Data streams and near-real time;
RFID tags, sensors and smart
metering.
Variety
Structured and unstructured.
Variability
Inconsistent; event triggered.
Complexity
Multiple sources; connect and correlate
relationships, hierarchies and multiple data
linkages.
Volume
Billions of sensors and machine to
machine transactions.
105. About Databricks
Founded by creators of Spark in 2013
Cloud enterprise data platform
- Managed Spark clusters
- Interactive data science
- Production pipelines
- Data governance, security, …
107. Please put up your hand
if you think your significant other
knows what Spark is?
108. open source data processing engine built around
speed, ease of use, and sophisticated analytics
largest open source data project with 1000+
contributors
109.
110.
111.
112.
113. “Spark is the Taylor Swift
of big data software.”
- Derrick Harris, Fortune
114. Analytics: a traditional approach
Hadoop is complex to operate
Data consistency is difficult to maintain
121. Enjoy RedisConf
Updates
• Your morning break is located downstairs in the Sponsor Showcase
• Three concurrent Breakout Sessions will begin at 11:15am
• The Developer Workshop begins at 1:00pm in the Developer Café
• Save your Redis Community Raffle ticket for today’s drawing
for a chance to win a Phantom Drone
122. 11:15am-12:00pm New Redis Capabilities: Own, Make, Share
1:00pm-1:45pm SSH I/O Streaming Via Redis-based Persistent
Message Queue
2:00pm-2:45pm Walmart and IBM Revisit the Linear Road Benchmark
3:15pm-4:00pm Back your App with MySQL & Redis, the Cloud
Foundry Way
4:15pm-5:00pm Postgres and Redis Sitting in a Tree
122
BREAKOUTS: WEDNESDAY, MAY 11