SpringOne Platform 2019 Session Title: Beyond Caching: Extending Redis Enterprise for Real-Time Streams Processing Speaker: Sheryl Sage, Director of Partner Marketing, Redis Labs Youtube: https://youtu.be/ldEqSZFm1kY

1. Beyond Caching: Extending Redis to
Real-time Streams Processing
October 7–10, 2019
Sheryl Sage, Director Marketing
Redis Labs
Beyond Caching: Extending Redis to
Real-time Streams Processing
October 7–10, 2019
Sheryl Sage, Director Marketing
Redis Labs

2. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
• What is a data stream?
• What are the typical challenges
you face while managing data
streams?
• What is Redis Stream?
• How Redis Stream addresses the
challenges?
• How to use Redis Streams?
2

3. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Who We Are
Open source. The leading in-memory NoSQL database,
supporting any high performance operational, analytics or
hybrid use case.
The open source home and commercial provider of Redis
Enterprise technology, platform, products & services.

4. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Redis Top Differentiators

5. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Key
"I'm a Plain Text String!" Strings
Hyperlog-logs
Cache
Counting
Data Structures - Redis’ Building Blocks
[ A → B → C → D → E ] Lists Queues
{23334}{112345569}{766538}{665455} Bit field Efficient Integers
{ A: “foo”, B: “bar”, C: “baz” } Hashes Sessions/profiles
{ A , B , C , D , E } Sets Recommendation
Engine
{ A: 0.1, B: 0.3, C: 100, D: 1337 } Sorted Sets Leader Board
00110101011001110010101010 Bitmaps Bitmap encoding
{ A: (51.5, 0.12), B: (32.1, 34.7) } Geospatial
Indexes Location Services
00110101 11001110 10101010
{id1=time1.seq1(A:“xyz”), id2=time2.seq2(D:“abc”, )} Streams

6. strings, ttl
lists, sets,
pub/sub
streams
search
ML
+
Graph
Relationships Graph
RAM + Flash
+ Disk
Client Location /
Proximity Geo
Intelligent Session
Management Hash, Sets
API Rate Limiting
Hash
User SessionsHashes, Sets
Web App Architecture 101
Pub/Sub
Messaging Pub/Sub
JSON Storage
JSON
medium.com

7. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Redis Enterprise for Production Deployments

8. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Node Architecture

9. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Delivered in Any Form

10. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What is a
Data
Stream?

11. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Simplest form of a Data Stream
ConsumerProducer

12. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
But Often You Have M Producers and N
Consumers
Producer 2
Producer m
Producer 1
Producer 3
Consumer 1
Consumer n
Consumer 2
Consumer 3

13. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
It’s a bit more complex than this

14. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Data Stream Components

15. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Good News

16. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Redis Supports All the Functions of a Data
Stream

17. Understanding Redis
Streams

18. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Let’s look at the challenges first

19. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
How to Support a Variety of Consumers
Analytics
Data Backup
Consumers
Producer
Messaging Real-time
Real-time or Periodic Lookup
Periodic Read

20. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
The Catch Up Problem
Image ProcessorProducer
Consumption Rate: 100/secArrival Rate: 500/sec
Redis Stream
Fast Slow
Backlog

21. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
How to Address The Catch Up Problem?
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
Scale Out
New Problem: Mutual Exclusivity

22. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Data Recovery from Consumer Failures
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
Scale Out
Yet Another Problem: Failure Scenarios

23. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Connection Failures
Producer
Image Processor
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
The solution must be resilient to failures

24. Here comes Redis Streams

25. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What is Redis Streams?
Pub/Sub Lists Sorted Sets
It is like Pub/Sub, but
with persistence
It is like Lists, but decouples
producers and consumers
It is like Sorted Sets, but
asynchronous
+
• Lifecycle management of streaming data
• Built-in support for timeseries data
• A rich choice of options to the consumers to read streaming and static data
• Super fast lookback queries powered by radix trees
• Automatic eviction of data based on the upper limit

26. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
1. It enables asynchronous data exchange
between producers and consumers
MessagingProducer
Consumer

27. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Analytics
Data Backup
Consumers
Producer
Messaging
2. You can consume data in real-time as it
arrives or lookup historical data

28. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
3. With consumer groups, you can scale out and
avoid backlogs

29. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Classifier 1
Classifier 2
Classifier n
Consumer Group
XREADGROUP
XREAD
Consumers
Producer 2
Producer m
Producer 1
Producer 3
XADD
XACK
Deep Learning-based
Classification
Analytics
Data Backup
Messaging
4. Simplify data collection, processing and
distribution to support complex scenarios

30. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Quick Reference Guide
https://redislabs.com/docs/getting-started-redis-streams/
Redis Streams Commands

31. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Commands: XADD, XREAD
Asynchronous producer-consumer message transfer
MessagingProducer
Consumer
XADD XREAD
XADD mystream * name Anna
XADD mystream * name Bert
XADD mystream * name Cathy
XREAD COUNT 100 STREAMS mystream 0
XREAD BLOCK 10000 STREAMS mystream $

32. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Commands: XRANGE, XREVRANGE
Lookup historical data
Analytics
Data Backup
Consumers
Producer
Messaging
XADD
XREAD
XRANGE
XREVRANGE
XRANGE mystream 1518951123450-0 1518951123460-0 COUNT 10
XRANGE mystream - + COUNT 10

33. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Scaling out using consumer groups
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Redis Stream

34. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Scaling out using consumer groups
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Redis Stream
Consumer Group
Consumer 1
Consumer 2
Consumer 3
Consumer n
Unconsumed List
Consumers
XADD XGROUP XREADGROUP
XACK
XPENDING
XCLAIM

35. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Command: XGROUP CREATE
Create a consumer group
Unconsumed List
mygroup
Alice
Bob
edcba
mystream
edcba
App A
App B
XGROUP CREATE
XGROUP CREATE mystream mygroup $ MKSTREAM

36. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Command: XREADGROUP
Read the data
Unconsumed List
mygroup
Alice
Bob
App A
App B
a
ed
cb
XREADGROUP
XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream >
XREADGROUP GROUP mygroup COUNT 2 Bob STREAMS mystream >

37. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Command: XACK
Consumers acknowledge that they consumed the data
Unconsumed List
mygroup
Alice
Bob
App A
App B
a
cb
XACK
XACK mystream mygroup 1526569411111-0 1526569411112-0

38. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Command: XREADGROUP
Repeat the cycle
Unconsumed List
mygroup
Alice
Bob
App A
App B
a
cb
XREADGROUP
XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream >

39. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
How to claim the data from a consumer that
failed while processing the data?
Unconsumed List
mygroup
Alice
Bob
App A
App Bcb
Unconsumed List
mygroup
Alice
Bob
App A
App B
cb
XCLAIM

40. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Commands: XPENDING, XCLAIM
Claim pending data from other consumers
Unconsumed List
mygroup
Alice
Bob
App A
App B
cb
XPENDING mystream mygroup - + 10 Bob
XCLAIM mystream mygroup Alice 0 1526569411113-0 1526569411114-0

41. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Java Jedis https://github.com/xetorthio/jedis
Lettuce https://lettuce.io/
Python redis-py https://github.com/andymccurdy/redis-py
JavaScript ioredis https://github.com/luin/ioredis
node_redis https://github.com/NodeRedis/node_redis
.NET StackExchange.Redis https://github.com/StackExchange/StackExchange.Redis
Go redigo https://github.com/gomodule/redigo
go-redis https://github.com/go-redis/redis
C Hiredis https://github.com/redis/hiredis
PHP predis https://github.com/nrk/predis
phpredis https://github.com/phpredis/phpredis
Ruby redis-rb https://github.com/redis/redis-rb
Redis Clients that Support Redis Streams
41

42. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Sample Use Case: Social Media Analytics
https://github.com/redislabsdemo/IngestRedisStreams

43. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Redis Streams
Twitter Ingest Stream Influencer Classifier
https://github.com/redislabsdemo/IngestRedisStreams

44. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
44

45. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Questions
?
?
?
?
?
?
?
?
?
?
?