Title TF Encrypted + RedisAI + Redis Streams Video: https://youtu.be/zYfS9SVeqSU Agenda 1. Redis AI + TensorFlow + PyTorch + Redis Streams Speaker: Dave Nielsen, Head of Community and Ecosystem @ Redis Labs (https://www.linkedin.com/in/dnielsen) 2. TensorFlow Encrypted Machine learning applied to healthcare or otherwise sensitive data may be blocked if privacy isn’t adequately addressed. In this presentation we give an overview of how the TF Encrypted open source library can be used to make predictions and fit models on encrypted data in TensorFlow, without first needing to master advanced cryptography. Speaker: Morten Dahl, Research Scientist @ DropOut Labs (https://www.linkedin.com/in/mortendahlcs) Date/Time 9-10am US Pacific Time (Third Monday of Every Month) ** RSVP & LOGIN HERE ** Eventbrite: https://www.eventbrite.com/e/1-hr-fre... Meetup: https://www.meetup.com/Advanced-Spark... Zoom: https://zoom.us/j/690414331 Webinar ID: 690 414 331 Phone: +1 646 558 8656 (US Toll) or +1 408 638 0968 (US Toll) Related Links PipelineAI Home: https://pipeline.ai PipelineAI Community Edition: https://community.pipeline.ai PipelineAI GitHub: https://github.com/PipelineAI/pipeline PipelineAI Quick Start: https://quickstart.pipeline.ai Advanced Spark and TensorFlow Meetup (SF-based, Global Reach): https://www.meetup.com/Advanced-Spark... YouTube Videos: https://youtube.pipeline.ai SlideShare Presentations: https://slideshare.pipeline.ai Slack Support: https://joinslack.pipeline.ai Web Support and Knowledge Base: https://support.pipeline.ai Email Support: help@pipeline.ai

1. Redis Data Structures and
Redis Streams
MAY 2019 | DAVE NIELSEN

2. 2
• Why is Redis so popular?
• Redis Data Structures (Cache, User Sessions, etc)
• New Use Cases - Microservices Framework
• What is a data stream?
• What are the typical challenges you face while
managing data streams?
• What is Redis Stream?
• How Redis Stream addresses the challenges?
• How to use Redis Streams?
Agenda

3. Who is Redis Labs?
3
Open source. The leading in-memory database platform,
supporting any high performance operational, analytics or
hybrid use case redis.io
The home of open source Redis, and commercial provider of
Redis Enterprise technology, platform, products & services.
redislabs.com/redis-enterprise/
Redis Enterprise Cloud redislabs.com/redis-cloud
- Redis Cloud Essentials
up to 30mb FREE!
- Redis Cloud Pro

4. 4
Examples

5. Redis Top Redis Differentiators
5
Simplicity ExtensibilityPerformance
NoSQL Benchmark
1
Redis Data Structures
2 3
Redis Modules
Lists
Hashes
Bitmaps
Strings
Bit field
Streams
Hyperloglog
Sorted Sets
Sets
Geospatial Indexes

6. 6
Possible Data Stream
Use Cases
Top Redis Use Cases

7. Simplicity: Data Structures - Redis’ Building Blocks
Lists
[ A → B → C → D → E ]
Hashes
{ A: “foo”, B: “bar”, C: “baz” }
Bitmaps
0011010101100111001010
Strings
"I'm a Plain Text String!”
Bit field
{23334}{112345569}{766538}
Key
7
2
”Retrieve the e-mail address of the user with the highest
bid in an auction that started on July 24th at 11:00pm PST” ZREVRANGE 07242015_2300 0 0=
Streams
à{id1=time1.seq1(A:“xyz”, B:“cdf”),
d2=time2.seq2(D:“abc”, )}à
Hyperloglog
00110101 11001110
Sorted Sets
{ A: 0.1, B: 0.3, C: 100 }
Sets
{ A , B , C , D , E }
Geospatial Indexes
{ A: (51.5, 0.12), B: (32.1, 34.7) }

8. 8
Redis Data Structures
1. Strings - Cache SQL queries, pages, fast data
2. Bitmaps - Store 1s/0s (Online/Offline) like Pinterest
3. Bit Fields - Store arrays of complex numbers
4. Hashes - Store record-like info (user sessions, favorites, etc.)
5. Lists - Store ordered data (for ingesting data)
6. Sets - Store unique/unordered data (tags, unique IPs, etc.)
7. Sorted Sets - Store real-time sorted data like a MMPORG Leaderboard
8. Geospacial Indexes - Store real-time location (Uber, Lyft, etc.)
9. Hyperloglog - Store probabilistic data (Google Search count)
10. Streams - Store and share event data (similar to Kafka)
Lists
Hashes
Bitmaps
Strings
Bit fields
Streams
Hyperloglog
Sorted Sets
Sets
Geospatial Indexes

9. What is a cache?

10. • GET mykey
• If mykey = (nil) then
– run SQL Query
– SET mykey = “query results” EX 5
• GET mykey
• For more, visit redis.io/commands/set
Redis cache is very easy …

11. Redis Streams

12. 12
What is a
Data Stream?

13. Simplest form of a Data Stream
13
ConsumerProducer

14. But Often You Have M Producers and N Consumers
14
Producer 2
Producer m
Producer 1
Producer 3
Consumer 1
Consumer n
Consumer 2
Consumer 3

15. It’s a bit more complex
15

16. Data Stream Components
16
Ingest Stream
Message Broker
or
Stream Processor
Output Stream Analytics
Collect large
volume of data
arriving in high
velocity
Store the data
temporarily
Transform
incoming data
into one or
more formats
as required by
the consumers
Push the data to
the consumers
Allow backward
looking queries to
pull data
Store the data and
perform analytical
operations such as
predictive analytics,
ML training, ML
classification and
categorization,
recommendations,
pattern matching, etc.
IoT
Activity logs
Messages

17. 17

18. Redis Supports All the Functions of a Data Stream
18
Ingest Stream
Message Broker
or
Stream Processor
Output Stream Analytics
Redis
• Streams
• Pub/Sub
• Lists
• Sorted Sets
• Streams
• Pub/Sub
• Lists
• Streams
• Sorted Sets
+
• Spark Connector
• ODBC/JDBC Connector
• Sorted Sets
• Sets
• Geo
• Bitfield
+
• RedisGraph
• RediSearch

19. 19

20. Understanding Redis Streams
20

21. Let’s look at the
challenges first
21

22. How to Support a Variety of Consumers
22
Analytics
Data Backup
Consumers
Producer
Messaging Real-time
Real-time or Periodic Lookup
Periodic Read

23. The Catch Up Problem
23
Image ProcessorProducer
Consumption Rate: 100/secArrival Rate: 500/sec
Redis Stream
Fast Slow
Backlog

24. How to Address The Catch Up Problem?
24
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
Scale Out
New Problem: Mutual Exclusivity

25. Data Recovery from Consumer Failures
25
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
Scale Out
Yet Another Problem: Failure Scenarios

26. Connection Failures
26
Producer
Image Processor
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
The solution must be resilient to failures

27. Here comes Redis Streams
27

28. What is Redis Streams?
28
Pub/Sub Lists Sorted Sets
It is like Pub/Sub, but
with persistence
It is like Lists, but decouples
producers and consumers
It is like Sorted Sets, but
asynchronous
+
• Lifecycle management of streaming data
• Built-in support for timeseries data
• A rich choice of options to the consumers to read streaming and static data
• Super fast lookback queries powered by radix trees
• Automatic eviction of data based on the upper limit

29. 1. It enables asynchronous data exchange between producers
and consumers
29
MessagingProducer
Consumer

30. 30
Analytics
Data Backup
Consumers
Producer
Messaging
2. You can consume data in real-time as it arrives or lookup
historical data

31. 31
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
3. With consumer groups, you can scale out and avoid backlogs

32. 32
Classifier 1
Classifier 2
Classifier n
Consumer Group
XREADGROUP
XREAD
Consumers
Producer 2
Producer m
Producer 1
Producer 3
XADD
XACK
Deep Learning-based
Classification
Analytics
Data Backup
Messaging
4. Simplify data collection, processing and distribution to
support complex scenarios

33. Quick Reference Guide: https://redislabs.com/docs/getting-started-redis-streams/
Redis Streams Commands
33

34. Commands: XADD, XREAD
34
Asynchronous producer-consumer message transfer
MessagingProducer
Consumer
XADD XREAD
XADD mystream * name Anna
XADD mystream * name Bert
XADD mystream * name Cathy
XREAD COUNT 100 STREAMS mystream 0
XREAD BLOCK 10000 STREAMS mystream $

35. Commands: XRANGE, XREVRANGE
35
Lookup historical data
Analytics
Data Backup
Consumers
Producer
Messaging
XADD
XREAD
XRANGE
XREVRANGE
XRANGE mystream 1518951123450-0 1518951123460-0 COUNT 10
XRANGE mystream - + COUNT 10

36. Scaling out using consumer groups
36
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Redis Stream

37. Scaling out using consumer groups
37
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Redis Stream
Consumer Group
Consumer 1
Consumer 2
Consumer 3
Consumer n
Unconsumed List
Consumers
XADD XGROUP XREADGROUP
XACK
XPENDING
XCLAIM

38. Command: XGROUP CREATE
38
Create a consumer group
Unconsumed List
mygroup
Alice
Bob
edcba
mystream
edcba
App A
App B
XGROUP CREATE
XGROUP CREATE mystream mygroup $ MKSTREAM

39. Command: XREADGROUP
39
Read the data
Unconsumed List
mygroup
Alice
Bob
App A
App B
a
ed
cb
XREADGROUP
XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream >
XREADGROUP GROUP mygroup COUNT 2 Bob STREAMS mystream >

40. Command: XACK
40
Consumers acknowledge that they consumed the data
Unconsumed List
mygroup
Alice
Bob
App A
App B
a
cb
XACK
XACK mystream mygroup 1526569411111-0 1526569411112-0

41. Command: XREADGROUP
41
Repeat the cycle
Unconsumed List
mygroup
Alice
Bob
App A
App B
a
cb
XREADGROUP
XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream >

42. How to claim the data from a consumer that failed while
processing the data?
42
Unconsumed List
mygroup
Alice
Bob
App A
App B
cb
Unconsumed List
mygroup
Alice
Bob
App A
App B
cb
XCLAIM

43. Commands: XPENDING, XCLAIM
43
Claim pending data from other consumers
Unconsumed List
mygroup
Alice
Bob
App A
App B
cb
XPENDING mystream mygroup - + 10 Bob
XCLAIM mystream mygroup Alice 0 1526569411113-0 1526569411114-0

44. Now is the time to
44

45. Sample Use Case: Social Media Analytics
45
a
b
x
Language classifier
XREADGROUP
Real-time reports
XREAD
Sentiment analyzers Consumer is x times
slower than the
producer
Location classifier
Influencer classifierRedis Streams

46. Demo
46
Redis Streams
Twitter Ingest Stream Influencer Classifier
https://github.com/redislabsdemo/IngestRedisStreams

47. Java Jedis https://github.com/xetorthio/jedis
Lettuce https://lettuce.io/
Python redis-py https://github.com/andymccurdy/redis-py
JavaScript ioredis https://github.com/luin/ioredis
node_redis https://github.com/NodeRedis/node_redis
.NET StackExchange.Redis https://github.com/StackExchange/StackExchange.Redis
Go redigo https://github.com/gomodule/redigo
go-redis https://github.com/go-redis/redis
C Hiredis https://github.com/redis/hiredis
PHP predis https://github.com/nrk/predis
phpredis https://github.com/phpredis/phpredis
Ruby redis-rb https://github.com/redis/redis-rb
Redis Clients that Support Redis Streams
47

48. Questions?
dave@redislabs.com
@davenielsen
Thanks to @roshankumar for the demo
48