RedisConf19

1. PRESENTED BY
Atom
The Redis Streams-Powered
Microservices SDK
Dan Pipe-Mazo
Elementary Robotics, CTO

2. PRESENTED BY
Agenda:
1. Starting with a demo!
We’ll see atom and redis in action using a depth-sensing camera
2. SDK Architecture (featuring Redis Streams!)
We’ll dive into the SDK in the context of the demo and take a look
at what’s going on under the hood
3. Open Source + Signup Give-Away
We’ll discuss open-sourcing the core of atom and the hardware
we’re giving away to our early developers

3. PRESENTED BY
DEMO:
Object Detection and Segmentation Built
with Atom & Redis

4. PRESENTED BY
Atom OS Overview
An in-depth dive into how we built the core of Atom

5. PRESENTED BY
1. Atom is a specification and a set of
client libraries that allow users to
create reusable microservices that
interact with each other through Redis
2. Through Docker and docker-compose
we can link together these
microservices to launch applications
3. By abstracting applications into
microservices we can do the following:
● Allow each piece of code to be
developed in the optimal language.
● Easily reuse and share code
elementary-robotics/atom atomdocs.io
Atom OS: High Level

6. PRESENTED BY
1. Use microservices to allow for reusable code elements with
zero install or dependencies and scalable message passing.
2. Industries still rely on copying and pasting code into new projects
far too often and we need better reusability to scale.
3. With full-stack hardware and software products, things like
machine learning and computer vision (python) should be
implemented in different language from your embedded code (C).
The Goal of Atom OS

7. PRESENTED BY
Redis 5.0+ Server
Unix socket
and/or TCP
Atom Command
Line Interface
English-like interface
to interact with Atom
Nucleus
Architecture and Terminology

8. PRESENTED BY
Atom
Language
Clients
Multi-OS
Graphics
Architecture and Terminology
+ more in
development!

9. PRESENTED BY
- Any language with an atom language
client
- Publish data
- Expose commands and send
responses
$MY_ELEMENT
Your
Code!
FROM elementaryrobotics/atom
Architecture and Terminology

10. PRESENTED BY
Realsense
Elements
Stream Viewer
Segmentation
Nucleus
tmpfs
Redis Unix Socket
Demo Architecture: Camera

11. PRESENTED BY
Atom: Powered by Redis Streams
Powerful, efficient time-series data pub/sub and req/rep

12. PRESENTED BY
Our goal:
● Abstract out complex engineering problems into reusable, sharable elements.
● Don’t sacrifice performance or increase complexity in doing so
Architecture Question: Why microservices?
Questions we asked ourselves before building Atom:
● Do we need another microservice framework?
(gRPC, thrift, REST, ROS, zeroMQ, DDS)
● How would we do it if we weren’t going to build this in a reusable, abstracted
fashion?

13. PRESENTED BY
Data Publication and Subscription
● Publishing should be stateless and fire-and-forget
● Consumption should be able to be regulated by the consumer
○ Solve the “slow subscriber” problem, i.e. how to handle a
subscriber who only wants 1Hz updates on a 1kHz stream
● Low latency
● Support many parallel clients without any
extra burden on publisher / performance hit
Command and Response
● Should be able to call commands and receive responses across
as many languages as possible
● Easy to make command either synchronous or asynchronous
● Easy load-balancing without complicated multi-threading
Serialization
● Optional and not overly burdensome on either the CPU or the user’s sanity
● If serialized, messages can ideally be read without knowing the schema
Architecture Question: What are we looking for?

14. PRESENTED BY
Install and OS Requirements
● Write code once, it should work on any
OS (including graphics!)
● Setup should be as minimal as possible.
OS and/or system-related install bugs
are the worst.
Language Support
● Support as many languages as possible.
● Allows each problem to be solved in it
(or its programmer’s) ideal language.
● Atwood’s law: any application that can
be written in JavaScript, will eventually
be written in JavaScript.
Architecture Question: What are we looking for? ...Contd
Service Discovery
● Should be able to discover other
microservices in the system.
● For each microservice, should be
able to identify its health, available
commands and streams.
Logging
● Everything should be able to be logged.
● Failures should be easily traceable to an
outside observer.

15. PRESENTED BY
It turns out that’s a pretty big list of things we’re asking for, but there’s a solution!
>25 languages
Streams allow for
novel data flows
Production-tested
and well-supported
Redis
Requirements
installed in container
Multi-OS
Docker-compose
Docker
>25 languages
Easy, fast, pretty
much JSON
Completely optional
MessagePack
+ +
Architecture Solution

16. PRESENTED BY
Redis Streams
Enabling new data paradigms

17. PRESENTED BY
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
...
Redis Streams: Overview

18. PRESENTED BY
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB
...
stream: s2
Redis Streams: Overview

19. PRESENTED BY
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
XREAD BLOCK N STREAMS s1 $
XREAD BLOCK N STREAMS s1 ID
Subscribe to all entries from one stream
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB
...
stream: s2
Redis Streams: Overview

20. PRESENTED BY
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
XREAD BLOCK N STREAMS s1 $
XREAD BLOCK N STREAMS s1 ID
Subscribe to all entries from one stream
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB
...
stream: s2
XREAD BLOCK N STREAMS s1 s2 ID1 ID2
Subscribe to all entries from multiple streams
Redis Streams: Overview

21. PRESENTED BY
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
XREAD BLOCK N STREAMS s1 $
XREAD BLOCK N STREAMS s1 ID
Subscribe to all entries from one stream
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB
...
stream: s2
XREAD BLOCK N STREAMS s1 s2 ID1 ID2
Subscribe to all entries from multiple streams
XREVRANGE s1 + - COUNT N
Get the latest N entries from a stream
Redis Streams: Overview

22. PRESENTED BY
XADD s2 MAXLEN ~ X kA vA kB vB …
XADD s1 MAXLEN ~ X k1 v1 k2 v2 …
stream: s1
0
-
k1
v1
k2
v2
1
-
k1
v1
k2
v2
2
-
k1
v1
k2
v2
X
-
k1
v1
k2
v2
XREAD BLOCK N STREAMS s1 $
XREAD BLOCK N STREAMS s1 ID
Subscribe to all entries from one stream
...
0
-
kA
vA
kB
vB
1
-
kA
vA
kB
vB
2
-
kA
vA
kB
vB
X
-
kA
vA
kB
vB
...
stream: s2
XREAD BLOCK N STREAMS s1 s2 ID1 ID2
Subscribe to all entries from multiple streams
XREVRANGE s1 + - COUNT N
Get the latest N entries from a stream
XRANGE s1 - + COUNT N
Get the oldest N entries from a stream
Redis Streams: Overview

23. PRESENTED BY
Realsense
USB
Realsense
element
Host Compute
XADD
frames
realsense:pointcloud
realsense:color
realsense:depth
Redis Streams: Overview

24. PRESENTED BY
Realsense
USB
Realsense
element
Host Compute
XADD
frames
Stream
viewer
element
XREAD BLOCK N
realsense:pointcloud
realsense:color
realsense:depth
Redis Streams: Overview

25. PRESENTED BY
Realsense
USB
Realsense
element
Host Compute
XADD
frames
Stream
viewer
element
Segmentation
Runtime
element
XREAD BLOCK N
XREVRANGE + -
realsense:pointcloud
realsense:color
realsense:depth
Redis Streams: Overview

26. PRESENTED BY
Realsense
USB
Realsense
element
Host Compute
XADD
frames
Stream
viewer
element
Segmentation
Runtime
element
Segmentation
retraining
element
XREAD BLOCK N
XREVRANGE + -
XRANGE last +
realsense:pointcloud
realsense:color
realsense:depth
Redis Streams: Overview

27. PRESENTED BY
Improvements over Pub/Sub
● Redis acts as an N-value last value cache
○ Can query for the most recent piece of data without having to monitor the stream
● Can traverse the stream as convenient, asking for all data since last read
● N is set by publisher and redis auto-prunes when efficient (when using XADD MAXLEN ~)
Consumer Groups
● Single consumer: Redis keeps track of where you were in the stream
● Multiple consumers: Redis auto-routes messages and provides
introspection and failover handling. (XACK)
Multiple Interaction Paradigms
● Can replicate pub-sub if desired, else can use last value cache
● Producer doesn’t care how the clients are interacting with the data
Benefits of Data Publication and Subscription

28. PRESENTED BY
Requestor: “foo” Responder: “bar”
elem:bar:req XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
elem:foo:rep
Redis Streams: Async or Sync Command and Response

29. PRESENTED BY
Requestor: “foo” Responder: “bar”
elem:bar:reqXADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z
elem:foo:rep
XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
Redis Streams: Async or Sync Command and Response

30. PRESENTED BY
Requestor: “foo” Responder: “bar”
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
elem:foo:rep
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
Redis Streams: Async or Sync Command and Response

31. PRESENTED BY
Requestor: “foo” Responder: “bar”
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
elem:foo:rep
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
Redis Streams: Async or Sync Command and Response

32. PRESENTED BY
Requestor: “foo” Responder: “bar”
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
elem:foo:rep
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T
(entryID, {“elem”: bar, “id”: ID, “time”: T})
Redis Streams: Async or Sync Command and Response

33. PRESENTED BY
Requestor: “foo” Responder: “bar”
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
elem:foo:rep
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T
(entryID, {“elem”: bar, “id”: ID, “time”: T})
XREAD BLOCK T STREAMS elem:foo:rep lastEntryID
Redis Streams: Async or Sync Command and Response

34. PRESENTED BY
Requestor: “foo” Responder: “bar”
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
elem:foo:rep
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T
(entryID, {“elem”: bar, “id”: ID, “time”: T})
XREAD BLOCK T STREAMS elem:foo:rep lastEntryID XADD elem:foo:rep MAXLEN ~ X elem bar id ID resp R
Redis Streams: Async or Sync Command and Response

35. PRESENTED BY
Requestor: “foo” Responder: “bar”
elem:bar:req
(entryID, {“elem”: foo, “cmd”: Y, “data”: Z})
elem:foo:rep
XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID
XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID
XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T
(entryID, {“elem”: bar, “id”: ID, “time”: T})
XREAD BLOCK T STREAMS elem:foo:rep lastEntryID
(entryID, {“elem”: bar, “id”: ID, “resp”: R})
XADD elem:foo:rep MAXLEN ~ X elem bar id ID resp R
Redis Streams: Async or Sync Command and Response

36. PRESENTED BY
Consumer Groups
● Single consumer: All commands go to single copy of
element, redis keeps track of where you are in
processing your command queue
● Multiple, idempotent consumers: Redis auto-routes
commands to instances of the same element and
load-balances for you
Sync vs Async
● Completely up to the caller, can choose to wait for
response or not
Logging and Introspection
● Each command in the system can be uniquely identified
by the tuple of
(element, stream ID)
Benefits of Command and Response
Requestor REDIS
elem:bar:req
elem:foo:rep
Responder

37. PRESENTED BY
Obligatory: None
● No serialization required; redis supports binary data
● Serialization agreement is left to the two sides of the
message
Supported in the spec: MessagePack
● Allows for type-strict messaging if desired using explicit
casts of received messages
● Allows for optional parameters if using JSON-like objects
● Can sniff the wire and decode a packet since the types
are encoded into the schema
Camera Data
● Typically leave in native/raw format. Don’t waste
time/CPU.
Redis Streams Serialization
Requestor REDIS
elem:bar:req
elem:foo:rep
Responder

38. PRESENTED BY
DEMO:
Voice-Activated Waveform Generation,
Data Capture and Plotting

39. PRESENTED BY
Voice
Elements
Waveform
Record
Nucleus
tmpfs
Redis Unix Socket
Demo Architecture: Voice
Voice Demo

40. PRESENTED BY
Atom: Summary

41. PRESENTED BY
Atom OS: What is it?
● A specification around using redis streams to implement an RPC and messaging protocol with the
functionalities described in this talk
● A set of language clients that implement said specification in as many languages as possible
● A pre-compiled docker container that has all of the requirements and language clients you need to get up
and running quickly
○ elementaryrobotics/atom
● A suite of reusable elements that are deployed by us and the community on dockerhub that expose
functionality using the atom SDK
○ Realsense
○ Stream-viewer
○ Segmentation
○ Voice
○ Record
○ More coming soon!

42. PRESENTED BY
Atom OS: Open Source
● Source code for Atom and elements can be found on github
○ https://github.com/elementary-robotics
● Docker containers are built and shipped with CI/CD using CircleCI to dockerhub
○ https://circleci.com/gh/elementary-robotics/atom
○ https://hub.docker.com/u/elementaryrobotics/
● Documentation with walkthroughs and examples
○ https://atomdocs.io
● Language support currently implemented for C, C++ and Python
○ Please help us add languages!
● Specification Improvements for v2.0
○ Want to add heartbeat, better consumer groups, parameter server, any more ideas you have!

43. Thank you!

44. PRESENTED BY
Atom
The Redis Streams-Powered
Microservices SDK
Dan Pipe-Mazo
Elementary Robotics, CTO

45. PRESENTED BY