RedisConf17 breakout session

1. IoT backend
With Redis & S3 (and node.js)

2. Stefano Fratini
Engineering lead
Redis enthusiast
Working at Hypothesis Consulting
About me

3. Smart meters and IoT
IoT
Backend
API
DB

4. Smart meters and IoT - Problems
IoT
Backend
API
DB
x 100k
Cheap, cheap, cheap
bottleneck
20k tps 1k tps

5. Lots of devices
It has to be cheap
It has to be fast
It has to be simple
Problems

6. Old solution
IoT
Backend API
Mysql
Recent data + working data set (cache)
Working data set + historical data
Scheduled
task
Recent data + working data set (cache)

7. It worked OK
A hybrid solution designed around
a relational design
Mysql can only scale up and store
so much data
Working data set replicated and
eventually consistent
How did the old
solution score?

8. Old solution - evolution
IoT
Backend API
Mysql
It starts like this...
Master of truth
Caching layer

9. Old solution - evolution
IoT
Backend API
Mysql
… it ends like this
Master of truth
Dumb datastore

10. The Problem
Cleaner design
Forget Mysql and E-R
diagrams
Scalability
Overcome the limitations
of Mysql
Single Source Truth
Get rid of caching as
much as possible
Can we do any better?

11. The new solution
IoT
Backend API
Primary datastore
Historical dataset

12. How we store the energy data
1 day
Divide the time in blocks
1494892800
00:00
05/16/2017
GMT
1494979200
00:00
05/17/2017
GMT
1495065600
00:00
05/18/2017
GMT
1494978971

13. The solution - storing energy data
Data from the devices
Redis Lists (288 items)
with TTL
S3 files

14. The solution - querying energy data
Redis Lists (288 items)
with TTL
S3 files
API
Today’s data
+ working data set
Historical data

15. The data model?

16. The data model
TYPE REDIS TYPE Example
Objects (Users,
Devices, Accounts)
HASH vizi:ww-prod:devices:D704206006772
Indexes SETs or
ZSETs of IDs
vizi:ww-prod:devices:D704206006772:clients
Energy data Linked LISTs
of JSON
encoded data*
vizi:ww-prod:le:D704206006772:1494979200
*Similar to Josiah Carlson’s Redis as a Time Series DB

17. Learning curve on modeling the
data
Querying S3 and Redis is complex
Unforeseen indexes curse
S3 has some limitations
Issues?

18. Unforeseen Indexes
Use SCAN
More scalable than a
RDBMS table scan but
still O(n)
Rebuild your indexes
Use offline scripts to do that
OR
Bundle your indexing with your
backend code as a startup
script
You don’t always know in advance how you will need to
query your data

19. S3 as a datastore
● S3 = infinitely scalable key/value datastore
● There is no magic (Know thy database!)
● Documentation online:
○ 100 req/sec/partition on write
○ 300 req/sec/partition
● Partitioning based on file names
○ /vzdata/dev/le/00acd-D8000000001C9
● Querying and inserting based
on token-bucket approach

20. Buy one Redis instance
+ 1 API management tool for
free
Wait there’s
more!

21. API management
Authentication
Is this API key valid?
Usage
Who’s hammering my
APIs?
Throttling
Can we prevent misuse?
Can we achieve the following with Redis?
GET key ????

22. API management
Usage & Throttling
let now = parseInt(new Date().getTime()/1000);
let tpsTs = now - now % 5; // 5 seconds
let tpsKey = redisKeys.getApiKeyTpsCounterKey(apiKey, tpsTs);
let pipeline = this.redisClient.pipeline();
pipeline.incr(tpsKey);
pipeline.expire(tpsKey, 60); // 1 minute
pipeline.get(tpsKey);
pipeline.execute((err, res) -> { … })
apiKeys:key_08f...:tps:1488931200
we divide the time in buckets
we get the current tps back

23. Achievements
Supporting 100k devices on a
single m3.medium + 100k req/day
on the APIs
Ultra low TCO <100$/mo
No data maintenance required

24. Quick tour

25. In a nutshell
Redis lets you be efficient
Know your bottlenecks
It ain’t relational, baby

26. Questions?