jFrog uses Redis to horizontally scale its data processing for bintray. Learn how.

1. XXX
Horizontally scaled data
processing architecture
Using Redis
@

2. About me
Ophir Hordan
Member of Jfrog swamp
Leading Bintray development

3. Bintray
– Leading Distribution platform
– Support distribution of Maven,
Debian, RPM, Docker, nuget….
– Use CDN for fast download
– Runs in multiple site around the
world

4. Some Numbers___________
Packages 300k
Downloads 900m/Month

8. Services
– Download binaries
– Web application
– REST API
– Backend micro services

10. The Layout
UI &
REST API
MongoDB
Download
Server
CouchDB
Index
Calculation
Async
Process
Event
Processing
Event
Processing

11. No single point of failure

12. True distribution

13. Quiet period

14. Data Sharding

16. and Twemproxy aka nutcracker
Twemproxy

17. The flow
Service
Twemproxy
Micro
Service
Twemproxy

18. DB 0
Inbound Data
DB 1
Queue
management and
quite period
DB 2
Working DB and
retries counter
- Zset by Emestemp
- Hash for data
payload
Hash keys
Value with data payload
- Hash for data payload
- Hash for retry
counEng and recovery
data
Redis structure
LUA
Scan all keys
Move to db1
zrangeByScore
(current Emestamp
minus 5 seconds)

19. Service Pod
MicroserviceRedis
Outbound
Twemproxy
Redis
Pod
Pod
Inbound
Twemproxy
Microservice

20. Things we learned on the way…
• Multiple values in single key is problematic to
monitor
• aof file disk flush settings has big impact on
Redis performance
• No official package of Redis and Twemproxy
• Twemproxy only support single Redis db
• Redis can easily hold Millions of keys
• Protocol buffer works well as Redis document
• Twemproxy doesn't have fail over
• Twemproxy has limited sharding support (was
good enough for us)

21. QUESTIONS?