1. How you
can benefit
from using
Redis
javier ramirez
@supercoco9

7. I was
Lois Lane
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

8. In the beginner's
mind there are many
possibilities, in the
expert's mind there
are few.
Shunryu Suzuki
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

9. myth: the bottleneck
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

10. redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop,mset -P 16 -q
On my laptop:
SET: 513610 requests per second
GET: 641436 requests per second
LPUSH: 845665 requests per second
LPOP: 783392 requests per second
MSET (10 keys): 89988 requests per second
On a small digital ocean server ($5/month)
SET: 227816 requests per second
GET: 228258 requests per second
LPUSH: 251098 requests per second
LPOP: 251572 requests per second
MSET (10 keys): 43918 requests per second
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

11. myth: Redis is just like
memcached
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

12. myth: Redis is for queues
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

13. open source, BSD licensed, advanced
key-value store. It is often referred to as a
data structure server since keys can contain
strings, hashes, lists, sets and sorted sets.
http://redis.io
started in 2009 by Salvatore Sanfilippo @antirez
104 contributors at
https://github.com/antirez/redis
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

14. The Redis Manifesto
1.A DSL for Abstract Data Types
2.Memory storage is #1
3.Fundamental data structures for a
fundamental API
4.Two levels of API
5.Code is like a poem; it's not just something
we write to reach some practical result
6.We're against complexity
7.We optimize for joy
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

15. Redis makes
you think
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

16. Redis data types
Strings
Hashes
Lists
Sets
Sorted Sets
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

17. basic commands
Setting/getting/deleting/expiring
Increment/decrement
Pushing/popping
Checking membership, size...
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

18. much cooler commands
Setting only if exists
Blocking pop
(Blocking) pop from one list, push to another
Get/set string ranges (and bit operations)
Set intersections/diffs (and store)
Pub/sub
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

19. A chat in 6 lines of code
r=Redis.new driver: :hiredis
r.subscribe “chatrooms:42” do |on|
on.message do |topic, msg|
puts “#{topic}: #{msg}”
end
end
r.publish “chatrooms:42”, “Hello Fosdem”
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

20. atomicity
single threaded, so no
concurrency problems
transactions and lua
scripts to run multiple
operations atomically
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

21. Scripting with lua
You can use Lua for scripting Redis when you need to
atomically execute a sequence of commands in which
the output of a command is used as input for another
It reduces the need to use complex lock mechanisms
and simplifies dependencies between clients
You can even extend the functionality of Redis by using
Lua scripts
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

22. Redis keeps
everything
in memory
all the time
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

23. Persistance: RDB
Compact binary format
Saves snapshots every few minutes
Good for backups and synchronizing
If Redis crashes, a few minutes worth of
data will be lost
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

24. Durability: AOF
Log text format
Configurable durability
Large file, can slow down startup
If Redis crashes, typically one second of
data could be lost
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

25. replication & scaling out
one master, several
read-only slaves
sharding
Twemproxy & redis cluster
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

26. what's being used for
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

27. twitter
Every time line (800 tweets
per user) is on redis
5000 writes per second avg
300K reads per second
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

28. twitter
write API (from browser or client app)
follo
e per
on
user id
tweet id
wer
fanout (flockDB)
metadata
rpushx to Redis
user info from
gizmoduck
(memcached)
javier ramirez
@supercoco9
tweet info from
tweetypie
(memcached + mysql)
https://teowaki.com
your twitter
timeline
FOSDEM 2014

29. World Of Warcraft
Blizzard is quite secretive about it
Known to have at least 8 servers with 64Gb each to
serve user avatars
Known to have seen 1.000.000 users concurrently in
the Asian version of the game
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

30. pinterest object graph
per user
Redis SortedSet, with timestamp as the score to
store the users followed explicitly
store the users followed implicitly
store the user’s explicit followers
store the user’s implicit followers
store boards followed explicitly
Redis set to store boards unfollowed explicitly
per board
Redis Hash to store a board’s explicit followers
Redis Set to store a board’s explicit unfollowers
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

31. viacom
Object dependency graph. Cache on steroids
Redis as a queue for background jobs
Activity tracking and view counts buffering before saving
to mysql
Lua script working on a slave to recalculate ranking and
popularity of contents, then send the data to master.
The new process takes 1/60th less than the old one in
mysql
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

32. stack overflow
Three level cache:
local cache (no persistence)
sessions, and pending view count updates
site cache
hot question id lists, user acceptance rates...
global cache (separate Redis DB)
Inboxes, API usage quotas...
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

33. youporn
Most data is found in hashes with ordered sets used to
know what data to show.
zInterStore on: videos:filters:released,
Videos:filters:orientation:straight,Videos:filters:categorie
s:{category_id}, Videos:ordering:rating
Then perform a zRange to get the pages we want and
get the list of video_ids back.
Then start a pipeline and get all the videos from hashes.
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

34. github
Git as a service internally. Redis is used for
storing routing info matching user
repositories to server names
hipchat
Redis for caching. Information like which
users are in which rooms, presence
information, who is online, etc, so it doesn’t
matter which XMPP server you connect to.
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

35. when things go wrong
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

36. the instagram case
moving from redis to
cassandra: 75% savings on
servers
lesson learnt:
know when redis is not appropriate
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

37. the twilio case
credit card hell
lesson learnt:
know what you are doing.
Don't change config on the fly
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

38. Numbers can be scary
Twitter: 2TB RAM for Redis on its own servers
Blizzard: At least 8 servers with 64Gb each
Pinteres:t 110 Redis instances
Instagram: 1.2 TB of data
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

39. openredis
redis as a service on AWS
serving over 1000
customers with a single
machine
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

40. how teowaki is using redis
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

41. Abusing sidekiq/resque
keep track of every activity in the system, even if you
don't need them all right now:
- every page view
- every API request
- every time a record is created/updated/deleted
benefits:
- highly decoupled system
- easier to divide into services
- you can add behaviour without changing your app
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

42. intermediate cache
* As a very fast lightweight storage for analytics data before
sending them to our google bigquery based solution
* As a cache for attributes frequently looked up in join tables
(names, nicknames, guids, delegated or included
model names...)
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

43. Some of our uses of Lua
Expiring attributes inside a Redis hash
Inserting notifications into a list only if there are not
pending notifications from the same user for the
same scope
Paginating a list by an attribute
Manipulating JSON directly at the Redis layer
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

44. counters
Atomic counters can be safely invoked concurrently
from anywhere, so you can implement “like”
features, global sequences or usage monitoring
systems in highly concurrent applications for free.
You can share your counters with any other internal
application and still be sure they won't collide.
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

45. Temporary data
Redis allows us to self expire keys after a time has
passed. You can use this mechanism for a simple
cache
If you operate on a key with expiration time, you
can change its value and still keep the expiration
going. By combining a decrementing counter with
an expiration time, implementing usage quotas is
trivial
Also, you can inspect which keys you have in your
server efficiently using SCAN commands
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

46. nginx + lua + redis
apache + mruby + redis
Multiple levels of cache by using Redis on the webserver/
middleware layer
http://wiki.nginx.org/HttpRedis
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

47. summarizing
* Redis is more powerful than it seems
* Very fast, easy to use, simple, good documentation
* In-memory data structures, distributed, shared and persisted
* Good as data store, intermediate data store, cache or queue
* Lots of use cases, both in huge and smaller systems
You should probably use it a lot more
javier ramirez
@supercoco9
https://teowaki.com
FOSDEM 2014

48. Find related links at
https://teowaki.com/teams/javier-community/link-categories/redis
Dank u.
Merci.
If you enjoyed this talk, please sign up for
https://teowaki.com
Javier Ramírez
@supercoco9
FOSDEM 2014