RubyFuZa presentation giving an overview of redis with a couple of examples. Also quick mentions of Resque, Resque-Scheduler and Resque-Retry

1. Redis, Resque and Friends
Christopher Spring
https://github.com/autonomous
@autonomous
www.christopherspring.com

2. WTF is Redis?

3. WTF is Redis?
Remote Dictionary Server

4. Key-value storage...
# Insert a value for a key
SET some:key “value”
# Retrieve value for a key
GET some:key # => “value”

5. ... on steroids!
# Lists
RPUSH chores "Mow lawn"
LPUSH chores "Get dressed"
# Sets
SADD sith "Darth Maul"
SUNION sith evil
# Sorted sets
ZADD background:workers 50 "w1.oss.com"
ZADD background:workers 19 "w3.oss.com"
# Hash
HMSET client username "womble" password "secret_womble"
HGET client password

6. Strings

7. Hashes

8. Lists

9. Sets

10. Sorted sets

11. $ brew install redis
$ redis-server /usr/local/etc/redis-conf

12. Example: Tagging
$ redis-cli
...
redis> SADD post:17:tag "ruby"
redis> SADD post:17:tag "rails"
redis> SADD post:17:tag "redis"
redis> SADD post:20:tag "resque"
redis> SADD post:20:tag "redis"
redis> SINTER post:17:tag post:20:tag
1. "redis"

13. Transactions

14. Example: Transactions
$ redis-cli
...
redis> SET "player:1:coins" 30
redis> SET "player:2:coins" 63
# ... Players decide to trade some coin
redis> MULTI
redis> INCRBY "player:1:coins" 15
redis> DECRBY "player:2:coins" 15
redis> EXEC
# 1. (integer) 45
# 2. (integer) 48

15. In-memory storage
• Simple single process event
driven design
• No locks
• 100K+ operations per second

16. Asynchronous persistence
• Snapshotting - save 60 1000
# dump every 60 seconds if at least 1000
keys changed
• Append-only file - durable with log
rewrite support

17. Master-slave replication
slaveof '192.168.1.1' 6379
• Multiple slaves
• Slaves of slaves
• Scalability
• Defer save to
slaves

18. ... I’ve got too much data!

19. Redis Virtual Memory
• Only values swapped to disk
• Most recent values in memory
• Predictable memory usage...

20. 1M keys: 160 MB
10M keys: 1.6 GB
100M keys: 16 GB

22. MySQL + memcached
• Multiple copies of data
• Mainly scales reads
• No higher level functions

23. Client libraries for all:
• C • Java
• C# • Lua
• C++ • Node.js
• Clojure • Objective-c
• Common List • Perl
• Erlang • PHP
• Go • Python
• Haskell • Scala
• haXe • Smalltalk
• Io • Tcl

24. Client libraries for all:
• C • Java
• C# • Lua
• C++ • Node.js
• Clojure • Objective-c
• Common List • Perl
• Erlang • PHP
• Go • Python
• Haskell • Scala
• haXe • Smalltalk
• Io • Tcl

25. $ sudo gem install redis

26. Redis as a database...

27. require 'rubygems'; require 'redis'; require 'json'
class RubyFuZa
def initialize args class Dude < Struct.new(:first_name, :last_name); end
@db = args[:db]
@duration = args[:duration] # seconds
@namespace = "#{self.class}:#{args[:id]}" chris = Dude.new 'Chris', 'Spring'
@s_key = "#{@namespace}:speakers" marc = Dude.new 'Marc', 'Heiligers'
@a_key = "#{@namespace}:attendees" steve = Dude.new 'Steve', 'Hoffmeier' # Is jy my pa?
end
def start! redis = Redis.new
return "Already started!" if started? two_days = 2*24*60*60 # seconds
@db[@namespace] = 'started' conf = RubyFuZa.new(
@db.expire( @namespace, @duration ) :db => redis,
end :duration => two_days,
:id => "2011")
def started?
@db[@namespace] == 'started'
end conf.add_speaker chris # true
conf.add_attendee marc # true
def how_much_longer? conf.attendees # [chris, marc]
t = @db.ttl(@namespace)
t == -1 ? 'It's over!' : t
end conf.started? # false
conf.start! # true
def add_speaker( speaker ) conf.started? # true
@db.sadd( @s_key, speaker.to_json )
end
sleep( 2000 )
def add_attendee( attendee ) conf.how_much_longer? # two_days - 2000
@db.sadd(@a_key, attendee.to_json)
end
def attendees
@db.sinter(@s_key, @a_key).map{ |sa| JSON.parse( sa )}
end
end

28. Publish/Subscribe

29. require 'rubygems' $ redis-cli
require "redis" ...
redis> PUBLISH rubyfuza:chat "Smoke me a
redis = Redis.connect kipper..."
redis> PUBLISH rubyfuza:chat "I'll be back
trap(:INT) { puts; exit } for breakfast."
redis> PUBLISH rubyfuza:chat "exit"
redis.subscribe('rubyfuza:chat') do |on|
on.subscribe do |channel, subscriptions|
puts "Subscribed to ##{channel}"
end
on.message do |channel, message|
puts "##{channel}: #{message}"
redis.unsubscribe if message == "exit"
end
on.unsubscribe do |channel, subscriptions|
puts "Unsubscribed from ##{channel}"
end
end

30. Resque
• Backed by redis
• Asynchronous job server
• Multiple queues
• Sinatra based web-ui

31. class PostalWorker
@queue :post_box
def self.perform( user_id )
User.where( :id => user_id )
msg = UserMail.welcome_message( user )
msg.deliver
end
end
...
Resque.enqueue( PostalWorker, user.id )

32. Resque-Scheduler
• Queue jobs in the future!
• Recurring cron-style queueing
• Delayed jobs

33. database_cleaning:
cron: "0 0 * * *"
class: CleanupWorker
args:
description: "This jobs removes junk from the DB"

34. Resque.enqueue_at(
5.days.from_now,
SendFollowUpEmail,
:user_id => current_user.id
)

35. Resque-retry
• Retry failed jobs
• Set number of retries
• Set delay between retries
• Exponential back-off
• Delays built on resque-scheduler

36. class WebHookWorker
extend Resque::Plugins::Retry
@queue = :web_hooks
@retry_limit = 10
@retry_delay = 5*60 # Seconds
def self.perform(*args)
# ... trigger web-hook
end
end

37. In summary...

38. Data structure server
• Strings
• Hashes
• Lists
• Sets
• Sorted sets

39. Main use cases:
• Database
• Cache
• Messaging

40. Features:
• Fast
• Atomic operations
• Transactions
• Master-Slave
• Persist to disk
• Expiring keys
• Pub - Sub
• Redis Virtual Memory

41. Baie dankie!

42. Questions?

43. Links
• https://github.com/bblimke/copy-with-style-tmbundle
• http://redis.io/
• https://github.com/antirez/redis
• https://github.com/ezmobius/redis-rb
• http://antirez.com/
• https://github.com/defunkt/resque
• https://github.com/bvandenbos/resque-scheduler
• https://github.com/lantins/resque-retry