In these slide following projects are presented:
* redis_wrap: A Pythonic wrapper that makes it nicer to work with builtin Redis datatypes
* redis_graph: A sample graph database
* redis_simple_queue: A simple queue implemented on Redis list structure
* bitmapist: a powerful analytics library using Redis bitmaps, great for retention and cohort tracking
* fixedlist: a fixed list structure that can optimize timelines (and other things)
* how to use Lua scripting for more advanced data structures and better performance
These are Python projects, but some of them (like bitmapist) have been ported to other languages.
This talk was given to PyCon Belarus on 31 Jan. 2015
18. bitmapist + bitmapist.cohort
• Implements an advanced analytics library
on top of Redis bitmaps
• https://github.com/Doist/bitmapist
• Hundreds of millions of events for Todoist
• O(1) execution
19. bitmapist: Features
•Has user 123 been online today? This week?
•Has user 123 performed action "X"?
•How many users have been active have this month?
•How many unique users have performed action "X"
this week?
•How many % of users that were active last week are
still active?
•How many % of users that were active last month
are still active this month?
•O(1)! Using very small amounts of memory.
21. bitmapist: Usage
# Mark user 123 as active and has played a song
mark_event('active', 123)
mark_event('song:played', 123)
# Answer if user 123 has been active this month
assert 123 in MonthEvents('active', now.year, now.month)
assert 123 in MonthEvents('song:played', now.year, now.month)
# How many users have been active this week?
print len(WeekEvents('active', now.year, now.isocalendar()[1]))
# Perform bit operations. How many users that
# have been active last month are still active this month?
active_2_months = BitOpAnd(
MonthEvents('active', last_month.year, last_month.month),
MonthEvents('active', now.year, now.month)
)
print len(active_2_months)
26. fixedlist: Cheating!
• Fixed timeline size
• O(1) insertion
• O(1) update
• O(1) get
• Cacheable
Solution that Facebook and Twitter use
27. fixedlist
2.5x faster than pure Redis solution
1.4x less memory than pure Redis solution
https://github.com/Doist/fixedlist
28. fixedlist: Usage
# Add a value to a list
fixedlist.add('hello', 'world')
# Add mutliple values to multiple keys at once
fixedlist.add(['hello1', 'hello2'], ['world1', 'world2'])
# Get valuesfrom a list
assert fixedlist.get('hello') == ['world', 'world1', 'world2']
# Remove a value
fixedlist.remove('hello', 'world1')
Saved Plurk tens of thousands of $
30. Redis+Python: Incr implementation
def incr_python(key, delta=1, system='default'):
client, scripts = get_redis(system)
with client.pipeline() as p:
p.watch(key)
value = delta
old = p.get(key)
if old:
value = int(old) + delta
p.set(key, value)
p.unwatch()
return value
31. Redis+Lua: Incr implementation
scripts = {
'incr': client.register_script(_load_lua_script('incr.lua'))
}
...
def incr_lua(key, delta=1, system='default'):
client, scripts = get_redis(system)
return scripts['incr'](keys=['key', 'delta'], args=[key, delta])
local delta = tonumber(ARGV[2])
local value = delta
local old = tonumber(redis.call('get', ARGV[1]))
if old then
value = value + old
end
if not redis.call('set', ARGV[1], value) then
return nil
end
return value
32. Perfomance: Lua 3x faster
Python
time python test_incr_python.py 300000
python test_incr_python.py 300000 37.77s user 12.00s
system 73% cpu 1:07.73 total
Lua
time python test_incr_lua.py 300000
python test_incr_lua.py 300000 10.76s user 2.85s
system 66% cpu 20.513 total
https://github.com/amix/demo-redis-python-lua
33. fixedlist in Lua
Proof of concept
Tokyo Tyrant example
https://gist.github.com/amix/f15508ac6a8b534c3290
