This document discusses how Redis can be used to solve various data and application scenarios for a social Q&A platform. It describes 6 scenarios including using Redis strings for counters, Redis lists for a message queue, Redis hashes for user activity tracking, Redis bits for feature flags, Redis sorted sets for a social wall, and Redis pub/sub for real-time monitoring. It provides examples of how Redis data structures map to each use case and discusses capabilities like performance, scalability and data consistency.

1. RIDING REDIS @

2. 60M REGISTERED USERS
200M MONTHLY UNIQUE USERS
450M PAGEVIEWS A DAY

3. 600 QUESTIONS / SECOND
20K REQUESTS / SECOND
50K LIKES / MINUTE
250K REGISTRATIONS / DAY

4. ~600 SERVERS
~300 RUBY NODES
~50 JAVA NODES
~35 REDIS NODES
~90 MySQL NODES

5. COMPARISON
13B PAGEVIEWS 200M UNIQUE USERS 32M QUESTIONS ANSWERED
13B PAGEVIEWS*
300M UNIQUE USERS**
76.9M POSTS CREATED*
130B PAGEVIEWS***
N/A UNIQUE USERS 340M TWEETS CREATED***
_________________________________________________
* http://www.tumblr.com/press
** http://yahoo.tumblr.com/post/50902111638/tumblr-yahoo
***http://en.wikipedia.org/wiki/Twitter

6. REDIS

7. Open-source
Key-value database
In-memory
Data-structure server
REDIS

8. Salvatore Sanfilippo
@antirez
REDIS AUTHOR

9. TWITTER
INSTAGRAM
STACK-OVERFLOW
BLIZZARD
WHO IS USING REDIS?

10. 6 SCENARIOS
6 REDIS DATA-STRUCTURES

11. COUNTERS STRING
MESSAGES QUEUE LIST
USER ACTIVITY MATRIX HASH
FUNCTIONAL SWITCHES BIT OPERATIONS
THE WALL SORTED SET
REAL-TIME MONITORING PUB / SUB

12. Scenario 1 : Counters

13. GLOBAL COUNTERS
ACTIVE USERS
QUESTIONS ASKED TODAY
LIKES TOTALLY
REGISTRATIONS THIS MONTH

14. SELECT count(*) FROM users
WHERE state='active';
NAIVE SOLUTION

15. REDIS STRING
MOST BASIC REDIS TYPE
BINARY SAFE
UP TO 512 Mb
OPERATIONS
INCR, DECR

16. @redis.incr "users/active/total"
...
@redis.decr "users/active/total"
users / active / total 61 771 480
users / inactive / total 6 943 366
EXAMPLE

17. USERS COUNTERS

18. @redis.incr "user/#{@user.id}/answers"
..
@redis.decr "user/#{@user.id}/likes"
..
@redis.incr "user/#{@user.id}/gifts"
user / :user_id/ answers 57
user / :user_id / likes 13 905
user / :user_id / gifts 27
EXAMPLE #2: USERS COUNTERS

19. COUNTERS CONSISTENCY

20. CONSISTENCY = 1 / PERFORMANCE

21. @redis.expire "user/#{@user.id}/inbox_questions", 1.day
SOLVED MOST CASES FOR US
REDIS TTL

22. SCALABILITY

23. @redis_shard_map = {
0 => Redis.new(:host=>"redis_host_0"),
1 => Redis.new(:host=>"redis_host_1"),
...,
7 => Redis.new(:host=>"redis_host_7")
}
@redis = @redis_shard_map[@user.id % 8]
@redis.incr "user/#{@user.id}/answers"
SCALABILITY (2)

24. Scenario 2 : Messages Queue

25. THIRD PARTY INTERACTION

26. RUBY BLOCKING MODEL

27. Resque to the rescue
https://github.com/resque/resque

28. REDIS-BACKED LIBRARY FOR
● CREATING BACKGROUND JOBS
● PLACING THOSE JOBS ON MULTIPLE QUEUES
● AND PROCESSING THEM LATER
RESQUE

29. REDIS LIST
SIMPLY LISTS OF STRINGS
MAX LENGTH 232
- 1
CONSTANT TIME INSERTION
OPERATIONS:
LPUSH, LPOP, RPUSH, RPOP
LREM, LRANGE

30. WORKFLOW

31. Why Resque?

32. Resque.enqueue(PostToFacebookJob, "Hello world")
ADDING A NEW JOB

33. @redis.lpush 'queue:facebook',
'{
"class":"PostToFacebookJob",
"args":["Hello world"]
}'
O(1)
COMPLEXITY

34. @redis.rpop "queue:facebook"
O(1)
GETTING A JOB FROM QUEUE

35. Scenario 3 :
User activity matrix

36. ROBOTS
CRAWLERS
SPAM
TIME-BASED RESTRICTIONS

37. POST
ASK A QUESTION
LOGIN
LIKE
GET
PROFILE VIEW
USER ACTIONS

38. LIMIT ACTIVITY
PER MINUTE
PER HOUR
PER DAY
REQUIREMENTS

40. REDIS HASH
MAPS BETWEEN
STRING FIELDS AND VALUES
MAX PAIRS 232
- 1
OPERATIONS:
HGET, HGETALL
HSET, HDEL

41. LIKES QUESTIONS REQUESTS
user/:uid/date/:today/minute/:minute 12 3 27
user/:uid/date/:today/hour/:hour 34 15 113
user/:uid/date/:today/day/:day 158 22 529
STRUCTURE

42. def register_users_like!(user_id)
minute = @redis.hincrby minute_key, 'like', 1
hour = @redis.hincrby hour_key, 'like', 1
day = @redis.hincrby day_key, 'like', 1
end
EXAMPLE: REGISTER ACTIVITY

43. def allowed_to_like_questions?(user_id)
minute = @redis.hget minute_key, "likes"
hour = @redis.hget hour_key, "likes"
day = @redis.hget day_key, "likes"
return per_minute < LIKES_PER_MINUTE_THRESHOLD &&
per_hour < LIKES_PER_HOUR_THRESHOLD &&
per_day < LIKES_PER_DATE_THRESHOLD
end
EXAMPLE: ALLOWED?

44. @redis.expire per_minute_key, 1.minute
@redis.expire per_hour_key, 1.hour
@redis.expire per_day_key, 1.day
CLEANUP

45. SCALABILITY
SCALE BY
USER_ID
DATE
PHASE OF THE MOON

46. CONSISTENT HASHING

47. Scenario 4 :
Functional switches

48. TURN ON/OFF ANY FEATURE ON SITE
REQUIREMENTS

49. PHOTO ANSWERS
VIDEO ANSWERS
WALL
STREAM
DATABASE SHARDS
SET POSTS PER PAGE
FUNCTIONALITY

50. BIT OPERATIONS
BIT OPERATIONS
SETBIT, GETBIT
BITCOUNT

51. @redis.setbit common_settings_key, WALL_ENABLED, true
EXAMPLE

52. MASTER / SLAVE REPLICATION
SCALABILITY

53. Scenario 5 : The Wall

54. THE WALL

55. THE WALL

56. SHOW FRIENDS POSTS
INITIAL REQUIREMENT

57. SELECT * FROM questions q
LEFT JOIN followships f ON (q.user_id = f.friend_id)
WHERE f.user_id = :my_user_id
ORDER BY q.answered_at
LIMIT 0,25
SOLUTION

58. LATER REQUIREMENTS
LIKES INTRODUCED
SHOW RETWEETS
UNIQUENESS OF ANSWERS
ORDERED BY FIRST OCCURRENCE
PAGINATION NEEDED
DO NOT SHOW OWN POSTS
SHOW RETWEETS SINCE STARTED FOLLOWING A
FRIENDS

59. MORE REQUIREMENTS
DO NOT SHOW RETWEETS IF ANSWERER OR
RETWEETER IS DISABLED
SHOW LATEST FRIENDS WHO LIKED A QUESTION

60. OUR SOLUTION

61. IDEA
STORE SEPARATE SET OF QUESTIONS FOR EVERY USER

62. NON REPEATING COLLECTIONS OF
STRINGS
EACH MEMBER ASSOCIATED WITH
SCORE
QUICKLY ACCESS
ELEMENTS IN ORDER
FAST EXISTENCE TEST
FAST ACCESS TO ELEMENTS IN THE
MIDDLE
REDIS SORTED SET

63. user/:user_id/wall
score_1 score_2 ... score_N
question_id_1 question_id_2 ... question_id_N
score - timestamp, when the question_id first occurred in a set
STRUCTURE

64. ● GET USERS WALL
○ ZREVRANGEBYSCORE - O(log(N)+M)
● USER ANSWERED A QUESTION
○ ZADD - O(log(N))
● LIKE
○ ZRANK - O(log(N))
○ ZADD - O(log(N))
● REMOVE ANSWER
○ ZREM - O(M*log(N)
OPERATIONS

65. GUARANTEED 1000-1500 POSTS ON WALL
PERIODICALLY CALL
ZCARD
ZREMRANGEBYRANK
CLEANUP

66. USER_ID
SHARDING

67. Scenario 6 :
Real time monitoring

68. PATTERN DETECTION
REQUIREMENT

69. HUMAN vs MACHINE

70. MySQL TABLE PULL
INITIAL SOLUTION

71. PUBLISH / SUBSCRIBE MESSAGING
PARADIGM
ALLOW PATTERN-MATCHING
SUBSCRIPTIONS
OPERATIONS:
SUBSCRIBE, UNSUBSCRIBE
PUBLISH
REDIS PUB/SUB

72. SCHEMA

73. MODERATORS PANEL

74. Time complexity: O(N+M) where N is the number of clients
subscribed to the receiving channel and M is the total
number of subscribed patterns (by any client).
SCALING

75. So, why Redis?

76. SIMPLE
FAST
FLEXIBLE
ROBUST
FREE
WHY REDIS?

77. CLUSTERING
WHAT'S MISSING

78. Not covered?

79. SETS
LUA SCRIPTING
TRANSACTIONS
PIPELINED
NOT COVERED

80. JAVA

81. Questions