In May of 2012, Socialcam exploded, gaining tens of millions of new users in just a few weeks. At the time, the service ran on 15 servers in a co-location facility in San Francisco. To meet new user traffic demands and continue to deliver maximum user satisfaction, Socialcam made the move to cloud services. With only two engineers and a constant barrage of users, there was limited time for technical transition, but Socialcam endured with no significant downtime. In this technical session, Socialcam co-founders Guillaume Luccisano and Ammon Bartram talk about their experience scaling Socialcam. They present the challenges they encountered, how they addressed them, and the technologies they used in the process. They focus particularly on how they used Amazon services in conjunction with their own hardware to keep Socialcam active with no significant downtime and no costly system redesign.

2. Ammon Bartram
• Wrote the iOs Socialcam app
• Previously lead video engineer at
Justin.tv
• Used to own a cow named Snork
Maiden

3. Guillaume Luccisano
• Built the Socialcam backend
• Previously worked on Rails
performance at Justin.tv
• French dude exploring the valley

4. We grew explosively
... and survived
Here is how

5. We’d just finished
Y-Combinator, and were
growing quickly

6. Simple rails stack, 15
servers in San Francisco,
Postgres/Mongo dbs

7. Everything was working

8. Then...

9. BOOM!
True viral growth

10. 30k new users an hour and growing...
Things start to break

11. Not enough CPU!

12. Day 1

13. Add boxes in the colo?
We can’t. Little space in the
colo. Servers take 2 weeks
to arrive

14. Add EC2 instances?
How can servers on AWS talk to our
database?
SSH tunnels!
Then VPN!

15. Are video comments
essential?
No
OK, kill them!

16. Is non US traffic essential?
No
OK, kill it

17. Yeah Baby!
Kill switches are born!

18. Barebone deployment...
Push code live to production
boxes
(And no more test suite :p)

19. It’s 9am, time to sleep...
Got back to work the
same day at 11am...
Site is up, but not
healthy!

20. Day 2

21. Move things to workers!

22. Add more monitoring

23. Add instances
Amazon has resource limits. We hit
them. Emergency late-night calls.
Thanks Tim!

24. DB is breaking
Too many reads!

25. Buy more
time!

26. We run our main db in
our own colo - a big SSD
box

27. We order faster drives
and more memory
(on amazon.com :p)

28. And take down the site to
upgrade
It’s bad, but sometimes, it’s ok!

29. Caching!

30. Purpose of caching:
Minimize backend requests per user
request

31. Common caching
techniques:
Static full-page caching
Server side includes
Ajax loading
Fragment caching

32. Couldn’t do static :/
API and website were too dynamic

33. We used a hybrid
solution
Using a combination of them all
depending on what makes sense

34. API list requests are cache
with a templating system
(ala Server Side Includes)

35. We use heavily web
fragment caching on the
website

36. Video lists are loaded
through ajax on the website

37. Object caching
Reduce database hits per backend
request

38. We ❤ Memcache
Super Fast in memory Key/Value
store

39. Optimizing memcache ;)
Our custom tricks!

40. Avoid keys expiration!

41. Use get multi
Video.get_by_ids([1, 2, 3, 4, 5])
=
One memcache call

42. Compress and serialize!
Cache = Rails.cache.instance_variable_get("@data")
SnappyCache = Memcached::SnappyCache.new(Cache)
SnappyYajlCache = Memcached::SnappyYajlCache.new(Cache)
SnappyMarshalCache = Memcached::SnappyMarshalCache.new(Cache)

43. DB is breaking
again Too many
writes!

44. (And Mongo doesn’t scale)

45. For us...

46. to the
rescue

47. Redis is like Memcache
But support complex structures:
Lists, sets, sorted sets, etc..

48. Initially, we stored videos in
Postres, and computed
feeds per request
Then we de-normalized it in
Mongo...

49. This broke.

50. We switched to a canonical
normalized list of videos
stored in Postgresql...

51. Coupled with a de-
normalized video feed for
active users stored in Redis

52. The mongo problem was
fixed, but all was still not
well

53. We had1Billion rows
stored in Postgres
(and there was only 25gb left on our main db)

54. Postgres could not take it

55. Joins are evil
Replaced them by
application logic

56. Then you can move
tables to their own DBs

57. (Don’t tell anyone: It’s ok to
duplicate some data to kill joins)

58. It was not
enough!
Followers table was too big for one server :/

59. Time to shard

60. Sharding is cutting a
table into multiple pieces,
each on its own server

61. Application logic routes
queries to the right shard
based on some field (id, name,
country...)

62. You can make that
simple or complex

63. We decided to go simple
:)
user_id % N

64. Relation.shard(user_id).count

65. With dynamic migration:
class Relation
def shard(user_id)
if Redis.sismember("relation_shard", user_id)
# return new correct sharded table
else
# return old relation table
end
end
end

66. To conclude
We
survived!

67. By... making heavy use
of AWS and open source
Postgres is awesome.
Redis is awesome.
Haproxy is awesome.

68. By not fearing hacks and
not being afraid of
breaking things

69. By turning off expensive,
non-vital features
(or entire countries; sorry Brazil!)

70. By calling in help from
friends

71. And most importantly...

72. By only fixing things that
were broken