Facebook is one of the top sites on the internet and supports more than 900 million users. It handles billions of messages, hundreds of millions of photos, and generates hundreds of terabytes of data - every day! This data is also becoming more complex and interconnected over time. Every page the site serves, requires processing large amounts of data and needs to be rendered in milliseconds. Business and practical constraints dictate that more users are served with less resources. In addition, product changes regularly occur in a rapid manner. These constraints dictate that the site requires an infrastructure that is scalable, fast, efficient and flexible beyond what has been built ever before. In this talk, we will share key learning from our experience in building an infrastructure that addresses the above challenges. In particular, we will discuss key components of the Facebook software architecture, instrumentation and data collection mechanisms that allow us to monitor the health of the site, and innovative tools that analyze vast amount of data to help us pre-empt site issues and help identify root causes when things go wrong. We describe how this infrastructure and tools allow the engineers to move fast and rapidly launch products as Facebook builds for a billion users and beyond.

1. Enabling Fast Pages and Furious
Development While Supporting a
Billion Users
Subbu Subramanian, Ph.D.
Software Engineer

2. Pottery Challenge

3. Pottery Challenge: Day 3
Team 1: Make a PERFECT Pot Team 2: Make 20 pots

4. 950,000,000

5. 500M 700B 30B 2.5M
daily active users minutes spent pieces of content sites using
on the site every shared each month social plugins
month
Latests stats @ http://newsroom.fb.com/content/default.aspx?NewsAreaId=22

6. Require tackling some
UNIQUE TECHNOLOGY CHALLENGES

7. Scaling traditional websites
Bob’s data
Bob

10. Scaling Facebook: Interconnected data
Bob

11. Scaling Facebook: Interconnected data
Bob Brian

12. Scaling Facebook: Interconnected data
Felicia Bob Brian

13. News Feed
950 million unique home pages

14. Actor ID - Object ID - Story type Multifeed
Actor ID - Object ID - Story type Multifeed
Stories for up to 5000 friends in milliseconds

15. Memcache
TAO (Custom Ca
1 Billion operations per second

16. while supporting growth … ?
Unified Mobile
Sites
Questions
800M
New2011
Profile
Messages
2010
Groups
2010
2010
Mobile Event
2010
</>
Places
2010
Photos Update
2010
Social Plugins
Open 2010
Graph
2010
The Stream
2009
Translations
2008
Sign Up
Platform launch
New Apps New Apps NewsFeed 2007
February 2004 2006
2004/2005
2004 2011

17. Pottery Challenge
Team 1: Make a PERFECT Pot Team 2: Make 20 pots

18. Scale
Move Fast
Photo by Eole: at http://www.flickr.com/photos/eole/2193801804// and used
under Creative Commons license

19. Move Fast
Moving fast does not mean poor quality
We want a high ship rate
Invest in removing friction that slows us down

20. Empower Engineers
Starting on day ONE
Follow your passion – pick your team
Push any time you want to
…

21. Commits per Month
1/1/2006 1/1/2007 1/1/2008 1/1/2009 1/1/2010 1/1/2011 1/1/2012

22. Move Fast Scale Big Be Bold
and build things with min resources and innovate
OMG!

23. Move Fast Scale Big Be Bold
How can Infrastructure support these goals?
q Pre-empt issues before they hit production
q Know immediately when things go wrong
q Know what to do when things go wrong
and build things with min resources and innovate
= LOTS OF INSTRUMENTATION, TOOLS and AUTOMATION

24. Perflab (aka Difflab)
• Performance test every commit
• Spot regressions before deploy

25. Perflab (aka Difflab)
• Also tracks slow drift regressions
• Helps us push thousands of revs per week

26. Gatekeeper
Simple code
but powerful check
if!
(gk_check(‘secret_project’,
$user) {
!
render_cool_feature();}
else {
!
render_normal_feature();}!
• Many options for precise
targeting
• 500M+ gatekeeper checks
performed every second

27. Rigorous Test Coverage

28. Assigning ownership to failures

29. Canary Tier and Delta view

30. Move Fast Scale Big Be Bold
q Pre-empt issues before they hit production
q Know immediately when things go wrong
q Know what to do when things go wrong
and build things with min resources and innovate

31. *Lots* of Instrumentation = Fire Hose of Data

32. Claspin
• High-density heatmap viewer for large services
• Find needles in a haystack - drilldown quickly

34. tasks sevmanager logview testconsole
differential wirehog domino groups
hipal hsh hud kobold
ods opsfeed scuba serf

35. Move Fast Scale Big Be Bold
q Pre-empt issues before they hit production
q Know immediately when things go wrong
q Know what to do when things go wrong
and build things with min resources and innovate

36. Scuba: a tool for diving into an emerald sea of data

37. Motivation

38. Requirements for data exploration
Need Don’t Need
ü Speed ⤫ Replication
ü Real-time data ⤫ Transactions
ü Ad-hoc filtering and grouping ⤫ Long retention
ü AVG, SUM, COUNT, histograms percentiles ⤫ Table joins
⤫ Unique keys
⤫ Full map-reduce

39. Hive (hadoop)
• “Unlimited” storage/CPU
• Full-featured Query
Language
• Numerous tools and
Frameworks
But Slow!

40. MySQL
Works, but …

41. Scuba: Data Model
Few pre-defined types and operations
“Data Sets”
- No upfront schema declaration
- Stored In memory
- Sorted by Time stamp

42. Scuba Data Types: Integers
VLQ encoded array of 64bit integers (single
char*)
O(1) lookup
Usage
Aggregate on these (SUM, AVG, etc...)
Filter (==, =, =)
“Which pages on the site have an average
wall time in the last hour 2 seconds”

43. Scuba Data Types: Normals
Strings mapped to ints, Stored as array of ints
size: 4 bytes
String Normalization
// char* = int32
'home.php', ’dc1', 'a2', 'en_US' = 32, 14, 3, 289
Usage
Group By value
Filter (==, !=, in set, not in set) “Top 10 countries that have the slowest pages today”

44. Scuba Data Types: Denorm
Array of plain ol' char*
To be used ONLY for unique identifiers that
would not benefit from normalization
size: 8 + strlen + 1 bytes
Usage
None other than displaying the value
Cannot filter or group by these. No native
regex support.

45. Scuba Data Types: Tagsets
A set of normals. Stored as bit vector
size: 8 + 2 + ceil(I / 8) bytes where I is the max
index represented
Usage
Filter (has all tags, has some tags, has none
of these tags)
Bit Set
'timeline', 'mercury', 'titan_drafts' = 0, 5, 14
“Is there a difference in cpu usage across users in test group A vs test group B”

46. Storage
1 aggregator / box
8+ leaves / box
Hundreds of boxes
Data stored over Terabytes of RAM!

47. Leaves
Multiple per machine ( #cores / N )
Only queried by the aggregator on the local
machine
Persist all write traffic to disk (compressed).
Replay all writes on startup
Store all samples efficiently in memory
Leaves are independent; No shared state

48. Aggregation
Queries distributed, and aggregated as a
binary tree
(For now, there is no sorting of results. All
aggregation operations must be
commutative and associative.)

49. Operations
4 functions:
visit, summarize, combine, and finalize
Also a Hive-SQL like query language
interface

50. Querying Front-end

51. tasks sevmanager logview testconsole
differential wirehog domino groups
hipal hsh hud kobold
ods opsfeed scuba serf

52. Move Fast Scale Big Be Bold
and build things with min resources and innovate