FutureStack16 NYC - Aug 9, 2016 Nic Benders, Chief Architect, New Relic

1. BUILDING A SYSTEM
THAT NEVER STOPS
Nic Benders
Chief Architect, New Relic

2. BUILDING A SYSTEM
THAT NEVER STOPS
GROWING

3. BUILDING A SYSTEM
THAT NEVER STOPS
LEARNING

4. BUILDING A SYSTEM
THAT NEVER STOPS
CHANGING

5. ©2008-16 New Relic, Inc. All rights reserved.
@nicbenders
http://nicbenders.com/presentations/scaling-a-system/

7. IN 1 MINUTE
requestsaccepts
over 21M
stores
over
analytic
events200M
aggregates
over 800M metrics
50Bqueries
over
data
points
▪

8. different
services
contains
over
250
maintained/
built by
25+ engineering
teams
▪
4.3more
than
SSD
storage
PETABYTES

9. ©2008-16 New Relic, Inc. All rights reserved.
8LESSONS FROM OUR DATA
PIPELINE

10. ©2008-16 New Relic, Inc. All rights reserved.
Collector
New Relic
App Agents

11. Confidential ©2008-15 New Relic, Inc. All rights reserved.
▪ Double-click to edit

12. ©2008-16 New Relic, Inc. All rights reserved.
Collector
New Relic
App Agents

13. ©2008-16 New Relic, Inc. All rights reserved.
Collector
New Relic
App Agents
Browser R.U.M.
Beacon
New Relic
Browser Agents

14. ©2008-16 New Relic, Inc. All rights reserved.
Collector
New Relic
App Agents
Browser R.U.M.
Beacon
New Relic
Browser Agents
Mobile Proxy
New Relic
Mobile Agents

15. ©2008-16 New Relic, Inc. All rights reserved.
New Relic
App Agents
New Relic
Browser Agents
New Relic
Mobile Agents
Zzz
Collector
Browser R.U.M.
Beacon
Mobile Proxy

16. Confidential ©2008-15 New Relic, Inc. All rights reserved.

17. ©2008-16 New Relic, Inc. All rights reserved.
NOTHING
Lasts Forever
Lesson 1

18. ©2008-16 New Relic, Inc. All rights reserved.
Lesson 2
Create a culture where
CHANGE IS OK

19. Confidential ©2008-15 New Relic, Inc. All rights reserved.

20. Confidential ©2008-15 New Relic, Inc. All rights reserved.

21. ©2008-16 New Relic, Inc. All rights reserved.
EXPERIMENTS
Don’t always work
Lesson 3

22. ©2008-16 New Relic, Inc. All rights reserved.
Lesson 4
SYNCHRONOUS
is going to be a problem

23. ©2008-16 New Relic, Inc. All rights reserved.
Kafka
Kafka
Browser Data
Consumer
Browser Data
Router
New Relic
App Agents
New Relic
Browser Agents
Collector

24. ©2008-16 New Relic, Inc. All rights reserved.
Kafka
Kafka
Browser Data
Consumer
New Relic
App Agents
Collector
Zzz
Browser Data
Router
New Relic
Browser Agents

25. Confidential ©2008-15 New Relic, Inc. All rights reserved.

26. ©2008-16 New Relic, Inc. All rights reserved.
Lesson 5
Master the
ROLLOUT

27. ©2008-16 New Relic, Inc. All rights reserved.
Rollout
Techniques
Kafka
Load Balancer Cluster
Kafka
Browser Data
Consumer
Browser Data
Router
Browser
Data
Consumer
V2
Browser
R.U.M.
Beacon
%- %+
New Relic Browser Agents
Collector
Old
New

28. ©2008-16 New Relic, Inc. All rights reserved.
Lesson 6
It’s the
OPERATIONS
stupid

29. Confidential ©2008-15 New Relic, Inc. All rights reserved.
Monitoring Tooling Operations Runbooks

30. Confidential ©2008-15 New Relic, Inc. All rights reserved.

31. ©2008-16 New Relic, Inc. All rights reserved.
Lesson 7
Use the right
WORKLOAD
DISTRIBUTION

32. ©2008-16 New Relic, Inc. All rights reserved.
Workload Distribution
D
D
D
D
D
C
B
B
B
B
A
A
A
A
A
C BD A B DDD
Active Management Random
C B BBD DAA D C
B BD
B C
AA D
D
C
C
A B

33. ©2008-16 New Relic, Inc. All rights reserved.
D
D
D
D
D
C
B
B
B
B
A
A
A
A
A
C BD A
B DDD
C B BBD DAA D C
B BD
B C
AA D
D
C
C
A B
Active Management
Random
Use Both!

34. ©2008-16 New Relic, Inc. All rights reserved.
Lesson 8
Technology enables
CULTURE

35. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166

36. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166
�����

37. 35

38. Confidential ©2008-15 New Relic, Inc. All rights reserved.

39. Confidential ©2008-15 New Relic, Inc. All rights reserved.

40. Confidential ©2008-15 New Relic, Inc. All rights reserved.

41. Confidential ©2008-15 New Relic, Inc. All rights reserved.
@nicbenders
http://nicbenders.com/presentations/scaling-a-system/