Monitorama 2016

1. @petecheslock
Everything @obfuscurity Taught
Me About Monitoring
Monitorama 2016

2. @petecheslock
“Just use Graphite”
- Jason Dixon

3. @petecheslock
Thank you

4. @petecheslock
Pete Cheslock
@petecheslock
Tech Ops & Support at Threat Stack

5. @petecheslock
get.threatstack.com/monitorama16

6. @petecheslock

8. constantly

9. @petecheslock
It is an AMAZING time to be
working in technology

10. @petecheslock
The “Good ‘ole Days”
kinda sucked

11. @petecheslock

12. @petecheslock

13. @petecheslock

15. @petecheslock

16. @petecheslock

17. @petecheslock

18. @petecheslock
I knew how to cd and ls

19. @petecheslock
# chown -Rf bob:bob files /
Whoops

20. @petecheslock
Should've used bcfg2?
<Path name="/home/file" mode="0644" group="bob" owner="bob">
<ACL type="access" scope="user" user="bob" perms="rwx"/>
</Path>
Won’t exist for 3+ more years…

21. @petecheslock

22. @petecheslock
ColdFusion

25. @petecheslock
* * * * * /scripts/mrtg-netmon

26. @petecheslock

27. @petecheslock

28. @petecheslock

29. @petecheslock
**PROBLEM Host Alert:
Everything Is FUCKED**

30. @petecheslock

31. @petecheslock

32. @petecheslock
Jen: Yo, are you ready to support ~10
beta customers on a completely
unautomated, unmonitored env?
Pete: New Phone, who dis?
Jen: 5 9s of availability.

33. @petecheslock
The Right Way™

34. @petecheslock

35. @petecheslock

36. @petecheslock

37. @petecheslock

38. @petecheslock
http://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-linux-system-calls/

39. @petecheslock

40. @petecheslock
Premature
optimization is the
root of all evil

41. @petecheslock

42. @petecheslock

43. @petecheslock

44. @petecheslock
collectd -> write_http ->
(60 Second)

45. @petecheslock

46. @petecheslock

47. @petecheslock

48. @petecheslock

50. @petecheslock
collectd -> write_http ->
(60 Second)
statsd plugin
app1 app2 appN

51. @petecheslock

52. @petecheslock
• Hosted TSDB is super useful and “just works”
• The opportunity cost of fully replacing it was
too high.

53. @petecheslock
• Use the same tools already deployed
• Devs wrote libraries to send metrics to statsd
• Don’t want to “redeploy the world”
• Limited time to invest

54. @petecheslock
• Use the same tools already deployed
• Devs wrote libraries to send metrics to statsd
• Don’t want to “redeploy the world”
• Limited time to invest

55. @petecheslock
collectd -> write_http ->
(60 Second)
collectd -> write_graphite ->
(10 Second)
statsd plugin
statsd plugin

56. @petecheslock

57. @petecheslock
https://github.com/obfuscurity/
synthesize
https://github.com/hw-cookbooks/
graphite

58. @petecheslock
https://github.com/obfuscurity/
synthesize
https://github.com/hw-cookbooks/
graphite

59. @petecheslock

60. @petecheslock
Want your Devs to Ops?
Build Consumable Services

61. @petecheslock

62. @petecheslock

63. @petecheslock

65. @petecheslock
collectd -> write_graphite ->
(10 Second)
statsd_plugin
app1 app2 appN

66. @petecheslock

67. A lot of people must have suddenly
just stopped caring

68. @petecheslock
Community Matters
Relationships Matter

69. @petecheslock
You are (probably)
not Google

70. @petecheslock
and that is OK
(Solve YOUR Problems)

71. get.threatstack.com/monitorama16
Thank You!