How to monitor unknown third party code? One of the hardest challenges we face running Clever Cloud, apart from the impressive scale we face with hundreds of new applications per week, is the monitoring of unknown tech stacks. The first goal of rebuilding the monitoring platform was to accommodate the immutable infrastructure pattern that generates lots of ephemeral hosts every minute. The traditional approach is to focus on VMs or hosts, not applications. We needed to shift this into an approach of auto-discovery of metrics to monitor, allowing third party code to publish new items. This talk explains our journey in building Clever Cloud Metrics stack, heavily based on Warp10 (Kafka/Hadoop/Storm based) to deliver developer efficiency and trustability to our clients applications.

1. Monitoring the unknown,
1000*100 series a day
Quentin ADAM - @waxzce

2. Who am I ?
Quentin ADAM from the Clever Cloud
@waxzce on twitter – github- soundcloud – instagram ….

3. My day to day work :
, the IT automation company

4. Keep your apps online. made with node.js,
scala, java, ruby, php, python, go…
Cloud & on premise

5. And learn a lot of things
about your code, apps,
and good/bad design…

6. Give back to the
community
NEVER GONNA
LET YOU DOWN
clever-cloud.com

7. Automate infrastructure

8. What are we talking about?

9. Immutable infrastructure
FR : https://www.youtube.com/watch?v=WrZCbgQsPVU
EN : https://www.infoq.com/presentations/server-cloud

10. Ephemeral instances

11. Pet
!=
Cattle

12. Monitoring service
!=
Monitoring host

13. Monitoring the unknow
Part 1

14. Which tech stack
peoples will use?

15. Business metrics
Application metrics
middleware app
system metrics
• logins
• leads
• sales
• clicks
• ...
• SQL sessions
• User session opened
• Connections pool
• …
• GC
• JVM management
• request per seconds
• Threads
• Jobs scheduling
• …
• cpu
• load
• storage
• RAM
• ...

16. Monitoring the unknow
Part 2

17. There is no ready solution…

18. Let’s build something

19. Data needs

20. Store measure in time

21. Analyze it
Most of the time using a micro batching approach

22. Traditional needs not required
• No ACID
• No Transaction
• Just a few full text
• Eventual consistency if enough

23. SCALE

24. Time Series DB

25. Series and Labels
• Instance.xxxxxx.ram.free = 2078
• {
• Application: myAppId
• Platform: Java
• Os: Linux
• …
• }

26. Watch out cardinality

27. Time series DB is an hot noSQL topic

28. Influx DB

29. Prometeus

30. Timescale DB
Based on

31. Open TSDB

32. Cassandra + Akka persistence/stream

33. Warp10

34. Standalone
Java fat jar

35. Distributed
• Consensus using Zookeepers
• Store and analysis on Hadoop
• Event bus with Kafka
• Several roles in a single JAR, all stateless

36. Warp10 roles
• Ingress
• Directory
• Store
• Egress
• Webcall
• Plasma
• Runner
• Fetcher

37. Ingress
🔑
Kafka
Hadoop Hbase
cluster
Store
Dictionary

38. So, We can now store data

39. How to send some

40. Agent: Telegraf

41. Agent goals

42. Collect data
• System
• Many connector to specific softs
• Open protocols…
• Push or Pull?

43. JMX

44. Statsd Simple UDP protocol

45. Prometeus

46. Buffer to send batched data
~100 metric point / minutes

47. Token based security

48. More tokens: JWT and Macaroon
EN : https://www.youtube.com/watch?v=SO5G0Gsm5-w
FR : https://www.youtube.com/watch?v=mvKeCsxGZhE

49. ✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
❔ Analyze data
❔ Trigger data
❔ Display data

50. WarpScript, distributed on hadoop

51. µBatching WarpScriptµBatching WarpScriptµBatching WarpScript

52. Build a new series

53. Listen on Kafka or Plasma for triggers

54. Immutable DataBase

55. ✅ Send data from all VMs
✅ Send data from others infrastructures parts
✅ Store data
✅ Analyze data
✅ Trigger data
❔ Display data

56. Quantum
Tool
+
PolymerJS lib

57. Quantum
Tool
+
PolymerJS lib

58. Which data expose?

59. Templating the data

60. Templating the data

61. Templating the data

64. Give access
to the API

65. Enhance developer work, happiness and
productivity

66. Give access to metrics and logs

67. Thank you
find me on twitter
@waxzce
Gift coupon for
clever-cloud.com:
bigDataConf2017