(Presented by Datadog) Gaining visibility into an application stack’s performance is necessary to understand how the stack is running and to configure alerts effectively. Instrumenting each component in the stack to produce metrics provides this insight. In an environment that scales automatically, hosts are being automatically added, removed, and reassigned. Using an automated methodology for instrumentation in these environments can improve results and save you time. This session includes a live demo component to show auto-instrumentation of hosts, graphing, and alerting on metrics.

1. Instrumenting your Application Stack in a
Dynamically Scaling Environment
Mike Fiedler, Datadog
November 15, 2013

2. The Past
• Server uptime
• Intertwined applications
• SPOFs

3. The Present
• SOA
• Scale or Fail
• Data-driven decisions

4. Modern application stack

5. Failure is inevitable

6. Design to tolerate failure

7. business/service
Services
API-based
Users
• application
• OS
• instances
• Amazon S3, S*S
• Amazon RDS,
Amazon Redshift
• etc.
• RUM
Things that produce metrics

8. The Desire

9. CPU comparison (hypervisor)

10. CPU comparison (Agent)

11. My Stack
• HAProxy
• Apache HTTP
• Bottle.py
– Custom python webapp
• MongoDB
• Amazon EC2
• Chef
• Datadog
• Users

12. Tracking database operations

13. Moving up the stack…

14. Even higher…

15. Using metadata for more flavor

16. Replace an instance, keep the metrics

17. Enter…

18. Aggregating numbers

19. Comparing numbers

20. My App
from bottle import route
import pymongo
import json
db = pymongo.Connection('mongodb://localhost').test
@route('/insert/:name')
def insert(name):
doc = {'name': name}
db.words.update(doc, {"$inc":{"count": 1}}, upsert=True)
return json.dumps(doc, default=default)

21. Add 2 lines and you win!
from bottle import route
import pymongo
import json
from statsd import statsd
db = pymongo.Connection('mongodb://localhost').test
@route('/insert/:name')
@statsd.increment('fullstack.insert')
def insert(name):
doc = {'name': name}
db.words.update(doc, {"$inc":{"count": 1}}, upsert=True)
return json.dumps(doc, default=default)

22. Profit!

23. Correlation

24. Don’t be caught unaware

26. Question Time!

27. We are sincerely eager to hear
your feedback on this
presentation and on re:Invent.
Please fill out an evaluation form
when you have a chance.