@rossbruniges and I talked with our fellow Mozilla Foundation engineers and development teams about getting the data for building a data driven operation using statsd, graphite, geckoboard, google analytics, and newrelic.

2. Why measure anything?
We want to continually improve, to further the
MoFo mission.
Humans, and also organizations, can't know if
they can or should do something differently
unless they get feedback or data to prompt the
question:
"How can we/I do better at this?"

3. Absent of any data, everything looks fine

4. First Requirement of Improvement
Realize there is something that needs
improvement
__________________

5. Knowing the truth is often scary and uncomfortable,
but at least you know when to run!

6. Fast, Confident, and Awesome
If an org, team, or individual:
-Knows, definitively, how they are doing now
-Knows, definitively, how they are doing after a change
-Knows, definitively, what variables changed when

7. Then that org, team, or individual:
-will be much faster to determine success vs. failure, making it possible to
resolve faster.
-will inevitably grow and learn, further and faster
-generates more trust...with users, contributors, org, team
and maybe most importantly...
-We'll be more confident to try more things, knowing we can quickly detect and
resolve.
-Thus, as we will try more things, it will lead more often to success.

8. Had Good Metrics Indicating Dogs Were Tame...
Confident.

9. What kind of stats are we talking?
Not just CPU and App Performance!
metrics << app / product
metrics << projects / teams
metrics << users / customers
metrics << contributors
metrics << operations
metrics << performance
metrics << developers
metrics << support

11. Proposed Layers of Metrics
1. Organization Layer - "How is Mozilla Foundation doing
at its mission?"
2. Team Layer - "How well is this team working together
and accomplishing goals? "
3. Application/Service Layer - "How are things with X
application or service?"
4. Operations Layer - "How are we doing, logistically?"

12. Organizational Metrics Examples
How many visits to all of our sites this year vs. last year?
How many people have donated to MoFo, and how much?
How many times were we or any of our sites mentioned on
social media and the news?
How many new "webmakers" did we bring into our
ecosystem across all applications and teams?

13. Team Metrics Examples
How many hours were worked/PTO'd on the
team?
How many projects is that team involved in this
week?
How many collaborator / volunteer / staff lines
of code / features were added?
How many milestones did the team meet this
month?

14. Application Metrics Examples
What was the average response time for popcorn.
webmaker.org this week?
How many videos were edited this week?
How many new videos were created this week?
What was the # of page visits for popcorn on average by
hour?
What was the ratio of errors to successes for popcorn's key
transactions this week?

15. Operations Metrics Examples
How much disk is currently being used?
How much memory and CPU utilization do we have in
production?
How many gigabytes of logs were generated this week?
What was the slowest loading page per application this
week?
How much did our infrastructure cost?

16. What tools will we be using?
Primary Tools
● Newrelic
● StatsD/Graphite
● Google Analytics
● Geckoboard
Secondary Tools (Not covering today)
● AWS, Pingdom, Opsview

17. New Relic (Server Monitoring)

18. New Relic
Real time data on your server and app
performance
Examples:
● what code is running slowly or erroring?
● how long are end users waiting before they
can interact with our content?
● How many database writes per minute?
● End user performance monitoring
● Which deploy caused a problem?

19. Google Analytics

20. Google Analytics
Numbers on who is using your app, how and
when they are using it
● what content is popular with who?
● which parts of the site are causing people to
leave?
● where are our users coming from?
● what things have they done while on our
sites?
● Can track campaigns, goals (ie, made a
popcorn video), conversion rates

21. statsd/graphite
Multiple stats, one stream

22. statsd/graphite
statsd - "Collects data by increment counters and timers,
but does not show, sort, or average(etc....)"
graphite - Shows the data in a presentable way, allowing
you to line up multiple metrics and perform calculations,
such as sort, avg, std deviation
(Examples of a few things we might count/time)
● someone made a project, remixed a project
● someone has been issued a badge
● someone has decided to delete their profile

23. StatsD examples (counting new vids in popcorn)
(note the data structure, env.app.what)
Time something (PHP example)
$start = microtime(true);
postvideo();
StatsD::timing("production.butter.postvideo.response.200", (microtime
(true) - $start) * 1000);
Count something (PHP)
StatsD::increment("production.butter.postvideo.response.200");

24. Geckoboard

25. Geckoboard
A friendly view into all of our high level stats.
● this is where Mark/Ryan will come in
● this is where we highlight the individual
metrics we get judges against
● one stop-shop over all MoFo/webmaker
properties
● encouraged to make public

26. Metrics Flow

27. Things to Beware Of
● Presenting data out of context for audience
● Instrumentation of anything to New Relic,
Google Analytics, Geckoboard should go
over ssl
● Do not use New Relic auto instrumentation
to capture any params in the database
● Gameable Metrics

28. Go fourth and metric!
See what you can do with these client libs:
● statsd:
○ https://django-statsd.readthedocs.org/en/latest/
○ https://github.com/sivy/node-statsd
● geckoboard:
○ http://pypi.python.org/pypi/django-geckoboard
○ https://github.com/danjenkins/node-geckoboard-
push
○ https://github.com/Hebo/node-geckoboard
● newrelic custom instrumentation:
○ https://newrelic.com/docs/instrumentation/custom-
metric-collection

29. Links/References
Good read with a further intro for statsd/graphite
How we will setup Opsview to watch StatsD/Graphite
Google Analytics: Setting up goals and tracking conversions
Easy setup instructions for statd/graphite server(if you want to test locally)