Given at the Dublin Google I/O Extended 2018

1. Measure Everything you Can
Business-related metrics with Google App Engine
@rbanffy

2. First things first
Two (very) different things:
● The Standard environment
○ The one Google App Engine started with
○ Originally supported only Python (they had Guido), then grew to Java, Go, PHP
○ Very inexpensive (to the point of "almost free")
○ Great for students
○ Imposes some (very sane and educational) rules
● The Flexible environment
○ Straightforward
○ Few restrictions

3. What does it make sense to measure?
CPU counters
Revenue
Profit
Strategies Host OSGuest OSPlatformApplicationYour processes

4. What can we easily measure from here?
CPU counters
Revenue
Profit
Strategies Host OSGuest OSPlatformApplicationYour processes
Maybe a
little bit
here
Off-limits for us
(Google SRE-land)Most likely
not visible
from here

5. What can you measure out of the box
Google App Engine offers the Stackdriver tools
● Handy visualization of performance bottlenecks in your
applications
● Nice APIs (although Python support lacking on some)
● Centralized searchable logging
● A killer profiler for Go, Node, or Java.

6. Your response times (from the trace list)
● Why?
○ Experience quality matters
○ Users are easily distracted
● You may also be under strict SLAs if someone depends on your APIs
○ Your money depends on how well you perform
○ Poor performance may cost money in fines
○ Can be caused by underlying API calls to external services
ApplicationYour processes

7. Your response times (from the trace list)
● You can specify which
requests to show
● Requests show as dots, y is
duration, x is time of request
● Useful for observing
clustering and getting a rough
idea of experience quality
ApplicationYour processes

8. Your response times (from the trace list)
● Traces can give you an idea of what each
part of your process costs
● You can name your own traces (move our
arrow into the blue region)
● You can use a Zipkin client to do it (if the
native APIs are not what you need)
● Here we see external APIs represent the
bulk of the time (which is money) spent
here
ApplicationYour processes

9. Your own metrics (and how your web app can help)
● You will want to measure your own things
● Some handy examples of what can be retrieved close to real-time
○ Average ticket price
○ Actions/goals per time
○ Time spent in purchase, products seen until purchase/no purchase
○ A/B tests
○ Going further, feature flags
ApplicationYour processes
This is a bit harder

10. Your own metrics (and how your web app can help)
● How you can go about this
● Stackdriver traces deals well with in-request traces
○ Standard environment is already traced
○ API allows creation of spans (you add the ones that make sense for your app)
○ Client libraries for the Flexible environment: Python, Ruby, JavaScript, Go, PHP, Java, etc
○ You can use it outside Google App Engine
● Stackdriver logging can deal with cross-request traces
ApplicationYour processes

11. Is there any other way?
● I use Datadog in combination with Stackdriver
● You can roll out your own
○ I won't really talk about this
○ This is flexible enough to warrant a multi-day workshop
ApplicationYour processes

12. How I use Datadog
● Has good App Engine support
● Allows you to create cross-request events
● Allows decorators for your data points (so you can add relevant data)
○ This is how I do cross-request
● Has some very nice visualizations
○ I love their heatmaps
ApplicationYour processes
Arrow went a little bit to the left now

13. How I use Datadog
ApplicationYour processes
● You need to add the library
○ Brain-dead-easy on Flexible
environment
○ Tricky in the Standard (at least for
Python it was)
● Once you are there, it's very easy
to start collecting

14. Visualizations
● The base dashboard
○ Every tool has it
○ Not really that interesting (mostly the
obvious measurements that fit any app)
○ These are not the metrics you are
looking for
ApplicationYour processes

16. The metrics we are looking for
● We want things that help us
make decisions
○ A/B tests
○ Changes that affect user
experience
○ Changes that affect our costs
(this one)
ApplicationYour processes
This is what it looks like when you get
10x more data than you should
The new normal

17. The metrics we are looking for
● What we need for that
○ Events (to help us navigate in
time - deployments, tests,
promos)
○ Metrics (with tags)
○ Tags (to sort through the data
- think A and B in A/B tests)
ApplicationYour processes

18. What if I got it backwards?
What if what you really want is to use App Engine to capture and store your
metrics?
● The Datastore is excellent for that (but I don't think any tools can visualize from
it directly)
● You'll need to build your HTTP API (sorry, no inbound UDP)
● You can massage the inbound data as much as you want
● You can use the queues to store your data (InfluxDB, Elasticsearch, RDBMS)
from costing too much and use the app as a nice queue server with built-in
transformation
● From there you can visualize with your favorite tool (mine is Grafana)

19. Key takeaways
● You can get a lot of data that's relevant to your business out of a Google App
Engine application
○ Stackdriver is built-in (and can track a lot of things)
○ You can use others for all others (we talked about Datadog)
● You can visualize all sorts of business-relevant data using these tools
○ Some visualizations make the data pop-out
■
● You can even do it completely differently and use an App Engine application to
process your data originated elsewhere before storing it elsewhere

20. A word of caution
1. Measurements have impact
2. HTTP APIs even more so (which is
why we use UDP in other settings,
usually for server/platform/app
metrics)
3. Instrument wisely - measure what you
need, avoid measuring what you don't
4. Drop costly measurements you don't
use frequently - or subsample those

21. ?

22. Thank you
@rbanffy