According to Google, SRE is what you get when you treat operations as if it’s a software problem. In this video, I briefly explain various practical alerting considerations and views from Google. Youtube channel here: https://youtu.be/EgpCw15fIK8

1. SRE Demystified
Practical Alerting
ganesh@ganeshniyer.com
ganesh.vigneswara@gmail.com,
http://ganeshniyer.com
Dr Ganesh Neelakanta Iyer

2. SRE
•
2https://image.slidesharecdn.com/devopssreatgooglescale-190121123035/95/devops-sre-at-google-scale-30-638.jpg?cb=1548074257

3. Monitoring
• Monitoring a very large system is challenging for a couple of
reasons:
• The sheer number of components being analyzed
• The need to maintain a reasonably low maintenance burden on the
engineers responsible for the system
• A large system should be designed to aggregate signals and
prune outliers
• We need monitoring systems that allow us to alert for high-
level service objectives, but retain the granularity to inspect
individual components as needed
3
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

4. Borgmon monitoring at Google
• White-box monitoring
• Instead of executing custom scripts to detect system failures,
Borgmon relies on a common data exposition format
• This enables mass data collection with low overheads and avoids
the costs of subprocess execution and network connection setup
• The data is used both for rendering charts and creating
alerts, which are accomplished using simple arithmetic
• To facilitate mass collection, the metrics format had to be
standardized
4
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

5. Instrumentation of applications
• Adding mapped variables for example
• An example map-valued variable
• Showing 25 HTTP 200 responses and 12 HTTP 500s:
• http_responses map:code 200:25 404:0 500:12
5
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

6. Storage in the Time-Series Arena
• A service is typically made up of many binaries running as
many tasks, on many machines, in many clusters
• Borgmon needs to keep all that data organized, while allowing
flexible querying and slicing of that data
• Borgmon stores all the data in an in-memory database,
regularly checkpointed to disk
• The data points have the form (timestamp, value), and are
stored in chronological lists called time-series, and each time-
series is named by a unique set of labels, of the
form name=value.
6
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

7. Storage in the Time-Series Arena
7
A time-series for errors labeled by the original host each was collected from
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

8. Labels and Vectors
• Time-series are stored as sequences of numbers and
timestamps, which are referred to as vectors
• Like vectors in linear algebra, these vectors are slices and cross-sections of
the multidimensional matrix of data points in the arena
• The name of a time-series is a labelset, because it’s implemented
as a set of labels expressed as key=value pairs. One of these
labels is the variable name itself, the key that appears on the varz
page
8https://landing.google.com/sre/sre-book/chapters/practical-alerting/

9. Labels and Vectors
• Example variable expression
{var=http_requests,job=webserver,instance=host0:80,service=web,zone=us-west}
9
Label Value
var The name of the variable
job The name given to the type of server being monitored
service A loosely defined collection of jobs that provide a service to users,
either internal or external
zone Location of the Borgmon that performed the collection of this
variable
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

10. Rule Evaluation
• The Borgmon program code, also known as Borgmon
rules, consists of simple algebraic expressions that
compute time-series from other time-series
• Rules run in a parallel threadpool where possible, but are
dependent on ordering when using previously defined
rules as input
• Aggregation is the cornerstone of rule evaluation in a
distributed environment
10
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

11. Example Rule
11
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

12. Example Alert Rule
• Creates an alert when the error ratio over 10 minutes exceeds
1% and the total number of errors exceeds 1 per second
12
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

13. Maintaining the configuration
• Borgmon configuration separates the definition of the rules
from the targets being monitored
• Borgmon also supports language templates
• The first class simply codifies the emergent schema of
variables exported from a given library of code
• Such templates exist for the HTTP server library, memory
allocation, the storage client library
• The second class templates are to manage the aggregation
of data from a single-server task to the global service footprint
13
https://landing.google.com/sre/sre-book/chapters/practical-alerting/

14. References
14

15. Dr Ganesh Neelakanta Iyer
ganesh@ganeshniyer.com
ganesh.vigneswara@gmail.com