Joshua Buss and Matthew Kemp of Bright Tag talk on "Automating Life in the Cloud" at PuppetCamp Chicago '12. Learn More: www.puppetlabs.com

1. Automa'ng
Life
in
the
Cloud
Joshua
Buss
&
Ma+hew
Kemp

2. Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
2

3. There
are
no
Silver
Bullets
(but
if
you
find
one
let
us
know)
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
3

4. Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
4

5. Virtual
Machines
Designing
for
the
Cloud
Have
to
go
with
what
cloud
provider
offers.
Not
always
ideal
for
every
workload.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
5

6. Scalability
Designing
for
the
Cloud
Focus
on
scaling
applica5ons
horizontally.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
6

7. Service
Oriented
Architecture
Applica5on
Design
Wikipedia
Defini5on:
SOA
as
an
architecture
relies
on
service-­‐orienta3on
as
its
fundamental
design
principle.
If
a
service
presents
a
simple
interface
that
abstracts
away
its
underlying
complexity,
users
can
access
independent
services
without
knowledge
of
the
service's
plaBorm
implementa3on.
Layman’s
terms:
A
complex
system
is
broken
into
simple
components
that
are
able
to
interact
with
each
other
(and
possibly
outside
sources).
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
7

8. Case
Study:
Services
Applica5on
Design
tagserve
datahub
database
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
8

9. Service
Division
of
Labor
Applica5on
Design
When
should
you
split
services
up?
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
9

10. Backwards
Compa5bility
Applica5on
Design
Changes
need
to
be
allowed,
but
compa5bility
needs
to
be
maintained.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
10

11. Asynchronous
vs
Synchronous
Applica5on
Design
Dependent
on
needs
of
business,
SLAs
and
technology.
Can
introduce
more
complexity
and
moving
pieces.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
11

12. Failures
Happen
Cloud
Design
Some5mes
the
failures
are
outside
your
control.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
12

13. Design
for
Failure
Cloud
Design
Keep
failures
self
contained.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
13

14. Case
Study:
Redundancy
Applica5on
Design
Run
a
full
stack
in
each
region.
tagserve
datahub
tagserve
datahub
database
database
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
14

15. Inter-­‐Region
Communica5on
Applica5on
Design
Need
some
data
available
in
all
regions,
but
keep
inter-­‐region
communica5on
to
a
minimum.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
15

16. Case
Study:
Cassandra
Applica5on
Design
East West
cassandra04
cassandra01
cassandra04
cassandra01
[192-­‐255]
[0-­‐63]
[193-­‐0]
[1-­‐64]
cassandra03
cassandra02
cassandra03
cassandra02
[128-­‐191]
[64-­‐127]
[129-­‐192]
[65-­‐128]
Key hashes to 157
Writes go here
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
16

17. Run5me
Controls
Applica5on
Design
Provide
mul5ple
modes
of
opera5on.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
17

18. Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
18

19. Smooth
Code
Pushes
Deployment
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
19

20. Mirror
Environment
Cutover
Deployment
Easy
migra5ons
and
upgrade
path.
Can
be
more
expensive.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
20

21. Rolling
Deploy
Deployment
More
complicated
migra5ons
and
upgrades.
Longer
deploy
window.
Usually
cheaper.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
21

22. Case
Study:
Fabric
Rolling
Deploy
Deployment
for
region
in
regions:
for
app
in
apps:
for
server
in
region:
if
app
on
server:
maintenance
app
scp
new
code
to
<d_tag>
dir
symlink
app/current
to
app/<d_tag>
restart
app
wait
for
healthy
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
22

23. Fabric
vs
Puppet
Deployment
Fabric
is
push,
puppet
is
pull.
Businesses
don't
move
as
fast
as
infrastructure
changes,
but
configs
have
to
stay
up
to
date
all
the
5me.
(/etc/hosts)
(systempoller.py)
(mashed_potatoes.env)
(dataserver.war)
puppet
=====================================
fabric
(real-­‐time
up-­‐to-­‐date)
(moderately
up-­‐to-­‐date)
(weekly)
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
23

24. Consistency
>
*
Puppet
Need
a
rock-­‐solid
founda5on
to
deploy
onto.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
24

25. Single
Puppet
Master
Puppet
Set
environment
per-­‐instance:
/etc/puppet/
puppet.conf
Symlink
/etc/puppet/environments/
on
master
to
various
folders
with
read/write
access
by
our
main
user.
$
cd
/etc/puppet/environments
$
sudo
ln
–s
~/src/puppet/prod_stable
$
sudo
ln
–s
~/src/puppet/stage_stable
$
sudo
ln
–s
~/src/puppet/dev_test
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
25

26. Source
Controlled
Puppet
Configs
Puppet
Each
environment
has
its
own
branch.
Make
a
new
branch
for
every
new
feature.
Merge
into
a
test
branch
to
test.
Merge
into
stable.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
26

27. Confidential, Property of BrightTag, Inc. Not to be disclosed, reproduced, or distributed without
BrightTag, Inc.'s prior written authorization. © 2011 BrightTag, Inc. All Rights Reserved.

28. What
is
Zerg?
Management
+
=
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
28

29. How
to
Reach
Servers?
Management
DNS
vs
/etc/hosts
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
29

30. Case
Study:
/etc/hosts
Management
$
curl
–s
'http://zerg/etchosts/us-­‐west-­‐1'
#
The
following
lines
are
desirable
for
IPv6
capable
hosts"
::1
ip6-­‐localhost
ip6-­‐loopback
fe00::0
ip6-­‐localnet
ff00::0
ip6-­‐mcastprefix
ff02::1
ip6-­‐allnodes
ff02::2
ip6-­‐allrouters
ff02::3
ip6-­‐allhosts
10.0.0.10
server01
#
External:
123.123.123.123
10.0.0.11
server02
#
External:
123.123.123.124
10.0.0.12
server03
#
External:
123.123.123.125
10.0.0.13
server04
#
External:
123.123.123.126
10.0.0.14
server05
#
External:
123.123.123.127
10.0.0.15
server06
#
External:
123.123.123.128
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
30

31. Instance
Crea5on
Management
Amazon
API
is
easy
...
but
just
crea5ng
the
instance
is
10%
of
the
work
Gesng
the
right
sotware.
Surviving
internal
API
failures.
Staying
hos5ng
provider
agnos5c.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
31

32. Case
Study:
Instance
Crea5on
Management
ROLE_MAPPING
=
{
"stage"
:
{
"supercloud"
:
{
"ops"
:
["zerg"],
"awesome"
:
["awesome",
"haproxy_awesome"],
"shabang"
:
["shabang",
"mashed_potatoes",
"haproxy_shabang"],
"whistles"
:
["gowhooo",
"shabang",
"thehardproblem",
"redis"],
"data"
:
["dataserver",
"dataleaf"],
"nosql"
:
["itshards",
"devnull"],
"lb"
:
["haproxy"],
"redis"
:
["redis"],
"graph"
:
["graphite",
"tattle"]
},
"prod"
:
{
"evenmoresupercloud"
:
{
...
}
}
}
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
32

33. Case
Study:
Instance
Configura5on
Management
/etc/instanceinfo
filled
out
with
values
based
on
Zerg's
defini5ons:
§ hostname
§ environment
§ region
§ roles
§ where's
my
local
zerg
server?
§ where's
my
local
graphite
server?
§ cloud
meta
info
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
33

34. Loadbalancer
Configura5on
Management
Update
5ming
tricky
to
get
right.
Too
important
to
leave
completely
autonomous.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
34

35. HAProxy
Configura5on
Workflow
Management
Zerg
(genera5on)
Git
(ops)
Script
(human)
Large
changes
to
templates
Git
(puppet)
(human)
Server
Server
Server
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
35

36. Applica5on
Descrip5ons
Management
APP_DEFS
:
{
"zerg"
:
{"type":
"http",
"healthcheck":
{"port":
19999,
"resource":
"/zerghealth"}},
"awesome"
:
{"type":
"http",
"healthcheck":
{"port":
20000,
"resource":
"/ahc"}
},
"haproxy_awesome"
:
{"type":
"http",
"healthcheck":
{"port":
20001,
"resource":
"/"}},
"shabang"
:
{"type":
"http",
"healthcheck":
{"port":
20002,
"resource":
"/"}},
"mashed_potatoes"
:
{"type":
"http",
"healthcheck":
{"port":
20003,
"resource":
"/"}},
"haproxy_shabang"
:
{"type":
"http",
"healthcheck":
{"port":
20004,
"resource":
"/hc"}},
"gowhooo"
:
{"type":
"http",
"healthcheck":
{"port":
20005,
"resource":
"/"}},
"thehardproblem"
:
{"type":
"http",
"healthcheck":
{"port":
20006,
"resource":
"/"}},
"redis"
:
{
"type":
"tcp",
"healthcheck":
{"port":
20007,
"resource":
"/rhc"}},
"dataserver"
:
{
"type":
"http",
"healthcheck":
{"port":
20008,
"resource":
"/"}},
"itshards"
:
{
"type":
"http",
"healthcheck":
{"port":
20009,
"resource":
"/"}},
"devnull"
:
{
"type":
"http",
"healthcheck":
{"port":
200010,
"resource":
"/hc"}},
"graphite"
:
{
"type":
"http",
"healthcheck":
{"port":
80,
"resource":
"/composer"}}
}
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
36

37. Case
Study:
HAProxy
Configura5on
Management
$
curl
-­‐s
'http://zerg/haproxy/<env>/<region>/<service>'
global
log
127.0.0.1
local0
log
127.0.0.1
local1
notice
stats
socket
/tmp/haproxy
blah
blah
defaults
log
global
mode
http
blah
blah
frontend
mashedpotatoes_vip
bind
*:30000
default_backend
data
backend
mashedpotatoes
blah
blah
options
server
shabang01
10.0.0.30:30001
check
server
shabang02
10.0.0.31:30001
check
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
37

38. The
Manual
Step
Management
$
./update_haproxy.sh
<env>
<region>
<service>
**
Git
is
clean
and
in
sync
with
origin..
now
waiting
for
zerg
http
response..
[prod_stable
f4qijo]
[puppetry]
Haproxy
Auto-­‐Commit
for
<env>
<region>
<service>
1
files
changed,
2
insertions(+),
2
deletions(-­‐)
**
Template
pulled
and
committed
**
Here
is
the
diff
from
origin
to
the
new
version:
diff
-­‐-­‐git
a/modules/haproxy/templates/haproxy_<env>_<region>_<service>_cfg.erb
b/modules/haproxy/templates/haproxy_<env>_<region>_<service>_cfg.erb
-­‐-­‐-­‐
a/modules/haproxy/templates/haproxy_prod_us-­‐east-­‐1_tagserve_cfg.erb
+++
b/modules/haproxy/templates/haproxy_prod_us-­‐east-­‐1_tagserve_cfg.erb
-­‐
oldyuckyserver01
-­‐
oldyuckyserver02
+
fastwonderfulnewserver01
+
fastwonderfulnewserver02
**
Do
you
want
to
push
this
change?
(y/n)
y
blah
blah
successful
git
push
message
**
Commit
successfully
pushed
to
origin
**
All
done!
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
38

39. Trust
is
a
Luxury
Best
Prac5ces
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
39

40. Uniform
Environments
Best
Prac5ces
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
40

41. Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
41

42. Why
monitor?
Monitoring
How
do
you
know
what's
going
on?
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
42

43. What
to
monitor?
Monitoring
Iden5fy
metrics
that
act
as
signals.
Add
alerts
ater
every
incident.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
43

44. Data
Collec5on
Monitoring
Need
system
level
metrics
and
applica5on
metrics
to
get
full
picture.
Everything
is
in
a
different
format.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
44

45. Case
Study:
Metric
Polling
at
BrightTag
Monitoring
datahub
datahub
redis
tagserve
redis
tagserve
mpoller
cassandra
haproxy
cassandra
haproxy
mpoller
mpoller
graphite
graphite
carbon
carbon
mpoller
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
45

46. Graphite
Monitoring
Storage
of
historical
metrics
allows
for
trending
and
comparisons.
Aggrega5on
is
performed
on
data
retrieval
via
the
webapp.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
46

47. Branches
and
Leaves
Monitoring
Expose
a
"metrics"
service
per
region.
Enables
a
flexible
topology.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
47

48. Real5me
Numbers
Across
Regions
Monitoring
Requests are farmed out to each metrics service.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
48

49. Visualiza5on
Monitoring
Different
visualiza5ons
tell
you
different
things.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
49

50. Red-­‐Yellow-­‐Green
Monitoring
Dashboards
provide
at-­‐a-­‐glance
high
level
overviews.
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
50

51. Ques5ons?
Confiden5al,
Property
of
BrightTag,
Inc.
Not
to
be
disclosed,
reproduced,
or
distributed
without
BrightTag,
Inc.'s
prior
wri+en
authoriza5on.
©
2011
BrightTag,
Inc.
All
Rights
Reserved.
51