Weitere ähnliche Inhalte Ähnlich wie StackStorm DevOps Automation Webinar (20) Kürzlich hochgeladen (20) 2. Vision à Specific Patterns!
2!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
We’ve been chatting with you –
what have we learned?!
• Let’s talk operational patterns. !
• A little on monitoring. !
• A lot on operations automation.!
!
We want to learn more today! !
!
What are we building?!
!
!
3. Market!
“So$ware
is
ea,ng
everything.”
{
developed
and
operated
in
the
DevOps
way
}
!
StackStorm
does
DevOps
opera,ons
automa,on
!
3!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
4. StackStorm Presenters!
Patrick Hoolboom, Unicorn Stormer!
• 7+ years building and running high pace DevOps
environment (2 years at Cloudmark)!
• Puppet, Chef, Nagios, NewRelic, Logstash and more!
• “Open source your process” – lets share operational
templates!
!
!
Evan Powell, Co-Founder & CEO!
• 15+ years in infrastructure software!
• Founding CEO of Clarus Systems, acquired by OPNT!
• Founding CEO of Nexenta Systems!
• Defined and led OpenStorage and SDS space!
• Led Nexenta to 5k+ customers, 280+ employees, $350 million+
of partner sales, $75 million in funding !
! 4!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
8. 8!
!
7/21/14
©
2014
StackStorm,
Inc.
ConfidenRal
Day
1:
Pre4y
well
sorted
Day
2:
Pre4y
much
chaos
• But
if
infrastructure
is
immutable
then
maybe
we
never
have
Day
2?
10. Patterns: Monitoring!
Monitoring as a service:!
• Check in your checks and monitoring is free!
• Transparency => shame => compliance!
Ongoing challenges:!
• Pager fatigue!
• Automate all the thresholds !
• Dependencies => deduplicating alerts!
• Drive => Host => Service => Application =>
User!
10!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
11. Patterns: Monitoring!
11!
!
Nagios
–
Sensu
–
APM
Zabbix
Graphite
StatsD
Humans
via
ChatOps
PAGER
DUTY
Auto
correcRons:
Nagios
or
other
Declare
event,
ask
for
help
-‐>>Event
pipeline
-‐>>
-‐>>Event
pipeline
-‐>>
Rieman
/
StackTach
12. Patterns: Monitoring!
Infer the threshold!
• Anomaly detection!
• Pattern recognition!
Start with a check!
And/or update the checks and thresholds as
you learn more!
Monitoring as a service "
Remediation as a service!
12!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
13. Remediation as a Service #1!
13!
!
Facebook’s
FBAR:
“AutomaRng
the
work
of
hundreds”
h4ps://www.facebook.com/notes/facebook-‐engineering/making-‐facebook-‐self-‐
healing/10150275248698920
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
14. Remediation as a Service #2!
14!
!
Microsoh
AutoPilot:
States:
• Healthy,
Failed,
ProbaRon
RemediaRons
(“Rs”):
• Restart,
Reboot,
Reimage,
RMA
h4p://research.microsoh.com/pubs/64604/osr2007.pdf
“Machine
learning
algorithms
to
analyze
these
data
in
order
to
understand
how
to
improve
the
policy
selngs,
with
the
ulRmate
goal
of
automaRng
many
of
the
current
manual
policies.
“
15. Patterns: Tools – Monitoring?!
The #1 tool we have seen is NewRelic!
• Roughly 75% of the discussions!
• Amazing because it does not fit all the
requirements!
The #2 tool we have seen is Splunk!
• Still seeing them more than LogStash,
ElasticSearch although it is close!
!
15!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
16. Patterns: Tools – Configuration!
Puppet vs. Chef!
• Ops vs. developers?!
Ansible and Salt growing in importance!
• Points of differentiation and also overlap!
Best practices include:!
• Separate source of truth !
• Code reviews (duh?)!
16!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
17. “State in the Repo” vs.
Single Source of Truth!
17!
!
• Provides visibility into
the system!
• Fights configuration drift!
• Reduces convergence
time of changes!
• Tied in to monitoring
and configuration
systems!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
18. 18!
!
Lack
of
trust
Hard
to
test
Simple
script
+
simple
script
+
simple
script
+
Zounds
What’s
Wrong
with
Day
2
AutomaCon?
Monitoring
My
script
vs.
your
script
IntegraCon
sprawl
19. Automation: Don’t Forget #1!
Simplicity
scales.
19!
!
Best.
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
BEST
PRACTICES
• Have
X
states
(three
seems
logical)
for
a
monitored
system
• Have
Y
remediaRons
• Document
both
• Automate
the
known
mappings
• Atomic
acRons
• Workflows
or
acRons
of
acRons
to
Re
them
together
Lessons
learned
from
many
operators
large
and
small
20. Automation: Don’t Forget #2:!
Failure
detectors
must
be
able
to
dis7nguish
between
the
symptoms
of
failure
and
overloading,
otherwise
overloaded
computers
may
be
marked
as
failed
and
removed
from
service,
amplifying
the
problem
and
triggering
a
cascade
of
failures
that
disables
the
enRre
applicaRon.
20!
!
h4p://research.microsoh.com/pubs/64604/osr2007.pdf
BEST
PRACTICE
Check
a
threshold
with
each
step
so
that
run
away
automaRons
cannot
occur.
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
21. Automation: Don’t Forget #3!
Humans
are
good
at
pa4ern
matching
–
if
they
perceive
the
pa4erns.
• Always
provide
context.
When
a
threshold
is
violated
that
requires
that
humans
get
back
involved
you
are
at
risk
of
pulng
the
human
in
a
difficult
place.
21!
!
BEST
PRACTICE
When
the
system
needs
a
human,
provide
the
human
with
context.
• Automate
the
delivery
of
that
context
for
the
sake
of
consistency
• Don’t
allow
the
automa,on
to
just
be
addi,onal
complexity
on
top
of
complexity
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
22. Automation: Don’t Forget #4!
AutomaRon
frees
up
humans
to
a4ack
technical
debt,
refactor
for
growth
and
so
forth.
However,
the
implicit
knowledge
formed
by
eliminated
manual
acRon
itself
may
be
lost.
22!
!
BEST
PRACTICE
Refer
to
the
prior
Don’t
forget
points
including
simplicity
of
state
analysis
and
simplicity
of
allowed
remediaRons.
• Also
–
allow
manual
invoking
of
automaRons
• Make
automaRons
human
readable
• And
otherwise
transparent
• Give
the
automaRon
an
edgy
personality
(huh?)
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
23. ChatOps is Brilliant!
Add
some
code
to
IRC
or
other
chat
to
do
stuff,
and
you
have
bot
executed
code.
Add
to
that
an
enRre
automaRon
library
–
plus
a
personality
–
and
the
requirement
that
ALL
changes
happen
through
chat
and
DevOps
happens.
23!
!
BEST
PRACTICES
Get
to
know
Hubot.
And
stay
tuned
for
StackStorm’s
take
on
this
approach.
ChatOps
benefits:
• Dev/Ops
interface
and
Newby
/
Old
pro
interface
• Context
for
humans
• Trust
in
automaRon
h4p://puppetlabs.com/blog/really-‐building-‐data-‐driven-‐infrastructure
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
26. Before!
26!
!
Events
AcCons
Events
AcCons
Scripts
Scripts
AcCons
Scripts
Events
#BadAuto!
• Who did what to what and how did it go?!
• Fragments of automation and configuration management!
• No opportunity for learning!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
27. StackStorm!
27!
!
Events
AcCons
#ShareAndLearn!
• Scripts -> automations!
• Workflow – stitch stactions!
• Close the loop – map events to
automations!
• Total audit & transparency!
• Still DevOps friendly !
• All automations are code in repositories!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
28. Technical Details!
• Centralized view (federated) into automations!
• Audit trail and RBAC access controls!
• Infrastructure as code!
• Full APIs, configurations as code!
• Collaboration inherent!
• Abstraction of automations for sharing!
• Two way integration into collaboration including ChatOps!
• Bi directional close relationship with monitoring!
• StackStorm helps users determine where to put
problem analysis – StackTach and others promising!
• Facility for self learning!
• Simple today headed towards controller synthesis
approaches!
28!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
29. Example Use Cases!
• OpenStack and hybrid management!
• Facilitated troubleshooting!
• Over time – automated resolution!
• Multi-stage deployments!
• Develop, stage, deploy for example!
• Response to security events!
• Post-hoc as opposed to real time!
29!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
30. Business Model!
OpenSource!
• Apache licenses!
Free!
• Free forever edition (community)!
Enterprise!
• On premise or hosted – all deployments now
are on premise!
!
30!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
32. What Can You Do?!
Register as a beta participant!
• Are you doing some OpenStack?!
• Can you provide feedback at least 2x per month?!
• Automating something already?!
• Interested in ChatOps? Not required….!
!
Once we GA – grab free version!
• Contribute to community w/ integrations and
automations!
• OpenSource and share your operations patterns!
32!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
33. Questions!
To respond, please unmute your line by pressing *6!
!
CI, CD, Day 2 operations?!
• Where are you in the continuum?!
ChatOps – do you use it?!
!
Have you had automations run amok?!
!
DevOps team? Title? Silo?!
!
33!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
34. Summary!
World is changing rapidly!
!
We’ve learned a lot!
!
Safe, composable automations with
circuit breakers and other controls
should help!
!
34!
!
7/21/14!
© 2014 StackStorm, Inc.!
Confidential!
