LeanOps Presentation DTO SOlutions

1. Open Source Tools and
Operations Methodologies
for Unlocking the Business
Value of the Cloud
Saturday, April 23, 2011

2. • Open Source Tools
• Operations Methodologies
• Business Value of Cloud
Saturday, April 23, 2011

3. Buying or Building a Cloud
Does Not Make Your
Business Agile!
Saturday, April 23, 2011

4. Buying or Building a Cloud
Does Not Make Your
Business Agile!
Building a Cloud is Easy
Managing a Cloud is Not!
Saturday, April 23, 2011

5. Agenda
• Introductions
• Devops
• Devops Stories
• A LeanOps Success Story
• How You Can Deliver LeanOps
Saturday, April 23, 2011

6. Company Background
• Founded in 2004 from:
• eTrade, NASA, Fidelity, Schwab
• Canonical, Opscode
• Agile IT and Operations Team
• Over 100 person-years and 6 books
• Financials, telco, transpiration, etc.
• Headquartered in San Mateo CA
• Offices in Atlanta, and San Francisco
Saturday, April 23, 2011

7. Our Customers
Saturday, April 23, 2011

8. John Willis
john@dtosolutions.com
twitter.com/botchagalupe
Saturday, April 23, 2011

9. Operations:
The Elephant in the Room
Once we move to software as a
service, everything we thought we
knew about competitive advantage
has to be rethought.
Tim O'Reilly - Operations:The New Secret Sauce
Saturday, April 23, 2011

10. XaaS
Saturday, April 23, 2011

11. XaaS
"anything as a service" or "everything as a
service." The acronym refers to an
increasing number of services that are
delivered over the Internet rather than
provided locally or on-site.
Saturday, April 23, 2011

12. The New Face of a
Rock Star
John Allspaw - VP of Technical Operations at Etsy
Saturday, April 23, 2011

13. Devops is a cultural and
professional movement!
Saturday, April 23, 2011

14. Devops is About Velocity
Saturday, April 23, 2011

15. Devops is About Velocity
Velocity of Innovation
Saturday, April 23, 2011

16. LeanOps
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Saturday, April 23, 2011

17. LeanOps
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Saturday, April 23, 2011

18. LeanOps
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Saturday, April 23, 2011

19. Devops
LeanOps
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Saturday, April 23, 2011

20. Devops
LeanOps
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Saturday, April 23, 2011

21. Devops
LeanOps
Agile IT
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Saturday, April 23, 2011

22. Devops
LeanOps
Agile IT
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Saturday, April 23, 2011

23. Devops
LeanOps
Agile IT
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Lean Startup
Saturday, April 23, 2011

24. Devops
LeanOps
Agile IT
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Lean Startup
Saturday, April 23, 2011

25. Devops
LeanOps
Agile IT
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Enterprise Lean Startup
Lean Startup
Saturday, April 23, 2011

26. Devops
LeanOps
Agile IT
Is a commodity based software
infrastructure that focuses on eliminating
waste and reducing the time between
market assumptions and customer
delivery.
Enterprise Lean Startup
Lean Startup
Saturday, April 23, 2011

27. Devops Lifecycle
Shorten
ah ha! Business Process ka ching!
Dev Ops
Remove
Saturday, April 23, 2011

28. Devops Lifecycle
Compressed SDLC
Shorten
ah ha! Business Process ka ching!
Dev Ops
Remove
Saturday, April 23, 2011

29. Typical Ops Resources
Saturday, April 23, 2011

30. Typical Ops Resources
LeanOps
Saturday, April 23, 2011

31. Typical Ops Resources
LeanOps
Saturday, April 23, 2011

32. Typical Ops Resources
LeanOps
Saturday, April 23, 2011

33. LeanOps Loosely
Coupled Toolchain
Saturday, April 23, 2011

34. LeanOps Staff Allocation to
Value Generation
Measuring how your team's time is allocated
Develpment Build Testing Deployment
Before
Build Deployment
Develpment Testing
After
Saturday, April 23, 2011

35. LeanOps Necessities
Saturday, April 23, 2011

36. LeanOps Necessities
Saturday, April 23, 2011

37. LeanOps Necessities
Fully Automated
Infrastructure
Saturday, April 23, 2011

38. LeanOps Necessities
Fully Automated
Infrastructure
Test Driven
Development
Saturday, April 23, 2011

39. LeanOps Necessities
Fully Automated
Infrastructure
Test Driven
Development
Continuous
Deployment
Saturday, April 23, 2011

40. LeanOps Necessities
Fully Automated
Infrastructure
Test Driven
Development
Continuous
Deployment
Immune
System
Saturday, April 23, 2011

41. LeanOps Necessities
Fully Automated
Infrastructure
Test Driven
Development
Continuous
Deployment
Immune
System
Agile
Culture
Saturday, April 23, 2011

42. LeanOps Ingredients
Saturday, April 23, 2011

43. LeanOps Ingredients
Saturday, April 23, 2011

44. DevOps Examples
Saturday, April 23, 2011

45. • Model Driven Configuration
• Automated Deployment
• Continuous Delivery
• Built Their Own Private Cloud
• Self Service Infrastructure
• SVN, Hudson, Puppet
• Nagios, JMX, Cassandra
Saturday, April 23, 2011

46. • Agile Infrastructure
• Every Two Weeks They Re-factor
Apps and Infrastructure
• Chef
Saturday, April 23, 2011

47. • Built a Self Service SaaS Cloud
• Chef, Puppet, Amazon, Azure, Splunk,
Zookeeper, RabbitMQ
• Cloudkick
Saturday, April 23, 2011

48. • First Push to Production on Interview
• 50 to 100 Production Deploys per day
• Puppet, Hudson, Zookeeper, Nagios,
Esper
• Behavior Driven Monitoring
• Immune System Deploys
Saturday, April 23, 2011

49. LeanOps Case Study
Saturday, April 23, 2011

50. “Very” Large Game Company
Started Out Small With a Cloud
Saturday, April 23, 2011

51. “Very” Large Game Company
Started Out Small With a Cloud
Dev
apache/php
memcached
mysql
Saturday, April 23, 2011

52. “Very” Large Game Company
Started Out Small With a Cloud
Dev
apache/php
memcached AWS s3
mysql
Rightscale
Saturday, April 23, 2011

53. “Very” Large Game Company
Started Out Small With a Cloud
Dev
s3put
ssh
apache/php
memcached AWS s3
mysql
Rightscale
Saturday, April 23, 2011

54. “Very” Large Game Company
Started Out Small With a Cloud
Dev winning.c
om
s3put
ssh
apache/php
memcached AWS s3
mysql
Rightscale
Saturday, April 23, 2011

55. “Very” Large Game Company
Very Fast Growth
apache/php
memcached
AWS s3 mysql
Rightscale
winning.com
Saturday, April 23, 2011

56. “Very” Large Game Company
Very Fast Growth
apache/php
memcached
AWS s3 mysql
Rightscale
winning.com
apache/php
memcached
mysql
AWS s3 apache/php
AWS s3 memcached
mysql
Rightscale
tigerblo Rightscale
od.com ck. com
s3put
ssh warlo s3put
Saturday, April 23, 2011

57. “Very” Large Game Company
Very Fast Growth
par ty, apache/php
memcached
3rd & are AWS s3 mysql
mid dlew
Rightscale
yuM winning.com
pUppEt
apache/php
memcached
mysql
AWS s3 apache/php
AWS s3 memcached
mysql
Rightscale
tigerblo Rightscale
od.com ck. com
s3put
ssh warlo s3put
Saturday, April 23, 2011

58. victims of success
COMPLAINTS
BEGAN TO MOUNT
Saturday, April 23, 2011

59. victims of success
COMPLAINTS
BEGAN TO MOUNT
SOFWARE
DOESN'T WORK
RIGHT !
Saturday, April 23, 2011

60. victims of success
COMPLAINTS
BEGAN TO MOUNT
SOFWARE
DOESN'T WORK
RIGHT !
MY BOXES GOT
CLOBBERED !
Saturday, April 23, 2011

61. victims of success
COMPLAINTS
BEGAN TO MOUNT
SOFWARE
DOESN'T WORK
RIGHT !
MY BOXES GOT
CLOBBERED !
GIVE ME MY
ENVIRONMENT!
Saturday, April 23, 2011

62. victims of success
COMPLAINTS
BEGAN TO MOUNT
ANOTHER DEAD
SOFWARE MACHINE ?
DOESN'T WORK
RIGHT !
MY BOXES GOT
CLOBBERED !
GIVE ME MY
ENVIRONMENT!
Saturday, April 23, 2011

63. started asking questions
Saturday, April 23, 2011

64. started asking questions
people ?
Saturday, April 23, 2011

65. started asking questions
people ?
process?
Saturday, April 23, 2011

66. started asking questions
people ?
process?
tools ?
Saturday, April 23, 2011

67. “Very” Large Game Company
Before LeanOps Implementation
NOC server
templates
TAR
RPM restarts
TAR
TAR
code,
EC2
content pupp
deploys et RS
studio RS deployment
devs deployment
S3
yum
repo
Provision-time builds
reconfigs
TAR EXE
TAR AUTO
server
TOOLS
code templates
TAR
RPM
middleware
devs platform CONTROL PROVISION RELEASE
system
admins
Saturday, April 23, 2011

68. “Very” Large Game Company
After LeanOps Implementation
process apache/php
messaging, memcached
queuing mysql
app middleware 3rd party
yum yum
winning.c rundeck
om
code, "D
content, te" e plo
metas o mo y"
"pr job &
jo "Re
jenkins yum b leas
e"
builds
dev prod
Saturday, April 23, 2011

69. “Very” Large Game Company
Infrastructure as SDLC
SVN
hud agent cfg
yum pup
son repo pet
packages
active users run packages
directory deck
commands
nag
splunk
ios
resource model
sys cfg
right instance create
scale
events
new node Node a log data
CONTROL PROVISION RELEASE
Saturday, April 23, 2011

70. Ops part of the architecture
rundeck
yum puppet
operations middleware tools
web[server array] memcached[server array] mysql
new Rightscale deployment
Saturday, April 23, 2011

71. How it is done
• Discovery process through workshop
Discover sessions and individual meetings
• Formal decision modeling to align
strategies to business goals
Align • Cost-benefit analysis
• Validate against reference process
architecture and Open-Source
application and system provisioning
Act
tool-chain (process vs platform
rationalization)
Saturday, April 23, 2011

72. Program Organization
Saturday, April 23, 2011

73. Program Organization
Assessment engineers
Program Management
Saturday, April 23, 2011

74. Program Organization
SVPs
Individual Interviews
& IT Architecture, Software Engineering, Software
Working groups QA, Systems Engineering
Assessment engineers
Program Management
Saturday, April 23, 2011

75. Program Organization
Advisory Various VPs
Committee
SVPs
Individual Interviews
& IT Architecture, Software Engineering, Software
Working groups QA, Systems Engineering
Assessment engineers
Program Management
Saturday, April 23, 2011

76. Program Organization
CIO
Sponsor
Advisory Various VPs
Committee
SVPs
Individual Interviews
& IT Architecture, Software Engineering, Software
Working groups QA, Systems Engineering
Assessment engineers
Program Management
Saturday, April 23, 2011

77. Priority
Saturday, April 23, 2011

78. Example Analysis
Systemic Issues
Bottleneck •8 group sessions
Dislocation
Inconsistency
•11 individual interviews
Instability
Opacity •133 problems identified
Rigidity
•41 candidate strategies
Traceability
Business Issues
0 3.75 7.50 11.25 15.00
Business Scale
Life-Cycle Phases
Control & Auth
Planning
Failure Rate
Requirement
Operational Throughput
Design
Resource Util
Development
Build 0 7.5 15.0 22.5 30.0
Testing
Release Consensus Raw
Delivery
0 5 10 15 20
Saturday, April 23, 2011

79. Example Strategies
Saturday, April 23, 2011

80. Example Prioritization/
Budget Proposal
Saturday, April 23, 2011

81. Devops Workshop
‣ Two Day Workshop
‣ Devops and LeanOps
Theory
‣ Devops Modeling
‣ Devops Team Build
‣ LeanOps Team Build
http://www.flickr.com/photos/asten/2159525309/sizes/l/
Saturday, April 23, 2011

82. dtosolutions.com
John Willis
john@dtosolutions
Saturday, April 23, 2011