The Evolving Design Patterns of Puppet Enterprise - Jonathan Spinks, Sourced Group & John Painter, Sourced Group

1. THE EVOLVING DESIGN PATTERNS OF PUPPET ENTERPRISE

2. John Painter Jon Spinks
Solutions Architect Solutions Architect
www.linkedin.com/in/painterj www.linkedin.com/in/jonspinks
sourcedgroup.com

3. Sourced Group architect and
deploy a globally distributed
multi-master puppet solution
for a large investment bank
February 2011

4. CUSTOMER REQUIREMENTS
• Provide a single point of control for a globally
distributed IT environment
• Reduce overall configuration drift in the existing
fleet
• Provide a change control process to manage code
updates

5. DESIGN PATTERN 1
Global Multi Master

6. CENTRAL MGMT. SYDNEY
Puppeteer
LDAP
ENC
Puppet
Master
Puppet
Console
Puppet
Master
NEW YORK
LDAP
ENC
Puppet
Master
Puppet
Console
Puppet
Master
LDAP ENC
Master
HONG KONG
LDAP
ENC
Puppet
Master
Puppet
Console
Puppet
Master
LONDON
LDAP
ENC
Puppet
Master
Puppet
Console
Puppet
Master

7. Follow-the-Sun Code Release
ASIA: 6pm - 8am Europe: 6pm - 8am USA: 6pm - 8am
Client Client
Puppet Master
Client Client
Puppet Master
Client Client
Puppet Master
Global risk management, simplified releases

8. AUDITING GOVERNANCE SIMPLIFIED
CONTROL
Control and report on the point of truth

9. KEY TAKEAWAYS
• LDAP is a great distributed ENC
• Puppet Reports can be large en masse
• A Puppet Master can used to manage other
Puppet Masters
• Version control is critical as a basis for a change
control process

10. Sourced Group architect and
deploy a globally distributed
multi-master puppet solution
for a large investment bank
February 2011

11. Sourced are the first to onboard
an Asia Pacific
telecommunications company
onto AWS, powered by Puppet
Enterprise
February 2012

12. CUSTOMER REQUIREMENTS
• Provide a single point of control for instance
provisioning
• Provide infrastructure orchestration
• Maintain portability of applications to the
traditional data centre

13. EVERYTHING IS
PROGRAMATIC
STANDARDISATION OF
INFRASTRUCTURE

14. DESIGN PATTERN 2
Puppet Managing AWS Compute

15. Puppet CloudPack
EC2 only support circa 2012

16. Public IP Public IP
Compute Compute
Compute Compute
Public IP Public IP
EC2
Customer
IP Whitelisting

17. Customer Compute Compute
Virtual Private Cloud!
Private IP Space
Compute Compute
VPN / Direct
Connect
Public IP
VPC

18. Bootstrap Agent
Classify Node
Application Deployed
Create Instance

19. KEY TAKEAWAYS
• Puppet Masters can be used for OS instance
delivery
• CloudPack supports EC2 and VPC

20. Sourced are the first to onboard
an Asia Pacific
telecommunications company
onto AWS, powered by Puppet
Enterprise
February 2012

21. “Using Puppet Enterprise
Edition as Heterogeneous
Cloud Glue”
PuppetConf 2012

22. CLOUDPACK UPLIFT
• Uplifted the existing Puppet CloudPack
to support AWS VPC
• Left Puppet Enterprise in the middle of
the provisioning and lifecycle
management
• Supported Multi-cloud provisioning
• http://www.youtube.com/watch?
v=mwiBjJZWraA

23. “Using Puppet Enterprise
Edition as Heterogeneous
Cloud Glue”
PuppetConf 2012

24. Sourced implement a
masterless Puppet Enterprise
Edition solution for a financial
services organisation
March 2013

25. CUSTOMER REQUIREMENTS
• Harness transient compute while enforcing
compliance
• Provide OS and application state transparency

26. STANDARDISATION AND
CONSOLIDATION

27. DESIGN PATTERN 3
Masterless Puppet with AWS Integration

28. CLOUD COMPUTING IS
FUNDAMENTALLY DIFFERENT

29. V1.0 V1.0
myapp.com

30. V1.0 V1.0
myapp.com
V2.0 V2.0

31. myapp.com
V1.0 V1.0 V2.0 V2.0

32. WEBAPP V. 2
TEST
BAKE
LAUNCH
TEARDOWN
WEBAPP V. 1
TEST
BAKE
LAUNCH
TEARDOWN
WEBAPP DEV -V. 1
TEST
BAKE
LAUNCH
TEARDOWN
UAT
Production

33. UAT
Production
APP V1.1
APP V1.0
V1.1
Features
V2.0
V1.1
V2.0

34. Infrastructure, deployment, and configuration
have standardised
!
It was not just servers that were transient, it was
entire application stacks
!
Cloud computing has significantly changed the
application lifecycle

35. “18 months really sucks”
–NIGEL KERSTEN - PUPPETCAMP SYDNEY 2014

36. “18 months really sucks”
- EVERY SINGLE PRODUCT MANAGER. EVER.

37. Weekly/Monthly -> Hourly

38. Load Time
$
$
$ $
$
$

39. PROBLEM SPACE
• Puppet Masters were designed to manage long lived
OS instances
• Transient compute doesn’t really need a dashboard
• Puppet Console accuracy was impacted by stale
instance data
• Requirement to simplify the Auto Scale process

40. MASTERLESS COMPONENTS
• CloudFormation for orchestration
• AWS EC2 Instance Tags provide node definition
• Facter is extended to understand AWS Instance
Metadata
• Manifests and modules are bundled and centrally
managed
• Centralised logging to provide state data

41. Version Control
CI/CD
AWS S3
Instance Instance Instance
Instance
Centralised Logging

42. Configuration Managent
Application Configuration
Governance and Compliance

43. Configuration Managent
Application Configuration
Governance and Compliance

44. KEY TAKEAWAYS
• Cloud Computing is fundamentally different
• Harnessing transient compute is extremely
valuable
• Losing the Puppet Master reduces overall
visibility when considering long lived and
transient instances as a whole

45. DESIGN PATTERN 4
Puppet Master and Transient Compute

46. DATA BUILDS
TRANSPARENCY
Getting the Puppet Master Back in the Loop

47. AWS COMPUTE TYPES
Transient
Compute
Stateful!
Compute

48. PUPPET AWS AUTOSCALING BROKER

49. Autoscale BROKER Simple Queue Service (SQS)
Instance Instance
SNS Topic
Simple Notification Service (SNS)
Autoscaling Group
Instance Instance

50. KEY TAKEAWAYS
• Using a Puppet Master as a point of control for all
compute builds transparency
• The Puppet Master framework is very extensible
• PuppetDB is very valuable
• Leverage AWS Auto Scale orchestration features

51. A LOOK FORWARD
Heterogeneous Puppet Governance

52. Application
Runtime
Operating System
Hardware

53. Application
Runtime
Operating System
Hardware

54. Rather than making the OS a good candidate for
the application, we are now tailoring the
applications to be a good citizen of a standardised
compute platform.

55. Pivotal CF
Docker
Azure Pack
OpenShift
Examples of higher-level application delivery frameworks that can run on
AWS

56. THE FRAMEWORKS STILL
NEED TO BE GOVERNED

57. HYBRID CLOUD
On Premise Managed
Stateful
Compute
Transient
Compute

58. GOVERNANCE
GOVERNANCE
GOVERNANCE
GOVERNANCE
GOVERNANCE
GOVERNANCE

59. In an ideal world with true vendor arbitrage we
would be able to deliver an application to AWS,
OpenShift on Rackspace and Docker running on
managed VMware using only CI/CD tooling

60. Puppet + Autoscale Broker + Global Distribution + Multi Cloud
APP APP APP APP
AWS - Sydney
Puppet Master
APP DELIVERY
FRAMEWORK
APP DELIVERY
FRAMEWORK
APP DELIVERY
FRAMEWORK
AWS - USA
Puppet Master
APP DELIVERY
FRAMEWORK
APP DELIVERY
FRAMEWORK
APP DELIVERY
FRAMEWORK
<CLOUD> - Singapore
Puppet Master
APP DELIVERY
FRAMEWORK
APP DELIVERY
FRAMEWORK
APP DELIVERY
FRAMEWORK
Runtime
Your Datacenter
Puppet Master
APP DELIVERY
FRAMEWORK
APP DELIVERY
FRAMEWORK
APP DELIVERY
FRAMEWORK

61. KEY TAKEAWAYS
• Through the adoption of PaaS the bar of
commoditisation continues to rise
• Governance and compliance are still critical
• Application delivery performance continues to
increase

62. SOURCEDGROUP.COM