Puppet for Everybody! - Federated and Hierarchical Puppet Enterprise - PuppetConf 2014
•Download as PPTX, PDF•
3 likes•2,068 views
Puppet for Everybody! - Federated and Hierarchical Puppet Enterprise - Chris Bowles, University of Texas at Austin
Recommended
More Related Content
What's hot
What's hot (14)
Similar to Puppet for Everybody! - Federated and Hierarchical Puppet Enterprise - PuppetConf 2014
Similar to Puppet for Everybody! - Federated and Hierarchical Puppet Enterprise - PuppetConf 2014 (20)
More from Puppet
More from Puppet (20)
Recently uploaded
Recently uploaded (20)
Puppet for Everybody! - Federated and Hierarchical Puppet Enterprise - PuppetConf 2014
- 1. Puppet for Everybody! Federated and Hierarchical Puppet Enterprise Chris Bowles, Senior Systems Administrator University of Texas at Austin
- 2. Puppet for Everybody? Absolutely! • Development • Operations • Management source: http://goo.gl/Mjr0dy
- 3. Continuum of Expertise Novice • Puppet Console • Variables Medium • Hiera Expert • Code • Custom Facts • Custom Functions
- 4. UT Puppet Canon • Inclusive • Secure by Default • Federation
- 5. UT Puppet Toolset UT Puppet Community Nested Configs Puppet Enterprise Code/Data Federation
- 6. UT Puppet Community UT Puppet Community Nested Configs Puppet Enterprise Code/Data Federation
- 7. UT Puppet Culture • Module Coding Standards • Module Documentation Standards • Power to the People
- 8. Puppet Console • Classes • Console Groups (role/profile) • Console Variables Configured Server! Module CodeHieraExpert UT Puppet Diagram Novice
- 9. Nested Configs UT Puppet (standards, culture) Nested Configs Puppet Enterprise Code/Data Federation
- 10. Nesting: Roles/Profiles • Wiki server configurationsRoles • Apache configurationsProfiles • Secure by default • standardized • configurable BASE
- 11. Minifigure Metaphor • Default “torso” provided • Configurable: can change the color of the cowl (black or very, very dark grey) • Role/Profile: Can choose the head and arms, cape, etc… From: https://www.flickr.com/photos/spielbrick/8201894577
- 12. Nest all the things! • Groups • Variables • hiera? (yup, more on that later)
- 13. Puppet Console • Nested groups • role/profile • assign classes & variables to nodes Configured Server! Module CodeHieraExpert Roadmap: Console Nesting Novice
- 14. Nested Console Groups source: http://goo.gl/tUdl5U
- 15. Nested Console Groups BASE profile_apache role_wiki wiki-01 secure defaults Apache configs Wiki configs Node-specific configs
- 16. Nesting (from the node POV) Contains Classes/Variables from: Node wiki-01 BASE profile_apache role_wiki
- 17. Don’t forget the Blog! Node-level Roles Profiles Secure Defaults BASE profile_apache role_blog blog-01 blog-02 role_wiki wiki-01 wiki-02 • Configurations come from nested groups • No repetition!
- 18. What’s in a name (prefix)? Role Profile Top BASE profile_apache role_blog role_wiki Puppet Console will display: (alphabetical) •BASE •profile_apache •role_blog •role_wiki
- 19. Console Building Blocks! source: http://goo.gl/CHwab0
- 20. BASE: BASE group
- 24. Puppet Console components • Classes • Variables • Group(s) • Nodes ssh $::ssh_port BASE, Profile_apache, role_wiki wiki-01
- 25. Class Inheritance (immutable) BASE assigns: ssh profile_apache inherits: ssh assigns: apache role_wiki inherits: ssh, apache node Inherits: ssh, apache
- 26. Variable Inheritance (child wins) BASE N/A profile_apache http_port = 80 role_blog N/A blog-01 http_port= 80 role_wiki http_port = 8080 wiki-01 http_port= 8080
- 27. All together now! source: http://goo.gl/K91CJA
- 28. wiki-01 (annotated) Variable overrides from role_wiki group Group membership and source(s) Classes: combined from nested groups
- 29. Puppet Console • Console Groups (role/profile) • Console Variables Configured Server! Module Code HieraExpert Roadmap: Hiera Nesting Novice
- 30. Hiera: for complex variables key: value key2: value2 • Arrays • Hashes source: http://goo.gl/ge45I1 Think backend data mapping
- 31. Nested Groups => Hiera paths BASE N/A profile_apache profile = apache role_wiki role = wiki wiki-01 Inherits: profile,role ./ ./apache/ ./apache/wiki/
- 32. Nesting Hiera w/ group variables Role(s) • ./$profile/$role Profile(s) • ./$profile/ BASE • ./ (no variable) $profile $role Broad to Specific
- 33. Hiera.yaml – specific to broad Specific to Broad --- :backends: - yaml :hierarchy: - '%{profile}/%{role}/common' - '%{profile}/common' - 'common' :logger: console :yaml: :datadir: /etc/puppetlabs/puppet/hieradata
- 34. Hiera.yaml – specific to broad Specific to Broad --- :backends: - yaml :hierarchy: - '%{profile}/%{role}/common' - '%{profile}/common' - 'common' :logger: console :yaml: :datadir: /etc/puppetlabs/puppet/hieradata
- 35. Putting it together "Denslow's Humpty Dumpty 1904" by William Wallace Denslow – Library of Congress [1]. Licensed under Public domain via Wikimedia Commons – http://commons.wikimedia.org/wiki/File:Denslow%27s_Humpty_Dumpty_1904.jpg
- 36. Console => Hiera Nested Console Groups Hiera profile_apache profile = apache Role_wiki role = wiki ./apache/ ./apache/wiki/ wiki-01 node profile= apache, role= wiki BASE none ./ Hiera search order 1. ./apache/wiki/common.yaml 2. ./apache/common.yaml 3. ./common.yaml
- 37. Advanced Hiera Usage • “This data is exactly what I need… almost” • firewall, sudoers • +1 • Check out: – hiera_hash – hiera_array
- 40. Code Federation puppet.conf modulepath= /opt/puppet/modules/base: /opt/puppet/modules/apache: /opt/puppet/modules/wiki: BASE Repo Apache Repo Wiki Repo VCSREPO Separate sources enable role separation via ACLs
- 41. Data Federation • ./hieradata/ = ./common.yaml ./apache/common.yaml ./apache/wiki/common.yaml BASE Repo Apache Repo Wiki Repo VCSREPO Separate ACLs for Hiera data as well
- 42. A peek into the future… source: http://goo.gl/9GwKyQ
- 43. Git Workflow • Instead of this… • 1 git repo / module Core SVN repo (modules) Apache SVN repo (modules) Head (production) branch Non-production branches (created as needed)
- 44. CI/CD • r10k push deployments (faster!) • Puppet Environments defined by code (Puppetfile) • Automated Testing/Deployment Git repos r10k Puppet
- 45. Takeaways • Puppet Enterprise can be: – Inclusive – Secure by Default – Highly Federated • Nurture your Puppet community • Nest your configs!
- 46. Thanks! Any Questions? • Slide deck available from PuppetLabs • UT Puppet Architecturehttps://wikis.utexas.edu/x/OreZAw • Contact information: – Chris Bowles • Email: cbowles@austin.utexas.edu • Twitter: @cbowlesUT Puppet Man, Sulayman Bowles 2014
Editor's Notes
- Intro: brief sketch about UT Austin “Going to give a brief overview about how we enable DevOps for everybody with Puppet Enterprise at the University of Texas.” UT is: very federated Lots of different departments/stakeholders
- Configuration Management is not just for sysadmins any more. People at every level of the organization have a vested interest in keeping their configurations verifiably correct. Development staff: Can use Puppet enterprise as a part of their deployment chain / to verify configurations Operational staff, naturally, use it to apply configurations using the trifecta (package / file / service) Management : Puppet Enterprise model is visible via the Puppet Console, can review reports, etc... However! These are very different use cases And very different levels of engagement with Puppet, the Puppet Enterprise console, and the Puppet RAL (Resource Abstraction Layer)
- The levels of expertise with Puppet in those different groups aligned along a continuum from Novice to Expert, with infinite levels between. We wanted a solution that would work for all of them.
- Puppet Enterprise as the core. Bolted on three different aspects to the stock Puppet Enterprise software (Puppet server, Puppet Console, etc…) UT Puppet Community – Role/Profile Federation - Code/Data Federation –
- UT Puppet Community
- Module Coding Standards Module Documentation Standards People Power (User Groups, Wikis, learning resources…) ----- Meeting Notes (9/18/14 16:13) ----- talk about empowering people. Good place to give. How does this affect the normal person. The scaffolding is there.
- Roles/Profile Federation
- Let’s talk about nested group. Useful for Class and Variable inheritance. You can use nested groups to: Eliminate Configuration Duplication Override variables
- Collapose everuything into this slide (leran to type)
- Here is what that looks like…. What if we want to add a Blogs server? Hierarchical Configurations. Allows for Puppet Enterprise configurations to be completely modeled in Puppet Console node groups and inherited by child node groups and nodes. By providing Puppet as a Service, we empower other sysadmins to take advantage of the power of Puppet without necessarily having to know This: Allows for capturing of configuration data that cannot be automatically obtained (i..e. via facter). The “business logic” is encoded into Puppet Prevents duplication of configurations Allows for Sr Sysadmins to provide standard builds for Jr Sysadmins
- Highlight configurations occur within the highlighted area Hierarchical Configurations. Allows for Puppet Enterprise configurations to be completely modeled in Puppet Console node groups and inherited by child node groups and nodes. By providing Puppet as a Service, we empower other sysadmins to take advantage of the power of Puppet without necessarily having to know This: Allows for capturing of configuration data that cannot be automatically obtained (i..e. via facter). The “business logic” is encoded into Puppet Prevents duplication of configurations Allows for Sr Sysadmins to provide standard builds for Jr Sysadmins ----- Meeting Notes (9/18/14 11:36) ----- Make this a tree/folder
- We found it useful to create a naming scheme for our hierarchical node groups. We add a prefix to each group name that gives the location in the hierarchy. This allows you to designate the hierarchy and find it in the flat list that puppet displays. group_foo = Department groups subgroup_foo = Service groups The generic example is on the left. Just as Puppet can model the configurations necessary for your servers through application of classes, the Puppet Enterprise Console can model your organization via groups. Creating a good hierarchy and placing configurations and variables in the proper level can provide major benefits in standardization and ease of Puppet management. You build your hierarchy by deciding how many groups you need. This is a balancing act that based upon the existing structures in your organization. I think that the rule of three applies well here. Default, Group, subgroup is a pretty good place to start. A specific example is on the right. Let’s say you have many different groups in your organization. One such group provides Database services, called “group_db”. Below that are subgroups that contain the MySQL and Oracle services. This is where we really chase the “infrastructure as code” concept. Ideally, there should be sufficient classes and variables contained within these three levels that you can take a newly provisioned node and attach it to one of the subgroups, run puppet and then you have a fully configured server. With these in place, you have configurations that are reproducible, scalable and can be stored in source control.
- All of our configurations are built from small, modular building blocks: Puppet Classes Variables Groups Nodes Let’s take a closer look at the details.
- The three primary moving pieces are Classes, Variables and Groups. Classes – link the puppet node to the appropriate Puppet class. I.e. Provides the SSH class Variables – Sets top-level variables. Provides parameterization to the classes (if they are configured correctly – more on that later) Groups – Collections of classes and variables. Note that groups can be members of groups. Nodes – The actual server. Where the rubber hits the road. All the classes and variables from the groups are combined here via inheritance rules.
- Classes inheritance is immutable, so that classes can only be added and never taken away. NOTE: this is a good reason to be cautious when creating your hierarchy to ensure that you really want the classes to apply to ALL the nodes contained within the group. More on that…
- Variables are a different beast. The child group or node inherits all of the variables, but in case of a conflict the child value wins. This is useful for two things: Broad default values can be applied at at very high-level group. These will be inherited by the child groups and finally applied to the nodes. At ahy point downstream, these values will show up in either the group or node. This adds visibility and makes it easy to override the variables via copy-pasta. Standards are important, here. More on that shortly. Beyond just All or node paramerization. It allows you to use Puppet Console groups to model the hierarchy within your organization in a way that is both intuitive and fairly straightforward. This is especially useful for variables that are based upon business process and are not easily codeable (i.e. SSH port and/or DNS name).
- Let’s take a look at a practical example of this process.
- A standard API (in the loose sense). This includes all of the standards that we have set for our Puppet Modules, documentation. Puppet Console always sets the variables for the Puppet Modules via top level variables These variables are named in the same way Documentation is standardized With the combination of the right class and variable, the configurations are applied.
- However, not all variables are simple. Some variables need to be expressed in complex data structures like arrays and hashes. They can also be arranged in hierarchical and federated group, just like the Puppet console variables. This involves creating a hierarchy in hiera.conf, as well as more console variables! Array: https://www.flickr.com/photos/ratatatratsy/5394750775/ Hash Found nothing!
- Collapose everuything into this slide (leran to type)
- federated hierarchies are created in hiera in a similar manner to the way that we did it in the console with variables. Instead of nested groups, we use nested folders. The root above is your hiera data directory, usually /etc/puppetlabs/puppet/hieradata (configured in hiera.yaml file) TOP: Common hiera variables (aka keys) go here, usually in a common.yaml file, but totally dependent upon how hiera.yaml is configured Group(s): These map to large divisions of your enterprise/organization. Any way it makes sense for you. Any number of groups. These should map 1-1 with the node groups created earlier, which I will explain next. Subgroup(s): These map to smaller divisions of the groups above. Any number of subgroups or sub-sub-groups. ----- Meeting Notes (9/18/14 16:56) ----- Replace with venn --> directory
- federated hierarchies are created in hiera in a similar manner to the way that we did it in the console with variables. Instead of nested groups, we use nested folders. The root above is your hiera data directory, usually /etc/puppetlabs/puppet/hieradata (configured in hiera.yaml file) TOP: Common hiera variables (aka keys) go here, usually in a common.yaml file, but totally dependent upon how hiera.yaml is configured Group(s): These map to large divisions of your enterprise/organization. Any way it makes sense for you. Any number of groups. These should map 1-1 with the node groups created earlier, which I will explain next. Subgroup(s): These map to smaller divisions of the groups above. Any number of subgroups or sub-sub-groups.
- federated hierarchies are created in hiera in a similar manner to the way that we did it in the console with variables. Instead of nested groups, we use nested folders. The root above is your hiera data directory, usually /etc/puppetlabs/puppet/hieradata (configured in hiera.yaml file) TOP: Common hiera variables (aka keys) go here, usually in a common.yaml file, but totally dependent upon how hiera.yaml is configured Group(s): These map to large divisions of your enterprise/organization. Any way it makes sense for you. Any number of groups. These should map 1-1 with the node groups created earlier, which I will explain next. Subgroup(s): These map to smaller divisions of the groups above. Any number of subgroups or sub-sub-groups.
- mysql-prod node inherits both locations, so hiera will search both locations
- A key reason to have multiple hiera locations Can use hiera_hash function to combine keys from various hiera files Allows for complete configurations to be compiled from multiple small sources Great for +1 requirements
- Puppet Enterprise as the core. Bolted on three different aspects to the stock Puppet Enterprise software (Puppet server, Puppet Console, etc…) UT Puppet API – Code/Data Federation – Hierarchical Configuration -
- Secure by Default Role separation -- BASE repos maintained by Core Puppet group; Apache maintained by Apache team many-jointed modulepath in puppet.conf Modulepath=:/path/to/dept1:/path/to/dept2 Each separate piece checked out to via puppetized VCSREPO runs Updates working copy only if newer Insert VCSREPO snippet here…. R
- Issues that we run into: # there is always a delay between commits and updates # No automated testing # ACL requirements limit collaboration
- Next steps are to update our existing SVN source control to a full-fledged git workflow. We are planning on taking advantage of the Atlassian “Git essentials” package, which includes a lot of additional workflow pieces (Continuous Integration, Jira Integration) The key wins are: Instead of monolitihic repos, we get small agile repos (1/module). This enables a more agile workflow and tracking In addition, git excels at creating new branches. So, new development can occur on non-production branches, which will really speed up development One cool feature with git essentials is that it allows “per-branch” ACLs, which will allow us to lock down the Head/production branch for our Core modules, but enable easy collaboration for the rest of our organization
- Using a git workflow opens up a lot of new possibilities: R10k push deployments instead of the existing pull deployments from SVN R10k can use Puppetfiles to create software-defined Puppet Environments (which is very cool) Automated testing Anybody can contribute to any module. Feeds back into our core principles
- Wrapping it up