If patch management is a problem in your infrastructure, then this talk is for you. This talk will walk through our journey to automate the patch management of our infrastructure. Details will be shared about our architecture, phased approach and implementation. We will also showcase our new Puppet module that can be used as a framework for patch management with Bolt and Puppet.

1. Automated Patching
With Bolt

2. Nick Maludy
@NickMaludy
github.com/nmaludy Encore Technologies
@EncoreTechCincy
github.com/EncoreTechnologies
encoretechnologies.github.io
DevOps Manager, Husband, Dad
Managed Services Provider

3. Encore

4. Got 99 Problems and Patching is One
•CVEs and Zero Days
•Manual
•Slow
•Burnout
•Broken Applications
•Snapshots… doh!
•Monitoring? … oops!

5. Landscape
•Windows
- 2008
- 2012
- 2016
•Linux
- RHEL 6, 7
- Ubuntu 14.04, 16.04, 18.04

6. Existing Tools…
•Windows
- SCCM
- WSUS
•Everything is a “suggestion”
•Lack of customization
•Linux
- RHEL = Satellite
- Ubuntu = ?

7. Requirements
•Security
• More often (weekly)
• Fast as possible (<1 day)
• Reports
•DevOps
• HA groups
• Customizable workflows
• Cross-platform
• Windows Update + Chocolatey

8. encore/patching
•Bolt
•Community
•Eat our own dog food
•forge.puppet.com/encore/patching

9. Design
•Framework
•Components / building blocks
•Agent-less
•Everything is a Plan and a Task
•Group based
•Common interfaces
•Customizable
•NOT MAGIC

10. Architecture

11. Available?
Create
Snapshot
Pre Update Post Reboot
Delete
Snapshot
Workflow

12. patching::ordered_groups
•Input = Array[TargetSpec]
•patching_order “var” assigns group
•Group by common patching_order
•sort() on patching_order
•Result = sorted array of groups
bolt plan run patching::ordered_groups

13. ---
groups:
- name: patching_a
vars:
patching_order: 1
targets:
- postgres01
- postgres03
- name: patching_b
vars:
patching_order: 2
targets:
- postgres02
[
{
“order”: 1,
“nodes”: [
TargetSpec(“postgres01”),
TargetSpec(“postgres03”)
]
},
{
“order”: 2,
“nodes”: [
TargetSpec(“postgres02”),
]
}
]
inventory.yaml patching::ordered_groups

14. $ordered_groups = run_plan('patching::ordered_groups’,
nodes => $targets)
$ordered_groups.each |$group_hash| {
run_plan(‘facts’,
nodes => $group_hash['nodes’])
}
Example

15. patching::available_updates
•Check for available updates
•Windows
- Windows Update Agent API
- choco outdated
•RHEL
- yum -q check-update
•Ubuntu
- apt upgrade –simulate
• Output = Array of updates
bolt plan run patching::available_udpates

16. {
"updates": [
{
"name": "Definition Update …",
"version": 200,
"server_selection": 0,
"id": "a54401ad-…",
"kb_ids": ["2267602"],
"provider": "windows"
},
{
"name": “notepad++",
"version": “4.0.0",
"pinned": "false",
"provider": "chocolatey"
}
]
}
{
"updates": [
{
"name": "puppet-bolt",
"version": "1.30.1-1.el7",
"repo": "puppet6"
},
]
}
{
"updates": [
{
"name": "puppet-bolt",
"version": "1.30.1-1xenial",
"repo": "Puppetlabs:xenial“
},
]
}
Windows RHEL
Ubuntu

17. patching::snapshot_vmware
•VMware only (for now)
- Bolt control node
- rbvmomi gem
•Optional
•Customizable
•Pluggable
- Dynamic Dispatch
bolt plan run patching::snapshot_vmware

18. Dynamic Dispatch in Bolt
plan patching (
TargetSpec $nodes,
String $snapshot_plan,
) {
# lots of things…
run_plan($snapshot_plan,
nodes => $nodes,
action => ‘create’)
}
plan patching::snapshot_vmware (
TargetSpec $nodes,
String $action,
) { … }
bolt plan run patching snapshot_plan=patching::snapshot_vmware
Requirement : Plans must conform to same “interface”

19. patching::pre_update
•Service health checks
•Backups
•Stop services
•etc
•Runs script on remote node
Linux = /opt/patching/bin/pre_update.sh
Windows = C:ProgramDatapatchingbinpre_update.ps1
•Customizable
bolt plan run patching::pre_update

20. Customizing with vars---
vars:
patching_pre_patch_plan: ‘mymodule::pre_patch’
patching_pre_update_script_linux: ‘/my/custom/patching/script.sh’
patching_pre_update_script_windows: ‘C:mycustompatchingscript.ps1’
plan patching::pre_update (
Target-spec $n,
String $script_linux = ‘/opt/patching/bin/pre_update.sh’
String $script_windows = ‘C:ProgramDatapatchingbinpre_update.ps1’,
) {
$vars = get_targets($n)[0].vars
$_script_linux = pick($vars[‘patching_pre_update_script_linux’], $script_linux)
$_script_windows = pick($vars[‘patching_pre_update_script_windows’], $script_windows)
# … do things
}
inventory.yaml
plan

21. Group custom vars
---
vars:
patching_pre_update_script_windows: C:awesomepatch_script.ps1
groups:
- name: regular_nodes
targets:
- tomcat01.domain.tld
- name: sql_nodes
vars:
patching_pre_update_script_windows: C:MSSQLstop_services.ps1
targets:
- sql01.domain.tld

22. patching::update
•Windows
- Windows Update Agent API
• Special snowflake scheduled task…
- choco upgrade all
•RHEL
- yum update
•Ubuntu
- apt-get dist-upgrade
bolt task run patching::update

23. Logs and Results
•Linux
- Writes stdout log /var/log/patching.log
- Writes results to /var/log/patching.json
•Windows
- Writes logs to C:ProgramDatapatchinglogpatching.log
- Writes results to C:ProgramDatapatchinglogpatching.json

24. patching::post_update
•Start services
•Waiting for services
•Health check
•etc
•Pluggable same as pre_upate
- Linux = /opt/patching/bin/post_update.sh
- Windows = C:ProgramDatapatchingbinpost_update.ps1
bolt plan run patching::post_update

25. patching::reboot_required
•Reboot strategy
•Windows
- https://ilovepowershell.com/2015/09/10/how-to-check-if-a-server-needs-a-reboot/
- https://gist.github.com/altrive/5329377
- http://gallery.technet.microsoft.com/scriptcenter/Get-PendingReboot-Query-bdb79542
•RHEL
- needs-restarting
•Ubuntu
- [ -f /var/run/reboot-required ]
bolt plan run patching::reboot_required

26. patching
$groups = patching::ordered_groups
$groups.each | $g | {
patching::available_updates
patching::snapshot_vmware # create
patching::pre_update
patching::update
patching::post_update
patching::reboot_required
patching::snapshot_vmware # delete
}
bolt plan run patching

27. Patching Now
•500+ VMs
• 5x environments
•1 engineer
•< 1 day
•Every week
• Dev = latest
• Prod = Dev last week

28. Lessons Learned - Bolt
•Simple tasks
•Tie tasks together with plans
•Standardize parameters
•Standardize results
•Keep large binaries out of files/

29. Lessons Learned - Linux
•Bash lowest common denominator
•+100s of systems in a group
•Remember to update cache
•Careful of /tmp and noexec

30. Lessons Learned - Windows
•Connect timeouts
- 200+ seconds
•100 nodes max
•Long tasks = bad
•Slow updates
•Slow File Tx WinRM
•PowerShell versions
•WUA = PITA

31. Roadmap
•Monitoring
•Reporting
•Notifications
•ServiceNow Changes
•Inventory creation from Satellite, WSUS, AD, IPA, VMware, ServiceNow
•More workflows
•Network and VMware patching
•Puppet Remediate Integration

32. Thanks!
@NickMaludy
github.com/nmaludy
@EncoreTechCincy
github.com/EncoreTechnologies
forge.puppet.com/encore/patching
slack.puppet.com
#puppetize-pdx
@nmaludy