Patch management is critical to reducing your attack surface and keeping your endpoints and business running smoothly. Unfortunately, it's also a process that must be repeated weekly, monthly, quarterly, and whenever critical fixes have been identified for your environment. The good news is: with the right tools and some advance planning, this process can run smoothly and leave your IT team with more time to support core business goals.
Join us to learn about trends in patch management, including the latest ways Ivanti is helping Security and IT teams work together like a well-oiled machine.
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
Patch Management Best Practices
1. OSB 130 – Patch Management Best Practices
Chris Goettl and Todd Schell
2. Our approach to security
Take actionProvide insightDiscover
Use best-in-breed tools to
act swiftly.
Clearly identify risk.
Easily find and quantify
the assets you need
secured.
Integrated, easy-to-use security offering
4. Know the People Involved
Immediate patch team
Extended team
Help desk
Network support
Application or desktop support
Management
Functional and organizational managers
CISO or equivalent
End users as needed
Special use machines
5. Know your Environment
System discovery
Talk to IT resources
Basic understanding of system use
Operating systems and applications
Network layout and access
Network maps
Scan the network
Verify access
Leverage existing procedures
Be aware of restrictions and responsibilities
8. Vendor Patch Release Models
Operating systems
Windows on famous Patch Tuesday
Oracle has quarterly CPU (Critical Patch Updates)
Others
Different subscription/access models
Publicly available, e.g. Microsoft
Open with community account, e.g. Cisco
License subscription by endpoint, e.g. Red Hat
Applications
In sync with Patch Tuesday, e.g. Adobe
As needed/available (most companies)
10. Windows as a Service
Phased rollout from engineering through production
Feature updates
March and September
Aligned with Office 365
Quality updates
Monthly and cumulative
Security and non-security
Semi-Annual Channel
Long-Term Service Channel
Source: Microsoft
11. Current Release Summary
Windows 10
1511 (November Update) – EOS 4/10/2018*
1607 (Anniversary Update)
1703 (Creators Update)
1709 (Fall Creators Update)
1803 (April 2018 Update)
Server 2016
10.0.14393 (September 26, 2016 @ Ignite)
Windows Server
1709
1803
*Education and Enterprise
12. Legacy Operating System Service
Support for Windows 7 through Windows Server 2012 R2
Security Only Quality Update
A Patch Tuesday release
Security patches for a given month
NOT cumulative month-to-month
Security Monthly Quality Rollup
Cumulative month-to-month from October 2016
Includes reliability fixes in addition to security fixes
Occasionally added security and reliability fixes older than October 2016
Source: Microsoft
13. Formulating an Initial Strategy
Grouping systems
Ownership or function
Common patch policy
Operational requirements
Patch availability as a function of risk assessment
Pre-deployment system impact and validation testing
Maintenance windows
Policy types
Baseline under configuration control
Continuous rollout as approved
For Windows, security-only or monthly rollup
20. Consider the ‘Big Picture’
One or Multiple Patch Servers
Scalability
Network access to endpoints
Geographic distribution
Internet Access
Content from Ivanti
Binaries from vendors
Patch Administrators
Roles
Responsibilities
21. Typical Test and Rollout Process
Testing
Representative target systems
Success criteria
Rollout
All patch groups
Multiple phased groups
31. Steps in Patch Operations
Standard Process
Monitor content
Build baselines and test (or)
Test latest released patches
Obtain configuration control approval if required
Send notification to employees/users
Scan, deploy, and patch
Monitor results
Emergency Patch
Build response patch set
Send notification to employees/users
Scan, deploy, and patch
33. Incident Response Plan
More cross-team activity
It’s not ‘if,’ it’s ‘when’ things go wrong.
Things to consider
Types of events and associated escalation procedures
Insider versus outsider events and threats
Standard procedures versus emergency procedures
Public disclosure versus information control
Perform regular tests to exercise and improve the plan.
34. Patch Exceptions include Additional Security Measures
If you can’t remediate, mitigate.
It’s all about control and containment.
Reduce user privileges.
Remove direct access to clientpurchase cardPII.
Virtualize workloads.
Reduce exposure to neighboring systems.
Remove direct Internet access.
Implement application whitelisting.
Implement containerization in cases where removal of Internet access is not
an option.
36. Automate and Orchestrate for Efficiency
Patching is one part of a service-oriented architecture.
Application and OS provisioning
Change management
Start with the basics.
Python, Ruby, PowerShell
Go with the big boys.
Terraform, Chef, Puppet, SaltStack
Ivanti provides the patch APIs you need.
37. Vulnerability Scan Automation
API Integration
Vulnerability scan results
Automated patch group creation
Integration of RES Automate
Make automation easy
Close the response loop
This is our approach to security and we are going to use this same process today as we talk about the best practices related to patching. First we’ll talk about what you need to find out about the environment you are responsible for. Next, we’ll talk about identifying which systems need to be patched and how to set up your patch solutions to best handle that process. And finally, we’ll discuss the actual patch process across your company and how to roll out your updates. In conclusion, we’ll talk about some current trends we see in the patch market and how we here at Ivanti are involved.
To begin our patch journey, we’ll start with the basics of investigation and planning.
You are about to start a process that can affect just about everyone in your company that uses a computer to do their job. As I’ll say many times throughout this presentation, every company is different so please adapt what we present to your situation. You may fit into any of these positions, but we’ll be taking the approach that you have the responsibility for patching in your organization and exist within a security or IT operations group.
You must know your immediate patch team. They may be in one location or scattered around the world. They must coordinate patch activities as we’ll discuss later so you must know their roles and responsibilities. Next up, be aware of your extended patch team. You will work with these groups in time of planning and time of emergency. Build a working rapport to ensure your success.
Know the key managers and C-staff in your organization. In many cases you will be providing them reports on your activities and the security status of their machines.
And finally, you must work with end users in some situations. There are often individuals or small groups that have special-purpose machines with special needs. You will work with them directly to ensure their patch and security needs are met. We often see this with critical systems in the infrastructure, such as special financial or web servers. Get to know them and how they operate so you can help them and not hinder productivity.
As patch activities are scheduled and occur, you must ensure a clear channel of communications across your organization.
You can’t patch systems blindly. You have to gather at least general information on every system you’ll touch remotely. As you are gathering or assembling this information you should be wearing your ‘patch hat,’ thinking about how and what you need to do to secure each and every system. What is this system used for? What operating system and applications are installed? When is it available for patch and reboot? How do I get access to this system – network and privileges? And many more.
Every organization is different, so you’ll need to understand where you are coming into this process. In a small organization you may have to figure this out on your own. In a large organization, there may be very clearly defined standards and processes for accessing the systems you will patch. This is all part of getting organized.
So you’ve identified the operating systems in your environment . . .
And the list of applications you need to support . . . What else do you need to know about patches?
Vendors release their security and feature update patches using many different models. Here are a number of examples. (Talk through the points on the slide. ) These are provided as ‘food for thought’ as we begin to formulate our patch strategy.
Before we go any further, I’d like to spend some time talking about Microsoft’s Windows as a Service model so we have a common understanding. This has an impact on how you will do your windows patching,
Vendors release their security and feature update patches using many different models. Here are a number of examples. (Talk through the points on the slide. ) These are provided as ‘food for thought’ as we begin to formulate our patch strategy.
Before we go any further, I’d like to spend some time talking about Microsoft’s Windows as a Service model so we have a common understanding. This has an impact on how you will do your windows patching,
The introduction of Windows 10 saw a major change in the methodical release of a Microsoft operating system. They now start with several phases of internal releases, then to the Microsoft Insider Program, and finally into Production for everyone. This phased rollout is shown here from the Microsoft web site.
The process has evolved from 2015 until this year to the point where we receive 2 feature releases a year coming around March and September. These are now aligned with the releases of Office 365.
This provides a quick summary of the releases available.
Server 2016 is now simply called Windows Server and is the first server to be released as part of the Semi Annual Channel. It is available in Core and Nano modes. Of the two, only the Server Core mode of the OS can be installed on a bare system. The Nano Server mode is only available as an operating system container
Microsoft needed to address the growing number and the complexity of patching vulnerabilities each month. With the apparent success of the Windows 10 model they decided to follow suit with the legacy operating systems and applications.
For companies that were only worried about security or were historically only applying security updates, they introduced the Security Only Quality Update, commonly called the “security rollup.” <See bullets.>
In pure Windows 10 fashion, they introduced the Security Monthly Quality Rollup. This includes an accumulation of all patches released starting from October 2016. It includes reliability fixes as well as security fixes. <See bullets.>
Microsoft knew this would panic a lot of companies because of their poor track records with reliability patches.
By this point you have a lot of information collected and should begin thinking about how to organize it into a patch process. You should consider how you will group your systems in your patch tool. You may group systems together that have common ownership so you can patch them all at the same time or perhaps you want to group systems that will have a common patch policy. We’ll talk more about how our products can help organize your patch process in a few minutes.
You should also consider the operational requirements as you begin to group systems. Consider which systems are most critical, most exposed and need to be patched first as soon as new patches are available. Or as another part of that process you need to ensure the stability of your systems, so you first deploy new patches to a series of representative target systems to verify they work as anticipated and do not interfere with your business applications. You also need to consider when systems are available for patching. Business critical systems may on have a small window of time when they can be offline while workstations on the LAN can be patched in a 12 hour window of your choosing. Also think about the implication of reboots as the patches are applied.
An finally think about what patches you will applying to the system. Based on the vendor release models, do you ‘save up’ your patches and only release a finite set of patches that have been tested and independently approved for deployment. This is very common in the MSP environment where you customer has a service agreement related to this process. Or as an alternative, can you more freely roll out patches on your schedule as time allows? This is not a black and white decision and there are many variations in between. You should also consider the patch approach you will take with your Windows systems.
Before we move into the next section of our presentation, I want to point out that the planning session we just described is a living and breathing process. It should be repeated on a continuous, scheduled basis as your company changes. New systems will be added (or removed), personnel will change, and patch requirements must change with them.
This next section will cover the configuration of your patch system and we will look at the Ivanti Patch for Windows product. But before we begin, I want to provide a quick overview of the patch and content production processes.
DO NOT make changes to this slide without reviewing the slide automation!
There are two automation sequences in this slide. Click on the following locations to start:
Patch Data Flow Title – brings in Step 1 Content Production
3rd Party Vendors in upper right – shows content production sequence
Once the patches are announced by the vendor, Ivanti will download them to our production systems. The patches are processed to create what is referred to as patch content, which is typically an XML file with important information about the patch. This information is used by the patch system to handle patching your systems, as well as provide you with information to take action if needed. For example, the title, release date, CVE information, etc. is all included.
This slide is showing the general case where patches are freely available for us to download. In some situations we do not have access to patches and they must be provided by the customer. Due to licensing restrictions, we can provide the content only for that customer. Customers are paying for special support from the vendor.
DO NOT make changes to this slide without reviewing the slide automation!
There are two automation sequences in this slide. Click on the following locations to start:
Patch Data Flow title – brings in Step 2 Content Distribution
Ivanti Content Icon – shows content distribution sequence
The next step is to get the patch content to your system where it can be used. This is part of content distribution. Although I show it as a push in this slide, the information is sent down in a number of ways depending upon how your systems are configured. In some, it is by a scheduled pull request, in some it is pulled down and updated when the system is started.
I should note at this time that we have separate distribution systems for the former Landesk, Heat, and Shavlik customers with different URLs for each. We are working to consolidate our feeds and download addresses, but that will take time.
DO NOT make changes to this slide without reviewing the slide automation!
There are two automation sequences in this slide. Click on the following locations to start:
Patch Data Flow title – brings in Step 3 Scan and Detect
Patch Server Icon – shows scan sequence
The next step is to scan your endpoints using the latest content and determine what you need to patch. The patch server can send the content to endpoint agents or, if you are doing an agentless scan, can scan them directly to determine which patches you need. This process is fully configurable with the full spectrum of options from ‘do it now’ to ‘wait 20 days during my next maintenance window on the 20th of each month’.
The results are collected by the patch server and are shown here with the green icon. The next step is to get the patches and deploy them.
DO NOT make changes to this slide without reviewing the slide automation!
There are three automation sequences in this slide. Click on the following locations to start:
Patch Data Flow title – brings in Step 4 Obtain Patches
Patch Server Icon – shows patch request to vendor
3rd Party Vendors – shows patch download sequence
The next step is to get the actual patch binaries so you can patch your systems. I should mention at this point, that this process can occur in a different sequence than I am describing here. Your system may get the binaries on a regular schedule, it may download them every time you start the system or it may go out on request. Regardless, using the content information, the patch server knows where to get the binaries from the vendors you have selected. You can see that your firewall needs to provide access to these vendors.
I’m showing the most basic process here, but there are many variations. For example with Office 365, we can’t pull the actual binaries, but can trigger your Windows system to trigger the downloads and updates. In similar fashion, some vendors like Red Hat require you to set up a satellite server to pull the binaries, but then we can trigger the actual endpoint update with our system. However, for 95%+ this is the process we use.
So now we have our content, our test results, and our patches ready to go. The final step is to patch the systems.
DO NOT make changes to this slide without reviewing the slide automation!
There are two automation sequences in this slide. Click on the following locations to start:
Patch Data Flow title – brings in Step 5 Deploy Missing Patches
Patch Server Icon – shows deploy sequence
The final step is to deploy patches to your endpoints. The patch server can send the patches to endpoint agents or, if you are doing an agentless deploy, can push them directly and execute the patch install. This process is fully configurable with the full spectrum of options from ‘do it now’ to ‘wait 20 days during my next maintenance window on the 20th of each month’.
This completes the patch process is not meant to be the exact model you use in your organization, but a representation of what you need to plan for as you configure your patch system.
As we begin to set up our patch systems using Ivanti Patch for Windows or any of our patching products, there are several factors to consider. First, think about the number of endpoints to patch and the configuration of your internal network. Can you reach the endpoints from a single network location? Are your administrators co-located with the patch server? Also think about the geographic location of your endpoints and your staff. Do you need to set up separate patch servers to provide timely service?
We’ve just walked through the patch process, so you can see that you need to set up your firewalls to handle the network traffic. This includes both the content from Ivanti and the binaries from the vendors.
And finally give some thought to the roles and responsibilities of your staff. Will you be assigning specific systems to specific administrators? Will everyone need access to all the endpoints? This ties into the slide where we talked about putting together that initial strategy.
The next thing to think about is how you are going to test the patches to make sure they work in your environment and then put together a roadmap on rolling them out to your organization. This slide shows the typical test and rollout process used by many customers. It is easily understood, but requires some planning to effectively implement.
In our initial strategy formulation, we talked about planning how often you need to patch the various endpoints in your environment. As the patches are available from the vendors you collect the content on when you reach the required gate, either a set of patches or a specific date, can build your patch groups and prepare them for testing on select systems. Once testing is complete your next decision point is usually how well they perform. Assuming all pass or the side effects are understood if they don’t work as anticipated you can decide if you are ready for rollout to production. Although not shown on this slide, there may be additional gates associated with groups of production systems.
So now its time to dig into the products a bit. This is the first screenshot showing the organization of endpoints into patch groups. On the left is Endpoint Security and on the right Patch for Windows. In this example we are simply showing a few test groups laid out for servers and workstations.
Similar to the example used for Endpoint Security, this shows the selection process and wealth of information provided with from Ivanti content. This is a Server 2008 patch from July 2017. You can see is a Critical patch and based on the CVE-2017-8589 is likely to be exploited.
This shows how to create a Patch Group from a selection of patches. Select all, right click, and fill out the dialog box.
This shows the final patch group ready for deployment.
Patch for Windows uses Patch Scan Templates as a tool to set up what you are scanning for. In the previous slide I showed how to create a patch scan group which is used by this template as a baseline to scan. You can see over to the right side where this is located.
This is a scan template that looks at all applications except Java. Notice how this is set up. Rather than use a baseline set of patches for a particular month, this scan template can be used month after month and will look for all the latest vulnerabilities and let you know what is patched or not.
Putting this all together, we combine the patch scan template with the machine groups and some other configuration items we didn’t talk about which include how often to scan, how to copy the patches to the endpoint, and finally how the endpoint should react when that patches are applied.
This final slide shows one last aspect of the product – ease of use. In this case a performed a simple scan of my machine using the default templates and viewed the results. You can see with the latest scan, I have a number of patches to apply. With a simple right click, I’m able to immediately deploy the patches to secure my endpoint.
After all the planning and system setup, running patch operations is almost anti-climactic . . . almost!
We’ve talked about the patch process and this slide provides a fairly high level summary. In almost all situations you’ll follow a standard process based on your strategy. It will evolve as you learn more about your endpoints and how you interact with your customers. It will also evolve and change as the environment changes, e.g. acquisitions, new applications, etc.
You need to plan for emergencies and they will happen. For example, WannaCry put the world on alert and required emergency patch as a special response. Make sure you have this built into your processes.
Patching is an interactive operation between IT operations, security teams, and end users. You can’t over-communicate! If users know when something is coming, less time will be spent tracking down missed patches. The process is greatly improved.
This is just one example of the notices sent out in Ivanti.
You can start by considering this a patch best practice, but it can be so much more and builds on the cross-team activity. <Review all the bullets.>
You can’t possibly remediate all vulnerabilities—some will never be fixed and some are a waste to time to address. In these situations you need to mitigate the risk by employing additional security measures to contain any damage if things go wrong.
There is a rapidly growing market of tools that provide task automation and process/workflow orchestration. These have all grown in response to a need for a service-oriented architecture that can provide guaranteed provisioning and change management capabilities. And by guaranteed I am referring to providing a fully configured, exact system every time. Most of these tools are based on the fundamental scripting languages shown here, so you can start with the basics if you want and then move on to the more advanced capabilities these tools provide. Terraform (GO), Chef (Ruby), Puppet (Ruby), SaltStack (Python)
Ivanti provides some basic workflow automation in its products, but realizes our customers really need an API these tools can use to take patching to the next level. These APIs were released in the products shown here.
And our final slide for today is specialized case of the orchestration we just discussed. This is an area WE are investing resources today.
We’ve all been handed a list of vulnerabilities found by an audit and told to go fix them. We are using our APIs and tools from our latest acquisition – RES Automate, to make this job easier. We’re building tools that will take the results of a vulnerability scan and automatically build a patch group to remediate the found issues. Our goal is to make this process easy and allow you to provide quick response to the findings.