IT people, how do you patch management

[Luckyflux]

Hi,

So how do you folks do patch management? I have a meeting coming up where I might be forced to do a patch management deployment and strategy for my company. But I have never been in charge of patch management before. And out of all of the places I have ever worked, there has never been a strategy of patch management.

What are some of the most common issues that arise? I would hate to deploy a patch to the domain only to find out the patch bombs out a program. I could set up a test machine, but we have many different types of hardware throughout the facility, such as Dell notebooks, and IBM desktops. Wouldn't that make a difference with the patches....ya know the patch will work on a Dell, but not on an IBM?

Besides a test environment what else would I need to worry about?

What about patching servers?

Any help will be greatly appreciated, thank you in advance.

 

[Marc]

Hi,

So how do you folks do patch management? I have a meeting coming up where I might be forced to do a patch management deployment and strategy for my company. But I have never been in charge of patch management before. And out of all of the places I have ever worked, there has never been a strategy of patch management.

What are some of the most common issues that arise? I would hate to deploy a patch to the domain only to find out the patch bombs out a program. I could set up a test machine, but we have many different types of hardware throughout the facility, such as Dell notebooks, and IBM desktops. Wouldn't that make a difference with the patches....ya know the patch will work on a Dell, but not on an IBM?

Besides a test environment what else would I need to worry about?

What about patching servers?

Any help will be greatly appreciated, thank you in advance.

firstly you need to look at what you are you going to be patching - are we talking just Microsoft OS and Applications or what?

Secondly look at the environment - all in once location/scattered/regional offices or what?

I'd start off looking at Microsoft's WSUS (Windows System Update Server current at V3 Service Pack 2). This will handle the updated for Microsoft Window, servers (such as Exchange or SQL) and office applications.

You can build a a cache of patches so that if you need to rebuild a system the patches will be avaiable locally.

Drivers from various manufacturers that sare submitted to Microsoft can included but there's no guarentee that you can do all your hardware through the WSUS - that's probably going to be one for the manual repository building.

You can also build muliiple WSUS systems which update from a central server on your premises to releive the load.

Configuring the systems to use a WSUS server you've build can done through the Active Directory.

Nice thing about WSUS is the price tag - it's free.

For paid you've systems such as System Center Manager from Microsoft or the Shavlik range are two I've heard of or played with.

For anti-virus the best approach is to have a centrally managed anti-virus system where you run it from a central server.

 

[Luckyflux]

Does that do it automatically? What about proprietary software? Does WSUS need it's own server?

This is for an all local environment, patching mostly Windows infrastructure. Would like to be able to patch OS and all corp apps such as Adobe, Filemaker, etc...

I need a strategy ya know? We currently have Landesk which does patch management, but I don't want to update a machine without testing it. If I cannot duplicate a production environment, then how do I test? Or more importantly what is the procedure to do this? I have never been in charge of patch management before and I have to go into a meeting in 15 minutes about patch management which is going to fall directly onto my plate.

So I need help, but I also need understanding. Like what is your strategy towards patches? Do you deploy them right away? Do you test before you deploy? Do you have a suitable test environment? I know my boss is going to want some 411, so I need some direction towards implementing a patch management strategy.

 

[Marc]

Does that do it automatically? What about proprietary software? Does WSUS need it's own server?

This is for an all local environment, patching mostly Windows infrastructure. Would like to be able to patch OS and all corp apps such as Adobe, Filemaker, etc...

I need a strategy ya know? We currently have Landesk which does patch management, but I don't want to update a machine without testing it. If I cannot duplicate a production environment, then how do I test? Or more importantly what is the procedure to do this? I have never been in charge of patch management before and I have to go into a meeting in 15 minutes about patch management which is going to fall directly onto my plate.

So I need help, but I also need understanding. Like what is your strategy towards patches? Do you deploy them right away? Do you test before you deploy? Do you have a suitable test environment? I know my boss is going to want some 411, so I need some direction towards implementing a patch management strategy.

WSUS is pretty much Microsoft Products only.

For adobe products etc then you're going to have to look at something like Shavlik NetProtect.

http://www.shavlik.com/sol-patch-management.aspx

They usually give you the option to choose what patches to chose. With some patches it's pretty important to install them the right away (for example the recent IE patch).

But it's a tricky thing. For example updating your version of JAVA can easily break things.

I guess the best approach is a test bed but unless it's a machine in use you don't know whether you're going to run into any issues.

Probably the biggest issue is usually service packs which have been known to break stuff big time but you can usually hold off with loading them. But this can also depend on what the machine is running. A fairly standard system with Office and you'll not likely to run into issue. A system with more obscure software is another matter. About 7 years ago was part of a group of contractors brought into clean up some issues on a corporate network. Part of that work was loading on SP4 for Windows 2000 but we couldn't do their CAD systems becasue Autocad hadn't at that point certified the particular version as compatible with W2K SP4.

Also it's a problem compounded by the fact some patches cannot be uninstalled.

Networks I've worked on the past have been pretty much standard Windows & Office so WSUS has done the trick and just deployed new updates when they come through but I'm alwasy learning.

I guess in your case say deploy patches to your test machine (you're probably the local IT guru so I'll say you're the local IT guinea pig - so if something craps out you won't have users yelling at you). Before patch do a backup of the system (say the night before) and then deploy the patches but do it groups.

Also have it as policy that no data is to to be stored on the workstations and users run roaming profiles. That way if a machine has a problem due to a patch/update and you can't fix or can't roll backup, restore from backup. As there's no userdata on the machines you don't have to worry about losing anything (and if they break policy and store data local they only have themselves to blame).

Unless there's a compelling need, leave printer drivers out of the updates - they can be a major pain in the arse when the new version has different settings and options to the old. I've also seen a update HP driver cause a PC to blue screen even the user wasn't printing.

 

[Robert Maxwell]

As Marc said, have a test machine that's similar to your other office workstations. You can use automation software such as SMS (Systems Management Server, from Microsoft) or Kaseya. Both of those products can do software inventory and automatic patching, but the testing is the important part. There really isn't anything that can automate that for you--you'll need to try out the most common applications whenever a patch is applied and make sure they at least start up and function.

 

[Luckyflux]

It is a big onion isn't it? Where is and how is WSUS run? IS it on it's own server? Can it run from the PDC? It sounds good, but we would need to patch Adobe as well, so I don't think we can run that, but I am interested in any more info you have about it.

We had out patch meeting yesterday and came up with some good starting points. Then this project was officially dumped onto my lap so I was having a hard time coming up with a plan. That's when I made the first post in this thread.

So I went home, rolled up a twist and sat on the balcony to get taken away.....but as I was flying I was trying to think about how to do this whole patching thing. It was like the proverbial light bulb going off, and I had a great idea. Create a 5 user guinea pig test group. This group will get the first batch of new patches to be installed one week after the patch tuesday release date. Then one week later one-third of our users will get the same patch. Then a week later another third will get the patches, then one week later, the last third will go. That way we will stay a maximum of one month behind current.

There are other details but they are mostly location specific to my company, but that is the general gist. I was instantly elated when I came upon that idea, and I think it will work.

One part of the process is to disable sleep mode for machines and to prevent the OS from turning off the NIC to save power....if that is enabled it will prevent the machine from getting the patches. So I am trying to roll out a registry file that will set that off. Then I can deploy that with Group Policy and be set.

 

[Marc]

It is a big onion isn't it? Where is and how is WSUS run? IS it on it's own server? Can it run from the PDC? It sounds good, but we would need to patch Adobe as well, so I don't think we can run that, but I am interested in any more info you have about it.

WSUS can run from an existing server whether it's a file server or AD controller - it's not that heavy in rescources though depending on the patchs you keep etc you want to make sure it has at least 20GB of space availabale to it.

It uses an SQL server to store data. It will either install it's own MSDE/SQL Express instance or if you have an existing SQL installation you can make use of that.

No it won't patch Adobe. But consider a couple of facts.
a) the level of urgency with Adobe patches isn't quite to the same level as for Microsoft
b) look at how much Adobe software you've got. Do you have people running Acrobat (full version)/Photoshop/CS or is just people using Acrobat reader? I haven't touched CS in the better part of 12 months but I know Acrobat reader does have it's own update tool so it might possibel to leverage this (poke around on the adobe site and see what's there about updating automaticall). I guess the only draw back is the it machine will download the patch seperate which but the level of impact would depend on your data allowance.
this thread.

So I went home, rolled up a twist and sat on the balcony to get taken away.....but as I was flying I was trying to think about how to do this whole patching thing. It was like the proverbial light bulb going off, and I had a great idea. Create a 5 user guinea pig test group. This group will get the first batch of new patches to be installed one week after the patch tuesday release date. Then one week later one-third of our users will get the same patch. Then a week later another third will get the patches, then one week later, the last third will go. That way we will stay a maximum of one month behind current.

Just choose your guinea pigs wisely. Make sure they aren't the people who will nail your head to the wall if something goes wrong.

Your staggering plan sounds good but as I said earlier look at the software running on the machines - if they are pretty standard say Windows and Office you could bunch the deployment schedule up a bit closer.

One part of the process is to disable sleep mode for machines and to prevent the OS from turning off the NIC to save power....if that is enabled it will prevent the machine from getting the patches. So I am trying to roll out a registry file that will set that off. Then I can deploy that with Group Policy and be set.

No I don't think you'll have to disable the powersaving - the systems should wake up for the patch deployment becasue the update time is setup as part of the group policy.

In worst case you make use of wake on lan setup.

Also you don't need to touch the registry - do it all through group policy - you can point the systems at the WSUS server, set the update time etc etc.

What I would do also is device create groups or oragnisations within your active directory. Have one group the the guinea pigs, a second group for those who will get the patches after the testing and then one for the final group. That way you can have seperate group policies to stagger the deployment (this might also be possible though WSUS - I'd have to check).