BGINFO - A Posh Recreation
Recently I have been building a lot of Windows Servers in different environments - one
This Morning Grumpy Admin, has been battling against that old enemy the Windows Update Agent – making him more than grumpy and infact close to getting a flight to Redmond and dealing some good old fashion BOFH lift shaft action! Yesterdays, audit detected that some of our hyper-v servers were missing some updates.
This hyper-v servers are in a cluster. But we don’t have the Cluster-Aware-Update service installed… Instead, I normally manage the patches via SCCM. However, it doesn’t matter what I did I couldn’t get certain KB’s to be installed. So they weren’t fully patched and I might have ignored it for a few months. Hand in the air my bad! If the reporting features of SCCM were better I would of spotted it before, honest!
These Hyper-V machines were upgraded in place, from Windows 2008 R2 to Windows Server 2012. They are in the Server Core configuration state. How do I know they did this upgrade path… There is an windows.old folder right there on root of the c:\ ! Makes me grumpy they don’t tidy up after an upgrade. Worse, they decided to upgrade rather than clean install! There junk everywhere on these machines!
I was able to easily workout which updates were failing and causing the problems. The C:\Windows\WindowsUpdate.Log file is a good place to start, as well as the c:\Windows\SoftwareDistribution\ReportingEvents.log. Together with the event viewer (accessed remotely of course) provided enough information to try and troubleshoot this issue!
I was able to get error codes generated and the exact KB numbers that failed. Google searching for fixes for Windows Update Issues and Error Codes, just gets you trash results, websites that try to peddle you their “Automatic Fixing Tools”. Which will no doubt cause more issues than fix as well as provide them with unlimited access to your banking details. This makes me grumpy. I head once again to the TechNet forums and do some good reading there.
Everything I tried had been listed in forums. So knowing which updates installed and which ones don’t – I started to patch the first Hyper-V server in the hope if some of the applied updates actually fix something to allow the failed updates to apply. Sounded logical and sounded better to the boss than me sat their saying it doesn’t work and I don’t know why!
So the first thing, I had to do was to stop them from being pointed towards our local WSUS server. A legacy setting from the time before I used SCCM in my environment. With them now collecting updates directly from Windows Update – just so I can rule out any chance of corruption in my local WSUS server. I start the long process of getting the machine as close to fully patched as I can. These servers being server core, I found the quickest way to do it was to use the
SCONFIG
and use the option 6 and select all recommended updates (R) 🙂 Here I can manually select from the presented list of recommended updates which update install. The downside is normally it requires a reboot between installations. This could be a long slow process if you have lots of updates to apply. I select (S) option to install a single update and choose a KB, installed and rebooted with no problems. As annoy and slow as this was there only a few updates, so I repeat until I am only left with my troubled updates or so I thought.
By accident, selected a update that causes a failure, this of course failed to install upon the server reboot leading to a roll back. This also had a very interesting side effect. The installing a single update no longer worked. It would always try to install the KB that fails as well as any other updates. Basically it broke the Windows Update mechanism. This make me really freaking Grumpy!
To fix this, I had to wipe out the softwaredistrbution folder. To do this I had to stop my Windows Update Service, (stop-service cmdlet) renamed/delete my softwaredistrbution folder and then restart the service and it will rebuild the datastore file. This isn’t the best method of doing things, in fact often on the forums people where being shouted at for doing this! But I had no choice. I wasn’t authorised to rebuild this machine to fix any issues, so I had to sort this the dirty hack way if I could.
After many cups of coffee and reboot cycles of this poor Hyper-V Server, I got the as patched as I could, then I ran all the normal tools against the server to ensure there wasn’t anything iffy with it, just to waste time and look like I was doing stuff. When in fact I didn’t have clue what the hell was wrong with this Windows Update shit!
chkdsk
defrag
SFC
While these were running, I was browsing Microsoft Technet to try and troubleshoot the individual KB installations issues that I was having. I came across this blog post which, while not the exact same KB’s sort of felt like the same.
Server upgraded from 2k8R2 – Rolling back on reboot… he tried everything I had, including the SoftwareDistrution. Then a Jospeth from [MSFT] posted a very helpful tip.
“delete the following regkey HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Publishers\{e7ef96be-969f-414f-97d7-3ddb7b558ccc}”
I did that, not knowing what it would or what the hell the publishers under WINEVT did. That I would Google at a later date, for now all I want to do is get this machine patched! I was shocked, all but one of my failed updates installed and took! No Windows Roll backs on reboot! Yippy. See Grumpy Admin can be happy at times! After some more reading of forums and the likes. I decided that it just wasn’t going to happen with this last remaining hold out …
So in a last ditch effort, as someone reported that they had luck using DISM to install the patch. I decided to download the KB’s MSU. But rather then mess around with DISM right off the back, I thought…. I could use the Windows Update Standalone installer to install this.
This was as simple dropping in to a command line and running -:
WUSA <filename.msu>
It installed and all was right with the world finally, as I was starting to get a caffeine head ache 🙂 This server was finally patched and my work for the day was done… Now to Dilbert and TheRegister 🙂
So the thing to remember is when troubleshooting Windows Update Issues
TECHNET is your friend, SCONFIG can be a quick way to fire off windows updates – it actually uses a cscript.exe script to do this.
There will be Powershell scripts out there that people would have written! I might investigate these at a later date and try deleting the SoftwareDistrubution folder after stopping the Windows Update Service. Not Forgetting there is always the good old faithful wusa utility which you can use if you download the KB’s and copy them directly to the server 🙂
Hazzy