When Patches Attack
Last night I was sitting at home and using my VPN to dial into one of our servers (a Win2k3 server). I noticed that there were a couple of patches pending installation. Now as a rule I do not run every patch, nor do I ever let windows "manage" patching for me. Instead, I let windows download the patches and I choose when and what to install. Still, a couple of these patches were important security fixes (Usually a good idea) so I installed them. Now windows does not always require a reboot after patching, but sometimes it does, and yes it is one of the annoying things about Windows, so please don't use this post to comment on how much better Linux is than Windows or cheese or Santa Clause or sex or whatever. Anyway, this time it did ask and when I chose to restart things went "a bit wonky" as some of my UK readers might say.
After I clicked the restart button, my remote desktop session was immediately terminated (and this is normal). I opened a command line and set up a ping using the "-t" switch:
d:\>ping 10.0.0.245 -tThe "-t" switch is useful. It keeps pinging every few seconds until you break out of it with a Control + C. Now typical behavior of a server at this point is that it should keep responding for 30 or 40 seconds, then stop responding, then start responding again after a reasonable time to account for reboot (90-200 seconds depending on the server). This time however, the server just kept responding. It didn't shut down.
Now I was in a fix. I could no longer RDP into the server since that capability was already shut down. The server was in limbo with some services running and some services shut down. Obviously something - some software or service - was still hung and preventing the restart from completing. I needed to finish the restart in order to be sure that the server was functioning properly, but I was home in my Jammies with no physical access to the server. What could I do? Fortunately there is a fix for this problem.
The Fix
First, I logged into my regular desktop as the domain admin. I needed to be logged in as a user with admin rights to the server that was hung. Next, I right-clicked on "My Computer" and chose "Manage" from the context menu. I right-clicked on the top level item of the MMC object pane called "Computer Management (local)" and I chose "connect to another computer" where I entered the friendly name of my ailing server.
This hooked me up to the server for the purpose of managing services and looking at the event viewer etc. You probably already know about this capability, but there is another often overlooked feature of the management tool that allows me to manage "shut down" operations. Here's the scoop. I right-clicked on the "computer management (servername)" and choose "properties". This displayed the following:
I clicked on the "Advanced" tab and saw a few familiar items that I am used to seeing when logged into the desktop of the server - including the "startup and recovery" area with a "settings" button. I clicked on the "settings" button to see the following dialog box:
Notice the little "shut down" button? This button gives you a way of shutting down the server with a couple of useful options. I clicked on it and saw yet another dialog box that like this:
From here you can shut down and you can force hung applications to close. In my case I selected "Restart" and chose to force apps closed "if hung". I clicked ok and then went back to monitoring my ping - and Viola! I got the response I expected and in a few minutes my server was fully rebooted.
You might have guessed that one of the reasons I write posts such as this is so that I can remember the information when I need it again. I frequently find myself looking up posts I have written for obscure tips that I only need once in a blue moon. One of the beauties of blogging is that it serves as a knowledge base of stuff that would ordinarily escape my brain.
3 Comments
-
If I'm reading this correctly, (a bit tldr with all the extraneous stuff about patches), you ran into a moderately common issue with remote desktop (RPC service shutting down without actually rebooting the machine).
To avoid it in the future, there are two decent steps you can take.
1) launch in console mode. mstsc.exe /console
2) restart from the command line
shutdown /r /c "Reason for Shutdown" /t 30
Seems to mostly happen when someone else is logged into the machine and left a session open of some kind, but I don't think that's the only possible cause.
BTW -- please switch to a better captcha or have it skip l/i/1/etc... fail once because of that stuff and all you get is a black box that says "captcha not available" -
@JC,
Sorry about the captcha - but it does allow only the most dedicated commentors to get through :)
I always run remote desktop from the console. Your command line is a great tip - thanks! It looks like it might be specific only to win2k3. Does it work on Win2k as well?
Sites to Watch
Aggregators
RSS
