ColdFusion Muse

Why My Wire Cutters Have Been Retired (and My Servers Are Grateful)

I've never fried a server yet. I've had drives fail, motherboards crash, bad memory (computer memory... I think) and all kinds of hardware issues, but I have never personally been responsible for killing a server. That unbroken record was given a little scare today. One of our production CF 7 servers was making a noise. In fact it was the same CF 7 server that hosts this very blog. Not to be indiscreet, but the sound was a bit like... well... squeaky bed springs. For a moment I had this vision of our Windows server meeting a pretty young Apple X-serve with it's own rack and.... anyway we went to investigate.

We pulled the cover and found 2 very small fans in the back. One of the fans had failed long ago and now at long last the other one was failing. It was one of those very slim brushless fans that are crammed into every corner of a 1U server. Now I will confess that in my own data center I sometimes "take over" and do things like this - for no reason, other than I erroneously think I'm gifted. The wonderful folks who work for me are quite skilled and don't really "need" me mucking around with the equipment - but they do enjoy watching me do hardware stuff because it gives them something to laugh about.

Over the years I've developed a reputation for being... uh... innovative with hardware. In our data center we have some nice, clean, new servers. We also have a few servers that I have built myself using a hacksaw, a drill, electrical tape and a box of spare drive cages, chassis, power supplies, old motherboards, and various drives and cards whose manufacturers have long since been assimilated by the Chinese Economy.

For example, one of our servers is a dual PIII ATX board crammed into a modified mini-ATX half rack chassis with a modified P4 power supply. It features a drive cage that is fastened to the chassis with 2 drywall screws at an angle so the drives angles downward when you peer into the open bay. The chassis had to be modified to accept the slot 1 motherboard and chips. The result is that the heat synch would not fit with the Fans attached. I filed down the heat sink fins and inserted 3 little 1 inch fans in each set of fins with 3 plastic drywall anchors and some screws. Then I hot wired the assembly to the motherboard at one of the fan ports. Anyway, you get the idea. Back to the problem with my fan (one of my many fans actually).

"No big deal", I thought. I found one of those small fans that was part of a spare heat sink (ahah! ... I told them that this old PII heat sink might come in handy someday!), and came up with a plan. I unplugged 1 of the 2 fans and snipped the wires very close to the fan case. Then, using some electrical tape I wired my new fan to the old wire and snaked it back through the chassis. The new fan fit perfectly right next to the old fan and had a spot right at the vent. At this point you should know that we were doing this with the server running. Yes, you can often fix a fan problem with the server running. Just make sure you take some kind of container for screws or metal objects (snippers, screw drivers etc) and keep those objects below the server - so a dropped screw won't kill your server. As a rule, if you need to handle a screw or something over the motherboard you should shut the server down and unplug it. But fans don't always have that as an issue.

I plugged in the fan in and tried to spin it up - but it was a tight fit and although it looked ok, there was a wire that was keeping the fan from spinning. I pulled the fan out and examined the wire. It ran to the other (long dead) fan. I reached down and pulled up the wire so I could snip it and move it out of the way. Just before Mike (who is much better at this sort of thing than I am) let out a "...hang on a minute..." I snipped. My brain, which moves at a speed that is diametrically opposite to the importance of the server I'm troubleshooting, registered the fact that I had not first unplugged the fan at the same time it registered a nice little flash - a spark in fact. This spark had come (my brain told me) from the same severed wire that I was now holding in my hand.

The next thing I noticed were the fans on the heat sink winding down and a bright yellow LCD on the motherboard flashing mournfully - slow-fast-fast-slow. If I hadn't removed the speaker long ago I would probably be hearing a beep at this point. Mike was trying to suppress a grin. Our server was dead in the water. I was sure it was as dead as Jimmy Hoffa. We didn't panic though. We pulled the server, unplugged the power cord and waited a minute. In my head I thought of Walter Cronkite in Apollo 13 when they were going through reentry and could not talk to the astronauts.... "The only thing we can do is pray... if after 4 minutes we don't hear anything we must assume the worst."

Lucky for us the server booted and it seems none the worse for wear. Mike took away my wire cutters - which is just as well. I think I heard a squeeky power supply in the 3rd rack over....