Trashed libc - getting out of the doodoo

By Veghead

After spending the best part of 5 hours trying to patch a Cobalt RaQ 4, I thought it might be an idea to document what I had to do in case someone, somewhere ever finds themselves in a similar position. Yet again, for the second time this week, it was essentially the fault of rpms. I hate rpms. They're so very nearly good, but as the cliche goes, a miss is as good as a mile.

Obediently obeying the patch order, I applied a few patches to the RaQ using its web interface. One patch failed to apply and of course it was probably most temperamental patch of the lot (aside from the kernel) - glibc. How many times has redhat managed to ruin my day with glibc rpms ?

Unbeknownst to me, the root partition was at about 99% capacity. Unlike Solaris patches, rpms don't give you a warning if space is tight - they just go ahead and fail. I was left with no libc on the system at all and thus neck deep in cack.

As you will know, the only way to rectify this is with physical access to the machine and a CD. As this is co-located god knows where and it's sunday this was going to be a major hassle so I decided instead to repeatedly scream a selection of swear words and generally terrify the parrot.

However, by some miracle I had a root shell onto the box - but this wasn't much use as even the most basic of commands were broken - ever tried to navigate a n*x box without ls ? The other thing that seemed to be running fine was apache/php.

To cut a long story short, I managed to get the box back in gear remotely, but it was a MAJOR hassle. If you ever find yourself in this position then I hope you find this useful. Here's what I did:
 echo "<?php unlink(\"/tmp/cat\");
export LD_PRELOAD=/tmp/

If it worked, then then do a little dance and give the parrot some nice seeds. Oh yes it might also be an idea to move libc into a sensible place (like /lib) and try to fix the broken patch that caused this problem in the first place.

<< Back

Valid XHTML 1.0!