Finally after 2 hours or hard work, I got tagz up and running. Its currently running on my vps (the same machine which was running the staging server).
Here’s the background on the issue.
The main server was running Ubuntu 8.04 (Hardy Heron)
Last night, I ran a standard apt-get update; apt-get upgrade.
It upgraded libc and libc-dev, and that was it. After that, all perl processes would just hang spinning busy on the cpu. I couldn’t do a thing. Tried running a couple of the offending scripts in strace, and they all hang on a clone() syscall. Tried restarting the AMI, but it still persisted. The worst problem was I couldn’t even get a db dump because pg_dump wouldn’t work. And the last snapshot I had was about 17 hours old.
So, here’s what I did. I terminated the postmaster instance, took a backup of the db directory, scp’d it to my vps and tried using it there. Then I found that I had to recompile postgresql with –enable-integer-datetimes for it to accept the database. Did that and few other tweaks (I’d switched DNS to point to the VPS early on) and here we have it, up and running.
I’ve got to move back to EC2 soon (The VPS wouldn’t be able to handle the loads for long). But this time, I’m going back to Debian Stable, I’ve had enough of Ubuntu, I have no idea how something as innocuous as a libc upgrade can barf things up so badly.
Eric Hammond on October 2nd, 2008 at 4:29 PM says:
Thanks for reporting this problem. It sounds related to a problem which many others have experienced using EC2 + Ubuntu/Debian + EBS volume + XFS. Does this describe your setup? Also, could you provide the specific Ubuntu AMI id which you started with? Based on reports so far, it seems to only be a problem with AMIs prior to 2008-09 when glibc6-xen was added, but it’s not entirely clear yet.
admin on October 2nd, 2008 at 5:40 PM says:
@Eric: I’ve emailed you the details of the EC2 setup.
Eric Hammond on October 4th, 2008 at 1:23 AM says:
Though we’ve been chatting over email, I figured I should post here that I recommended starting with a more recent release of the Ubuntu AMI on http://alestic.com as folks are reporting similar problems with the older AMIs which you were running. I can’t guarantee this will fix it, but we’ll work through issues as they come up.