PowerBlogs.Com Development

Something is wrong with one of the powerblogs servers, and we're currently working on figuring out what and resolving it. I'll keep you up to date as I get more information.

Update: The problem appears to be with server's hardware. While the people at EV1's datacenter are working on fixing that, I've rented another server and will be restoring it from backups (we have a backup current as of early this morning EST). If I'm done before the other server is fixed, it will become the new primary server and the other server will become its backup when it's fixed. If the current server is fixed first, this new server will become a backup for the old server. I've gotten a lot of progress done on having the software support the live backup system that I described earlier, so when I've got two servers, I'll push to get the rest done ASAP (the log reports were a major bottleneck, but the new off-server generation method will now only take a little bit of modification to adapt to that problem).

Update: So it appears to be a hard drive problem. I've put in to have an EV1 specialist look at the machine and replace the hard drive if necessary. In the mean time, we're working to get the new server up and loaded with the backups. My guess now is that I'll have that up before the older server is fixed. For some people who set their DNS statically (rather than using a CNAME), once the new server is up we'll have to modify your DNS. (Once the new server is up, youraccount.powerblogs.com will work immediately, as will anyone whose DNS is a CNAME.) I'll keep you up to date on the progress with the new server we've ordered and are currently configuring.

Update: We've got the new server, and have done the base config. I'm working on turning it into a Powerblogs server at the moment, which is coming along well. Soon, I'll be testing it out. Once I've tested it, I'll start restoring from this morning's backups.

Update: I believe that I've finished configuring the new server as a Powerblogs server. I've been testing it, and I the tests are looking good now. I'm just about ready to start restoring from this morning's backups.

Update: I've done tests, and the server config definitely appears correct. I'm uploading the backup data to the server. As soon as it's up, I'll begin restoring from it.

Update: Reloading the backup data is going well. I've got the main database up and have done some initial integrity checks and it looks good. I'm still working on getting auxiliary data (stylesheets, uploaded files, etc) up and restored. Once I'm finished with that, I have to republish the blog pages, do a final check to make sure everything is ready, and modify the DNS settings, but once that's finished we'll be good to go.

Update: Oh, fyi, I still haven't heard back from the EV1 "systems support specialist" about the old server. They were going to begin investigating shortly 6 hours ago.

Update: I've got all of the data uploaded and am currently regenerating the blogs. After this, I need to verify that everything's correct and change the DNS, and we'll be all set.

Update: The regeneration is moving along. I've been doing some testing and things are looking good so far. It shouldn't be too long after the regeneration that we go live. I've got a few more tests to do, and then I might start changing the DNS for blogs as they finish their regeneration individually.

Update: The regeneration is going pretty well. Only volokh and whiteperil are left. If your DNS isn't working, please send an email to support@powerblogs.com and I'll investigate. I think that most people on this server are set up to automatically pick up the DNS change.

Also, I've heard from EV1 and it looks like we might be able to recover the data from the hard drive. I'll let you know more on that front as soon as they tell me.

Update: The regeneration is very nearly complete, and nearly everything is back up again. Scheduled posting is not yet enabled, however, and probably won't be until tomorrow. For various technical reasons, when I restored it re-created posts which were saved for later. I need to write a quick program to cull the posts which are saved for later but were already published, and I'm too tired at the moment to trust myself to do that correctly. I should have that up some time tomorrow. I'm going to bed now, but the DNS for the remaining blogs has been set and I've done a partial regeneration on them, so their front page works and they can be used. That means that as of now, all Powerblogs blogs can be used, except for scheduling posts (you can schedule them, they just won't be published until some time tomorrow).

Update: I'm working on fixing the problem where saved posts that were published have been "resurrected". I hope to have the redundant saved posts removed within an hour or so.

EV1 has indicated that prospects for file recovery looks hopeful, but that it will take some time. They said that they should have further results for me in the late afternoon (I believe that that's Texas time). I should mention, though, that even if all of the data from the drive can be recovered, it is possible that a post could have been caught at the exactly wrong time so that it was never written to disk in the first place.

I also just realized another layer of backup which we have — the mailing list archives. You can get to them at http://powerblogs.com/pipermail/yourhostname/ — any lost posts might be there, since that's hosted on a different machine from the one that went down.

Update: The mailing list archives are looking to be a real saver from this disaster. So, if you may have lost any posts that were made after the last daily backup was taken, check your mailing list archive. It's URL is http://powerblogs.com/pipermail/{yourhostname}/2005-December/date.html — e.g. the archive for dev.powerblogs.com is http://powerblogs.com/pipermail/dev/2005-December/date.html

Update: I'm nearly done culling the saved posts, and will very shortly re-enable the scheduled poster.

Update: I'm done culling the saved posts, and have re-enabled the scheduled publisher. Scheduled posts should be good from here on out.

Update: For some blogs, the trackbacks to their posts didn't restore correctly. The trackback data is not lost, and I'm working on restoring that.

I'd also like to take this opportunity, now that things are calming down, to give my heartfelt apologies that it was not handled better and more swiftly, and that it caused so much grief, frustration, and worry. I'm very sorry for that, and we're working on ways to improve our reliability.

Update: For some people, not all of the comment pages were written properly. I've triggered a republishing of everyone's comment pages. This is slowing things down a little bit, but that should be over with in an hour or two. Also, I've been tuning the system to improve performance.