Update: Clearly, we’re down again. John’s working on figuring out why the problem reappeared and how to fix it. Watch for updates on the home page (Abby)

The site is back up, after having being down for a day and a night. An errant script knocked down the “read” database, and when the box crashed, some of the db files were corrupted. This meant the whole db had to be restored from the master db. In addition to the db itself, some of the log files were also affected. Turns our MySQL is a lot more finicky about how log files are handled than I’d ever known, or hoped to know. After a lot of digging around, everything is back the way it belongs, and all that was lost is a night of sleep–no data was lost.

I’d like to apologize profusely to everyone who was inconvenienced. We’ll put more safeguards into place to try to minimize such outages in the future, hopefully.

Labels: 1