UPDATE: Many books found, but not all. Crash problem solved. See end of this post.

If you’ve been following the blog, you know that LibraryThing recently moved to a new, much better server. This move has made LibraryThing fast again, but the transition has been rocky. Yesterday, a glitch erased some recently-entered books, and the new server’s backup failed.

The losses are from books added in the last four days (ie., since the server switch). Some books added since then were recovered from “bin” files. And all books added via import have been recovered, but only as a re-import, so any subsequent edits to those books have been lost. The site will go live this evening, once the import queue has finished and I’ve sent messages to users who lost books (see below).

Here are the steps I am taking for users:

• If you lost books, the system will send you a profile message indicating how many.
• Users who lost more than 25 books receive a free membership or membership upgrade. You can use it on your own account or gift it. Check your profile for information.
• All members who lost more than 25 books and paid in the last 30 days can receive a full refund—thirty days is the period PayPal gives me to do this. Send me your user name and either the email PayPal knows or the PayPal receipt.

For the site, I and LibraryThing’s database administrator are doing the following:

• The import feature has been changed to screen out ISBN duplicates. This means that you can upload an exported file without duplicating or “writing over” books.
• A second server (the old one), which is housed separately, will act as a constant “live” backup, getting all the same information that the primary server does.
• The nightly backup has been fixed. (The new server’s didn’t have the right “permissions”).
• I will shortly improve the CSV export and add a tab-delimited export.

Although a “beta” project is subject to occasional lapses, this sort of problem is unacceptable. Those who have been affected have every reason to be angry. Since I am at present synonymous with LibraryThing, this is my fault. I offer you my apology. I will do whatever I can to make it right for you.

The person who has been helping me with the database offered the pithy remark “this sort of thing never happens twice.” It’s not that lightning never strikes twice, but that once you’ve been hit you stop bringing stainless steel cutlery to picnics. LibraryThing’s new backup regime—nightly backups and an enterprise level “hot backup” at a different location (in fact, on opposite sites of the continent)—will go a long way to ensuring future data security.

Thank you for your understanding.

UPDATE: Comments posted to this entry noted that some missing books predated the server transfer. This was a good catch. At the same time, the server could hardly stay up five minutes. The problems were related. (And related to the previously-mentioned crc32 change.)

I found all these older books, substantially reducing the number of users affected. (Over 100 users can keep their free accounts although they have fewer than 25 books missing—most have none.) As for books entered Monday night, few users managed to enter any, but those that were entered were lost—the crashes were just too severe. All this did have one salutary effect—I figured out the big problem. I think it will stay up now.

Labels: 1