Duplicate book warning. I’ve added a feature to the Add Books tab that flags when you add a book already in your collection. Since “duplicate” is such a slippery concept, it works exclusively by ISBN.

I will be rolling this feature out past the Add Books page in the coming days. I wanted to get this out there because a number of users wanted it as they re-add books lost during the crash.

Also—fair warning—I’m going to be forcing the ISBN fields to conform to a standard, hyphenless format (some libraries like to add hyphens). If you use the ISBN field for something else—like your own personal code—you may lose it in a day or two. Write me if you neeed help moving it.

PS: The book featured above, Sophie Belfort‘s The Marvel College Murders is fantastic, a hillarious, intelligent murder mystery—really a novel of manners—set at a fictional Boston-area university. I was amazed to find nobody else on LibraryThing had read it.

Scheduled downtime. LibraryThing will have a scheduled downtime at 2am EST (11PM PST, 8:00 GMT). My dba-helper is going to try something to improve stability.

Users with large libraries rejoice! I added a link to show all the “pages” in your catalog. Here are the pictures.

Click “all” and get.

Yeah, it’s a small feature. I’m still spending much of my time on the db issues, and answering a pile of mail… well, larger than my pile of gold When I recover my nerve, I’ll return to work disambiguation and groups.

Location search. The search page now allows you to search the locations users have entered on their profiles. As with everything else, providing your location is totally voluntary, but many users have entered on. It is also unstructured, rather than a zip code or something like that. For that reason, you may find it necessary to search once for “Massachusetts” and again for “MA.” Here are the results for my home-away-from-hometown, Boston—39 users!

This feature is fun now. It will be more fun when I get forums, and users can swap bookstore suggestions, etc.

Archiving comments. I don’t know about you, but I have a million comments.* So I added an “archive comments” link, so you don’t need to delete a comment to get it off the page. There a link to view your archived comments.

I’m weighing revamping the profile page, to put the comments on a separate page. A even bigger revamp would be to provide LT users with an ersatz reading blog. Reviews could—at your discretion—appear, as could independent blog entries. I’m not sure if this would be on the main profile page or not. Probably not. Reactions?

* I beg pardon for still working my way through these and the emails occasioned by the crash. I find it physically impossible to answer comments for more than a few hours at a stretch.

Before the crash I did complete one new feature, the accurate parsing of Library of Congress Catalog numbers.

LCCNs are slippery things, without a fixed number of digits or a single internal structure—”89-456,” “89-7890,” “2001-1114″ and “gm 71-2450″ are all perfectly okay. For this reason, LibraryThing was sometimes mistaking them for ISSNs and ISBNs, helpfully adding hypens and checksums digits to reinforce its misundersanding. Worse, the Library of Congress’ online catalog doesn’t allow searches by printed LCCNs. Instead, you have to turn them into a special machine-readable LCCN format, adding leading zeroes and removing hypens and spaces as they dictate (see here.)

Anyway, I worked through the tangle, and LibraryThing now handles LCCNs well, identifying them as such and converting them to the format the LC’s requires*. I can tell you from entering 180 of my own books this week, that LCCNs come in very handy with older books, many of which have their codes printed on one of the first pages or even the back cover. Of course, I lost all those books. But I gave myself a free membership, so we’re even.

*The one exception are LCCNs with “alpha prefixes.” I never found one of these in the books I cataloged, but the LCs catalog page says they exist. To parse them, I need to know the range of possible prefix. Would, say, 1-4 alphabetic letters plus a space cover it?

Crash update. Thank you again for all the support you’ve given LibraryThing. I do appreciate it. I am still working through the hundreds of blog posts, profile comments and personal emails I received. Thank you for your patience.

Cover orphanage. Users who lost books also lost cover images. But the cover images were not lost, they were just “orphaned”—stranded without a “parent” book or user. Between Feb. 2 and 7 some 1,100 covers were added, many of which are now orphaned.

I have created a page showing these orphans. The page is at http://www.librarything.com/coverorphanage.php . The page shows the covers in reverse chronological order. If you added covers one after another, your books are probably clumped together on the page. If you lost 2 covers, I wouldn’t bother. If you lost 50, you might find this useful.

To use one of the covers, right-click (or option-click, depending on your OS and browser) to get the image’s URL. You can copy the URL into the “change cover image” page.

Help for orphaned tags? As some users have pointed out, tag edits made on the crash days are there but “invisible.” Basically, this means that the tags are not printed in the catalog, but if you click on a tag, like “cooking,” it will still list all the books once tagged cooking. Some users have used this feature to power edit their tags back in.

Yes, I could redo the visible tags based on these “invisible” tags. The trick is this: The “invisible tags” have no guaranteed order and they have no date. So, if I did this everyone’s tags would lose their prefered order, even if they were tagged long ago.

I need to remove this inconsistency, either deleting all the invisible tags or making them visible, order be damned.

Some possible solutions:

• Delete all invisible tags after a week. Before then, enterprising users can play with power editing the tags back.
• Add a special field that shows the invisible tags, allowing users to edit them back into the visible tags.
• Allow users to choose to replace all their official tags with the invisible ones en masse. Users who did this would understand that their tag order might not be preserved. Perhaps I could force an alphabetic order, something some users prefer anyway. (Other users are vehement against that.)

Suggestions welcome, here or at the LibraryThing Google discussion group.

UPDATE: Many books found, but not all. Crash problem solved. See end of this post.

If you’ve been following the blog, you know that LibraryThing recently moved to a new, much better server. This move has made LibraryThing fast again, but the transition has been rocky. Yesterday, a glitch erased some recently-entered books, and the new server’s backup failed.

The losses are from books added in the last four days (ie., since the server switch). Some books added since then were recovered from “bin” files. And all books added via import have been recovered, but only as a re-import, so any subsequent edits to those books have been lost. The site will go live this evening, once the import queue has finished and I’ve sent messages to users who lost books (see below).

Here are the steps I am taking for users:

• If you lost books, the system will send you a profile message indicating how many.
• Users who lost more than 25 books receive a free membership or membership upgrade. You can use it on your own account or gift it. Check your profile for information.
• All members who lost more than 25 books and paid in the last 30 days can receive a full refund—thirty days is the period PayPal gives me to do this. Send me your user name and either the email PayPal knows or the PayPal receipt.

For the site, I and LibraryThing’s database administrator are doing the following:

• The import feature has been changed to screen out ISBN duplicates. This means that you can upload an exported file without duplicating or “writing over” books.
• A second server (the old one), which is housed separately, will act as a constant “live” backup, getting all the same information that the primary server does.
• The nightly backup has been fixed. (The new server’s didn’t have the right “permissions”).
• I will shortly improve the CSV export and add a tab-delimited export.

Although a “beta” project is subject to occasional lapses, this sort of problem is unacceptable. Those who have been affected have every reason to be angry. Since I am at present synonymous with LibraryThing, this is my fault. I offer you my apology. I will do whatever I can to make it right for you.

The person who has been helping me with the database offered the pithy remark “this sort of thing never happens twice.” It’s not that lightning never strikes twice, but that once you’ve been hit you stop bringing stainless steel cutlery to picnics. LibraryThing’s new backup regime—nightly backups and an enterprise level “hot backup” at a different location (in fact, on opposite sites of the continent)—will go a long way to ensuring future data security.

Thank you for your understanding.

UPDATE: Comments posted to this entry noted that some missing books predated the server transfer. This was a good catch. At the same time, the server could hardly stay up five minutes. The problems were related. (And related to the previously-mentioned crc32 change.)

I found all these older books, substantially reducing the number of users affected. (Over 100 users can keep their free accounts although they have fewer than 25 books missing—most have none.) As for books entered Monday night, few users managed to enter any, but those that were entered were lost—the crashes were just too severe. All this did have one salutary effect—I figured out the big problem. I think it will stay up now.

Update: By twiddling something, LibraryThing helper database guy, Austin Swinney (profile), appears to have fixed the character issue. (The sort order is still Swedish—literally—but that’s close to correct and I’ll change it the next downtime.) I’m not sure what’s happened to changes made in the last day. Can people send me problem instances?

So, LibraryThing got a new server. It’s a good deal faster than the previous one. Although there have been some slow-downs while I did work—when I can, I prefer to leave the site up—average speed has increased substantially for most users. I’ve also turned back on some of the features I dimmed, eg., book-by-book recommendations. Yesterday some books were getting confused—thinking they were other books—but the problem was resolved.*

Here are some kinks I’m working through:

• Accents and foreign scripts have been affected. The server move also involved an upgrade from MySQL 4.1 to MySQL 5, and I’m still straightening out the character-set changes. For what it’s worth, it would probably be better if you didn’t try to fix errors yourself, since they’ll just become errors again when I fix the system.
• The library-suggestion page is still down. I’m going to bring it back based on a different, better algorithm, which also says WHY a book is being suggested.

My next update will be to get the book/work issue resolved. This will go hand-in-hand with a redesign of the book-info pages. Here is a nonfunctioning sample for Charlotte Brontë’s Jane Eyre. Look at how many different editions (covers) people have!

*For techies, crc32 in PHP is dependent on system architecture, so the move from a 32- to a 64-bit machine changed all my work numbers. Arg!

I have to take LibraryThing down tonight at 11:00 EST (8:00 PST). I’m swapping in a new server, and there’s just no way to do that “on the fly.”

I don’t want to over-promise, but I think it will help. (It’s currently calculating book recommendations, at a rate of 500/minute!) More important than any short-term speed gain, it’s set up in such a way that I can snap on additional servers at need. LibraryThing has a lot of different sorts of database interactions—from catalog-display requests to highly complex social calculations. Some, like widget-serving, must never wait ten seconds. Others, like a complex all-library word search, can. Ultimately, all these different requests should not be running on one machine, and one that also serves all the web pages, etc.

Anyway, it’ll be down from 11:00 tonight until some time in the wee hours.