Update: The server migration appears to have gone off without a hitch—anyway, LT’s had of the usual random, corrupting crashes since the changeover. I am a little behind on email, but will be—amazing to say—out of touch today.

The “Users with your books” box on user profiles shows how many books you—or any user—share with other LibraryThing users. Unfortunately, it counts all books equally—Harry Potter as much as something rare. And there was no dampening of big libraries, so everyone had the largest library, ellenandjim, near the top.

LibraryThing used to have a page that munged your “shared books” in various ways. The algorithm was, however, very inefficient, so I had to drop it somewhere around 1 million books. I’ve brought it back. It better than ever and it wont clog the server (another side-benefit of the new “works” system).

You can see the new feature by clicking the “weighted” link in the “Users with your books” box on your profile. It takes account of both book obscurity and library size. It really works for me, anyway, sifting to the top a number of users I’d never seen, but who share some of my favorite stuff. Try it out and tell me what you think.

PS: The 2:30am EST downtime is still on.
PPS: Feel free to chime in on this topic. My first task in the next week or so is to work on bugs and infelicities. After that, should I work on a “groups” system or a “forum”? A groups system would, among other things, allow a group of friends, a club or other association to easily search a bunch of libraries. There would also be group profiles and so forth. A “forum” feature would bring interactive, mutli-person discussion to LibraryThing. It would be very closely tied to the work, author and tag system, not just being “another place” to discuss books. (It would, of course, have a place to discuss bugs too.)

LibraryThing will go down at 2:30am EST / 11:30pm PST / 7:30 GMT for a major swap. The “big new server” I got some months ago has proved very fast, but also glitchy. My database guy thinks it was some interaction between the OS FreeBSD and MySQL. He’s made a Linux server—exact same hardward—that seems never to crash under similar stress. That sure would be swell. Everyone cross your fingers!

Oh, and don’t worry about your data. Obviously the change-over will be backed up six ways from seven.

LibraryThing just hit 1.8 million books and 2.5 million tags. Since we’re going to hit 2 million books soon, I’ll talk about the tags today.

As the tags accumulate, they are also generating a lot more value. Tags are mostly useful personally and statistically. Tags are often played up baselessly—as if a few scattered and general tags are of any use to anyone. For statistical purposes you need a LOT of tags, so frequency patterns can emerge and anomalous entries fade into the background. And tags are primarily interesting in concert, not by themselves. Because tags are non-heirarchical and often short, they lack the “context” of something like the Library of Congress subject headings. Other tags can provide that context.

That’s why the “tag similarity” algorithm takes many tags into account, favoring recommendations that match on more than one. Take the messy example of a mid-level book, T. E. Lawrence’s Seven Pillars of Wisdom. What the heck is that? Its all over the map—literature, WWI, Middle East, Ottoman Empire, Arabia, history, autobiography, memoir, etc. The recommendations try hitting many of these tags at once—books like Fromkin’s A Peace to End All Peace (WWI, Middle East, history, Ottoman Empire, etc.) and Robert Grave’s Goodbye to All That (literature, memoir, WWI). It’s not perfect—Edward Said’s memoir!—but it’s a hell of a lot better than any single tag could produce.

And, most importantly, every book and tag makes the statistics better.

Lastly, I wonder how LibraryThing’s 2.5 million compares. I’m sure Flickr and Delicious have many times that number. But what else is out there? Amazon has encouraged product tagging for about three months, and they have thousands of times the traffic. I wonder how well that’s going?

I wanted to take a second to highlight an interesting use of LibraryThing. LibraryThing user _Celeste_ collects editions of the Clement Clarke Moore‘s “A Visit From St. Nicholas,” better known as “The Night Before Christmas.” Putting her collection online helps her—and the friends and family members who scout for her—keep track and avoid duplicates. In addition to putting her collection online, _Celeste_ has also added her own covers. (Needless to say, most of her copies are not available on Amazon.) Arrayed together, they are a pretty cool sight, and a monument to one collector’s dedication. It would be great if more collectors put their collections and covers on LibraryThing. Old covers have a low profile on the internet because nobody has much of a financial stake in them, and there isn’t anywhere central to “put” them. LibraryThing can be that place.

Check out The Night Before Christmas for most of the editions. Scroll WAAAY down to see her covers. Not all the editions have been combined into the master “work” (not should they), so also check out her dedicated tag and the books in her catalog. Great stuff.

The library came to my attention when Celeste reported problems with her 100 copies sending “shared book” stats through the roof. I’ve revisted how these are calculated. Profiles now list both the number of works and the number of books, if different. I’ve also brought back the “Shared books” box for all users’ profiles, not just yours.

UPDATE: I changed the way I load covers. Is anyone still getting a “stack” error?

In the last three days I’ve added a slew of new features, and a new structure to support future improvements:

“Works” : user-controlled book combination and separation

LibraryThing would be pretty useless if connections, ratings, comments and recommendations considered every edition of something like Pride and Prejudice a separate thing, as other online cataloging applications do. In the past LibraryThing made a guess, like Amazon does, and failed about as often. Library technologists are starting to do somewhat better, but there was no perfect answer. It was time to try something new.

Starting three days ago, I announced a trial project to let users determine what books belonged together, the first time anything like this has been attempted. Using simple check boxes, users could go through a favorite author’s works, combining and separating editions as necessary.

The response has been startling to say the least: In three days, users have combined 17,000 times, mashing together 42,000 works! Users have spent hours at the task, and debated the nuances in a blog post that now sports 182 comments. Although only a few of these Christmas elves are actual librarians, but most are experts on the authors they labor over. As one wrote on the blog, Isaac Asimov’s Nightfall the short story collection, is distinct from Nightfall the novel and from Nightfall One. Do libraries know that? Does Amazon?

As with tagging, reviews, ratings, uploading scanned covers and other user content, LibraryThing users are taking book information into their own hands. And they’re doing it because it’s fun and they see the benefit right away—filling out the catalog with covers and cataloging data they wouldn’t otherwise have, and connecting them with like-minded reasons, even if the person who’s read every Asimov book they have did it in Finnish.

Book pages revised

Book information pages (the card-catalog, people and pencil icons) have been enhanced in a variety of ways.

• Social pages show all the work’s covers, by popularity, including user-contributed ones. (Note: Some Windows IE machines fail to load all the images. If the book has very many covers, this can show an error. I am working to solve this.)
• Each edition links to Amazon, Abebooks, Alibris and other online merchants.
• Editions also link to the OCLC’s “Find in a Library” project.
• Users can now swap covers easily, and for the first time snap up covers uploaded by other users.

New Recommendation Options

The new “works” system has opened things wide up for improvements. One of the first is an enhanced “recommendations” engine, now based on the “full work,” not just some of its editions.

The new engine shows both “People who own X also own Y” and “Similarly tagged” books. The “raw” option shows “People who own X also own Y” without any weighting applied (J. K. Rowling rules!). The “exclude author” option is useful when a book triggers ten or fifteen suggestions by the same author, as happens with authors like Agatha Christie or Stephen King.

Search

LibraryThing now allows library-wide title and author searches based on the new work system.

Deweys and LC Call Numbers for everyone!

Until now, members who found books through the Amazon search (most of us), had no access to Library of Congress Call Numbers and Dewey Decimals, which only came through library searches. The new “work” system leverages everyone’s cataloging, bringing true library data to 175,000 works, including most popular ones. These numbers now appear in your catalog, in green to distinguish them from your own numbers.

In addition to Deweys and LC Call Numbers, card catalogue pages now allow you to browse works’ MARC records, a Matrix-like stream of data librarians are said to be able to decipher, and even enjoy looking at.

Forward

The new features have gotten a workout since starting testing three days ago, and users have been very helpful with bug reports and suggestions. Some systems are still transitioning, and some problems remain, but issues are being knocked down one-by-one. In my defense, developing LibraryThing is usually like working on a train while it’s in motion. The recent changes were like turning a train into a monorail while it was in motion…

Some anticipated improvements include:

• Fixing all occurrences of that MSIE6 “stack overflow” bug, and the pesky double-frames issue
• A definitive statement on when translations should be combined (I’m working on it!)
• Work disambiguation—there, I said it—available through the search system
• Improved author disambiguation
• Deweys and LC Call Numbers for all books, not just books in the system
• Browsable LC Subject headings for all and sundry
• A space-based laser to smite people who write in library books

Comments, suggestions, criticisms, complaints and bug reports always welcome.