So, I’ve added one RSS feed—recent books from your or someone else’s library. You’ll find the feed in users’ profile pages, marked with the familiar icon. The feed shows the last twenty books entered, linking to the book’s catalog page. The “description” field includes the user’s review (if there is one), their tags and the books publication data.

I made one feed to test the waters, and to provoke comment. So, what else do you want? I suggest:

• A feed of someone’s recent reviews
• A feed of someone’s recent books, but restricted to a given tag
• A feed of others’ review of books owned by someone (so you can track reviews of books in your library)

What else makes sense? Also let me know if you want the format changed, for example to include different data in the “description” or restrict it to ten books.

Another suggestion: I’d rather have a single page with feed buttons and maybe a way to create just the feed you want. I’d rather not be strewing orange buttons all over. Am I a fuddy-duddy?

PS: Forum is coming.

LibraryThing users officially cataloged over a half-million books. I would be stunned if my capacity for that emotion hadn’t been destroyed at 100,000 books. One million books by Christmas or bust!

I’m still waiting for the mainstream U.S. media to notice LibraryThing. If you agree, blog us. And tell your friends and neighbors, particularly if your neighbor is David Pogue, Walter Mossberg, Xeni Jardin or Hiawatha Bray. What’s up with tech reporters and kick-ass names anyway?

In other news:

• I returned from a tech conference in Boston, so I’m on LibraryThing 24/7 again. There were a few days there when no new features were added; can’t have that!
• The forum at BookCrossing has discovered LibraryThing. If any blog readers are also Book-Crossers, I’d love to hear how you think LT and BC can work together.
• The book pile contest is still open, mostly because the prizes are all free memberships and I haven’t built that feature yet… Flickr‘s got most of them posted. Great stuff.
• LT needs a forum. I think I may do one of my Mothboards. They’re linear; I really hate threaded discussions. And I want something that doesn’t look like the inside of a spaceship. On the other hand, a simple board would preclude people having open-ended discussions about books (as opposed to LibraryThing). But aren’t there enough places for that?
• Strange LT Meme: Phantom Scribbler wrote “Do any other Library Thing users feel lonely when you see that you’re the only one who owns a favorite book?” and suggested people blog about their “onlies.” So far, only No Fancy Name has taken the bait.

Universal Import is now truly universal. It accepts both files and web pages. I’ve successfully tested it with:

• Delicious Library, Readerware, Book Collector
• Amazon (Wishlists, Listmania, past orders), Barnes and Noble, Booksense
• Bibliophil (export or URL), BookCrossing, Reader2, Listal, What Should I Read Next
• Home-brewed text files
• Mumbling ISBNs near your computer, rotary telephone or toaster

See the post below for more on how it works. Again, it won’t fetch your comments, the date you bought something or track down books without ISBNs, but it should do most of what you want most of the time. If you have problems, be specific about them. Go ahead and send me files and URLs.

Let me know if you end up drawing books from a site I haven’t mentioned. I’m keen to add it to the list.

I’ve added a “Universal Import” feature. After wrangling with a dozen or so different formats, I chucked the nonsense and made a single Swiss-army-knife import. Universal Import works on:

• Desktop applications like Delicious Library, Readerware, Book Collector, etc.
• Online services that offer exports (eg., Bibliophil)
• Home-cooked text-files, spreadsheets and databases

For each one, it grabs the ISBNs and looks them up against the libraries you specify. The upside is the data is fresh, top-quality and drawn from wherever you want—from Amazon to libraries in Turkey. The downside is that it only grabs the ISBNs. It doesn’t try to wrangle all the other stuff.

This was not done lightly. Individual filters take a long time to build and require all sorts of compromises. LibraryThing users clamoring for imports are distributed among a half-dozen applications and various home solutions. So, instead of making 5% of my users 100% happy, I decided to make 100% of my users 95% happy.

I hope you like it.

Coming tomorrow: Imports from web sites like Amazon and AllConsuming!

I’ve added the ability to upload profile pictures, not just link to one somewhere else. Go to your profile and click “Edit your profile” to do this.

If this works reasonably well, I’ll also let people upload book covers. This will be restricted to paid users to avoid porn spam. (Also added to the terms.)

Alert LibraryThing-ers will notice the return of the divet ( ) when adding books from libraries. Click it and get a lot more information about the book: ISBN, publication and physical description info, and sometimes even summaries and tables of contents. It’s a small change, but actually a sign of the skeleton poking through—the database and parsing changes necessary for adding Italian libraries and bulk imports from Delicious Library and other desktop applications. (My efforts to get the desktop book cataloging people to discuss synching—or really to even talk to me—have failed. It’s time for import filters.

Contest update: So far, only one person‘s sent in a book-pile photo (a good combo). Anyone else? I figured someone would want a free membership!

400,000. LibraryThing passed 400,000 books cataloged. At current pace we’ll hit 1/2 million when I’m in Cambridge, MA. I think that calls for a Scorpion Bowl at the Hong Kong Restaurant, don’t you?

I’m getting very tired of my pile of books. It’s a bad pile, done in haste. But I’m seeing some fun “LibraryThing” photos on blogs and Flickr (my apologies for all the disarray LibraryThing is causing!). Wouldn’t it be fun to get more people’s library photos?

Therefore, I announce the first annual LibraryThing Book-Pile Photo Contest.

The rules:

• Take a photo of a pile of books—something like the home page photo.
• Some of the spines should face the camera so the titles can be read.
• The books should be against a white or light background, so I can snip them out of the background easily in Photoshop.
• Express your cleverness in the book-choice. Do not include yourself. I also think you should avoid football trophies, two-handed swords, etc, but go ahead and include dogs and cats. Don’t let the books fall on any babies.
• By submitting the image you agree to allow LibraryThing to use it, as well as anyone else so long as they’re talking about LibraryThing or the contest.
• Send original or otherwise large files to timspalding gmail.com, or just post it on Flickr and send me the URL.

The judging: If I get a lot of good stuff, I will probably do some sort of voting thing on LibraryThing. That would be fun. If not, the winner is whoever I want.

The benefit: Winner gets two free memberships and eternal glory. Two runners-up get one free membership each. Yeah, pretty pathetic.

Users like Wyvernfriend (299 reviews!) are finally getting their due. Although reviews remain—and will remain—secondary to cataloging, they are now a bit more “out there.”

Check your profile for a reviews page, listing the books you’ve reviewed (if any). That page also lists the books that others have reviewed but you haven’t. Over on the Zeitgeist page, I’m now listing the most prolific reviews and the most-reviewed books.

In other news, you’ll notice that clicking on another user no longer automatically sorts the books according to whether or not you share them. Instead, the catalog tells you how many books you share and gives you a special link to see them. The processing time (comparing all your books against all someone else’s books) was really slowing LibraryThing down. Indeed, you should find things a little faster overall.

A kind user sent me this inspiring October 2 LA Times story, “This Illiterate Brazilian’s Home Speaks Volumes,” about a poor Brazilian who, basically, started a library in his house.

I’d love to find somewhere to point people so they can donate. I’m sure even small donations would go a long way. If nobody official is collecting—some library association perhaps?—I’ll set up a collection page here.

There’d be nice symmetry there. LibraryThing has seen a lot of Brazilians since being profiled in two papers there.

It’s official: LibraryThing has expanded to thirty-one libraries in thirteen countries.

Not only Australians and Canadians now have major collections in their countries, but also Turks, Danes, Swedes and Dutch. US collections like Yale and the University of Chicago add more scholarly heft, and a user reported that the University of California system is excellent for paperback science fiction.

In addition to the libraries, LibraryThing also works with all the Amazons—now including Amazon Japan. All the national Amazons can also be accessed using the LibraryThing/Amazon bookmarklet, which allows you to add books to your LibraryThing library while browsing on Amazon.

European diacriticals now work well, albeit with some library-specific glitches (eg., the Australian National Library strips accents out and the Royal Danish Library sends the strangest character set). But internal searches still have some diacritical problems…

More libraries

LibraryThing will continue to add libraries. Unfortunately, not all libraries present open web interfaces. Here are global and UK lists of some of the libraries that may work. Some, like the BL, use a format I haven’t designed for yet. Feel free to suggest items off the list, or other open Z39.50 libraries you know about.

Other news

• LibraryThing will hit 325,000 books in about an hour. No, I’m not staying up for it.
• I’ve been trying to promote the site to the mainstream media. It seems unfair that it’s been profiled in two Brazilian papers, but I can’t get the Boston Globe to pick it up. Sheesh, I should start reading the Herald.
• I’ve been looking at similar sites, and pleased to discover LibraryThing’s user growth rate matches the hot social software site 43Things (see here, reporting 12,000 users in two months; LT did 6,000 in one month). 43Things has a bunch of employees and is bankrolled by Amazon (originally in secret).
• LibraryThing is not bankrolled by Amazon or anyone else, nor are we going to sell our data to them. (Amazingly, I never really announced that before. You are a trusting lot.)
• In related news, I just finished the site’s first Privacy Policy and Terms of Use. In the end, I decided against the naked-photography clause. Your pictures will be returned.

What I’m working on

• I’m taking a breather on new libraries. Maybe users will suggest some good ones. It’s hard to know what to add sometimes. I stopped when I realized that a particular Swedish word probably meant “veterinary.” Not a top priority.
• I’d like to add more data back into the search view. (The “little divet” that used to come up when you use the Library of Congress.)
• I should add RSS. I’ve been promising it for ages.
• Ditto power editing/tagging.

I’m going to blog tonight about a bunch of new features and the addition of some thirty libraries, now working well with language support. Before I do I wanted to ask if people are able to figure out the “more libraries” feature, and whether the libraries added to the experience. Do you use them? Is the user interface confusing? Do you want more ability to customize? Do you miss the little divet that used to give you the cataloging info for a book? (I do.)

So, the floor’s open for multiple-library issues…

LibraryThing has made another great leap forward. Until now LibraryThing drew exclusively on the Library of Congress and the five national “Amazons.”

Today ten more libraries have joined the mix:

• United States: Boston University, The University of California system, The University of Chicago, Yale University
• Britain: The National Library of Scotland, The London School of Economics, The National Library of Wales
• Canada: The Canadian National Catalogue
• Australia: The National Library of Australia
• Denmark: Det Kongelige Bibliotek

Searching for books should also be faster, both because connections are shared between users and because you can now shift libraries when one doesn’t respond well.

It is a starter list. The US additions are a strong start. I hope Canadians, Australians and Danes will be happy seeing their national libraries included. Brits may feel the absence of Cambridge, Oxford and the British Library—the latter two are open and will be added. The French and Germans were, I confess, slighted, although the Canadian National Library has a lot of French literature. And what can I say to the Brazilians who have flocked in such numbers after LT was profiled in O Globo? I’ve looked and I will keep looking. I have an open Z39.50-based library in Portugal, but it is either down or on the blink. There are also some private universities in Brazil that are said to have open catalogs. I will find something for you!

There are a number of ramifications to the change that aren’t yet resolved:

• Although I now have richer data and can populate additional fields, including series, language and edition, I have not yet exposed them to viewing and editing.
• You will note that newly added books lack accents. Look on the bright side—they don’t have the wrong accents.
• Fortunately, accents are coming. New books have a checkbox for “Update as cataloging data improves.” Keep this checked and accents will soon appear where once there was none. All I have to do is crack the obscure character set Marc records employ…

Cheers to all for another milestone!

Library-expansion is another day or so off, but librarians and other interested parties are invited to weigh in on what new fields I should provide.

“MARC” records open up a world of data. I can finally parse all names correctly, add secondary authors, strip “a”s and “the”s from sorting, separate out publisher, place of publication and date of publication, and even wring automated sense from “xix, 230; ill., 25 cm.”

I don’t want to go overboard. Library records have more in them than most users need. Who but an institution needs to distinguish between “Uniform Title,” “Title Statement” and “Varying forms of Title”? Who scans their shelves confused between Dan Brown (1964–), author of the Da Vinci Code, and Dan Brown (n.d.) the publisher of the 1704 tract Novum Lumen Chirurgicum vindicatum?

For starters I can dump publisher and physical info into the “publication” field. And I can put everything into a static “card catalog” field, as I currently do with Library of Congress data. But what details should I “pull out” and allow to be edited, sorted by, and displayed in catalog view?

My candidates are:

• Secondary authors, editors and illustrators (It is likely their role will be elided, except in the cataloging field)
• Number of pages (Arabic numeral pages only)
• Language

If you’re not a librarian, see Understanding MARC Bibliographic for Marc’s “commonly used” fields. Then thank you librarian for knowing this kind of thing AND dealing with people who come in and ask for “that red book, you know.” Incidentally, and taking that seriously, if a book’s “variant tile” includes such wishy-washy info as the spine-title (when different from the “real” title), why doesn’t it capture the dominant cover color? User hh219 appears to be doing that; perhaps it helps when looking for something.

Stay tuned for Oxford University’s collection, coming on line later today or tomorrow. Then I’ll start bringing other Z39.50 libraries on fairly quickly, in order of user interest, connection speed and record parseability (I’m trying to do everything with Marc records now). Oh, and they need to be open to anyone and available all the time. The Z39.50 world is fairly new to me, but there are apparently thousands of people involved in it.

If you’re a librarian and know what the heck I’m talking about, feel free to suggest candidates. For starters I’d like to add some large US university libraries, to fall back on when the LC is slow. Oh, and I need a shoulder to cry on: Marc21 is blithering nonsense!

Your complaints have been heard—Amazon gave it a thumbs up! You can now use your Amazon Associates id within a blog widget, and take back a small fraction of the $1/day all 200 or so blog widgets are making me. You don’t even need to have a paid account to run off with my haypence. That’s how generous I am.

(Frankly, I’d rather not to link to Amazon, but it’s a requirement if I use their cover image. Of course the text links go to your library or the book itself.)

People were complaining that the LibraryThing URLs were too complex and it was hard to find the URL for a direct link to your collection. From now on you can use URLs like:

• Your profile — http://www.librarything.com/profile/timspalding
• Your catalog — http://www.librarything.com/catalog/timspalding

The catalog one will “redirect” to a more complex URL, the server rules being extremely difficult and hazardous to play with while dozens of people are on. Your URL is also listed on your profile page.

Special thanks to a reader of the Brazilian newspaper O Globo for forwarding me their blurb. I never managed to find it on the website, but I have the original. After spending about 2 seconds puzzling over it, rusty French and Latin at the ready, I threw it into Google:

Sunday toast For who has some familiarity with the English, access to the InterNet and thinks about fichar its library. A small farm makes success that receives the heading from a book, searchs it in the Library of the Congress of the United States, captures its fiche and plays it in the archive of the customer. Librarything.com is called and was created by Tim Spalding, an American pc hacker with the feet in the classic culture. It is in the version Beta (with the risks that this means) and leaves favour for who wants to catalogue up to 200 books. For bigger libraries, it charges USS 10 for the limitless use of the instrument. In less of one month, librarything joined four a thousand users who ficharam 177 a thousand books. It gained news article in the “The Guardian” and the forecast of that somebody goes to gain money with this business. If it will not be Spalding, will be another person. The Library of the Congress is a colossus. Its catalogue has 28 book million in 470 languages. For example: 18 headings of Fernando Gabeira. (who to want to sapear, an acknowledgment: the instrument of not accepted search accents nor cedilhas.)

This is great, if largely from Brown’s Guardian piece. My only fear is that ten Brazilians will try to start their own LibraryThing, looking for that money. (The money ain’t happening, people. It’s nice not to feel guilty about searching for freelance, but this is not making me rich. Then again, I might be rich if I were in Brazil.)

One question: At least for me the “instrument” seems to work with “search accents and cedillas” (that part ought to be “the search engine does not accept accents or cedillas,” instrumento de busca não aceita acentos nem cedilhas). Am I the only one?

Users have now cataloged 200,000 books, and the pace is increasing day-by-day. Three quarters were cataloged in the last two weeks.

I expect LibraryThing to keep growing for some time. Not only do most users have a lot of books to go, but only a fraction of the world’s book-lovers have ever heard of LibraryThing.

Bloggers—Thank you for blogging LT; it would never have gone anywhere without you. Bloggers who haven’t blogged about it—What are you waiting for?

On a related note, LibraryThing will go down for scheduled maintenance at 2am Sunday morning EST. It will probably be down until 5am. When it comes up it will be ready for the next 200,000.

You can now switch to seeing your book covers displayed side-by-side together as if they were on a shelf. (On a shelf facing out, which no book-lover would do!) Check it out.

Some notes:

• This post was originally a request for QA help. I think all the errors can be traced to something which is now fixed. Before reporting an error, quit out of your browser and come back in. The “stylesheets” in your cache may be old.
• If you report an error, also tell me you browser and OS. If you’re feeling generous, also give me a screenshot.
• I will look into ways users can contribute cover images. I’m worried about it from storage, security and spamming angles. I might add such a feature, but restrict it to external URLs at established booksellers and publishers.
• The effect is something like what Delicious Library does, although Delicious Library improves Amazon’s images and puts them on a photo-realistic shelf. (Rectangles floating in space is about all I can handle right now!) For what it’s worth, I didn’t steal the feature from DL. I doubt he patented it, but I’ve got prior year prior art from an eBook project I worked on at my former employer, Houghton Mifflin.

I’ve made an Amazon bookmarket, a little bookmark you put on your toolbar. When you’re on Amazon and want to add a book to your library, click on the bookmarklet and it will add it to your library. Get the Amazon bookmarklet out the extras page.

I’ve added support for 13-digit UPC/EAN barcodes. Of course there’s never any reason to type a barcode—every book with a barcode I’ve seen prints the (shorter) ISBN above the barcode.
But I’ve received emails from people with barcode scanners that won’t make the transition, so this will be a help.

We’ve hit 175,000 books, with no slowdown in sight. The time has come for me to consider a book database of a quarter of a million, half a million, a million or more.

It’s time to ask for advice. Programming PHP/MySQL for thousands of records is just different from doing it for millions. (Even if the books don’t hit a million, the tag database will.) I’ve been reading everything I can on enterprise-scale database optimization, but I lack hands-on experience. There’s a million things I can do, but I don’t have a relative sense of their costs and benefits. It’s also hard to “test” solutions—just building the search index took an hour, during which time LibraryThing was completely unavailable.

So, I’m not too proud to ask! Praise, free accounts and gift accounts are available.

I’ve added a “true” search feature, or anyway a better one. Search is now available at all times, from a “search’ tab. You can search for books (pretty much all data is indexed), for just tags or for users.

For now it works well—searching is much faster—but I’m concerned it slows down adding records. Basically, every time it adds a book it needs to also update the index. As I’m currently indexing pretty much every field, this takes a little longer than I’d like it to. Let me know if you see the difference. And if you’re a MySQL guru, drop me a note and give me your take on all-RAM index and other options.

I’d like to add the standard bells and whistles (+/-/quotes, etc.), but I need to upgrade my database first. For now it just works work with what you give it.

(Excuse the exclamation points. I need to express my glee somehow.)

Check out similar libraries feature. It tells you whose libraries are most like yours, but not by adding up the number of shared books. That method is distorted by common books and by large libraries. For example, you don’t share all that much when someone has Harry Potter—everyone has Harry Potter! It’s much better to be the only two people sharing “The Joy of Scottish Lacework.” Similarly, it doesn’t mean much that you share 10 books with oakesspalding—he’s got 2,100 of them.

I didn’t want to just toss up a number, so it computes it four ways. First, it adjusts for book obscurity. Second it adjusts for library size. Third it combines the factors. And fouth it gives you everyone who makes the other three lists—your library doppelgängers, if you will.

Let me know what you think. It works wonders in my case. Zette‘s library is huge and very similar. Oakesspalding‘s library is much the same (although the fact that we give each other books for Christmas distorts it). Languagehat shares many of my interests, like Greek and even Hittite. CaveatLector is the best find—he has only 106 books so I would have otherwise overlooked him, but the nine books he shares with me are all obscure treasures.

Let me know if it works for you, and which of the measures is the most accurate.

LibraryThing now provides good support for the familiar—to many—non-English diacriticals such as é and ñ and gamely handles the exotic—to many—ones like č and å. The encoding is now “UTF-8” so, in theory, is should be able to handle a much wider variety of scripts, but neither Amazon nor the Library of Congress transmit material in non-Latin scripts, so you’ll have to type it yourself for now.

LibraryThing is now ready to start adding foreign libraries, and that is high on my list of goals. I shall also see if I can get Amazon Japan working.

Everything should be good going forward. The trick was getting all the old material into shape. LibraryThing-ers had employed all manner of tricks to get diacriticals to work, and some of that editing was hard to detect and correct for. Again, French, Spanish and German books should be fine, but I’m not so sure about the Georgian books languagehat entered. (In this case, my Mac simply won’t show Georgian, so I couldn’t tell anyway.)

The other gap is material entered between 8pm and 1am. I shut down book editing and tried to process everything coming in, but at almost 1,000 books/hr, a delay of thirty seconds between processes may have left some books “half cooked.”

I’m spending the day on alphabet-support issues—getting all those diacriticals to work. Doing this requires some database and programming changes, some of which need to go through various steps, so you may see all your diacriticals go bad one minute, then correct the next. Pay no attention to the man behind the curtain.

Shiver me timbers, LibraryThing hit the Yahoo “new and notable” page (http://dir.yahoo.com/). In 1995 this would have been HUGE. I’m not sure anyone checks it anymore, but maybe they do. It’s below and to the right of the main entry, “Talk Like a Pirate Day.”

I did something technical that should eliminate long waits for the Library of Congress “session” to begin. Basically everyone now shares the same “session.”

Let me know if problems increase or decrease.

UPDATE: Or if it explodes, as it did! I guess the LC limits requests by session id. Sorry about that and thanks for taking the hit.

Aside from having thousands of copies of Harry Potter and no exhibitionists in the stacks, this is getting to be a decent sized community library!