Some quick updates:

• The site has been slow, and today there were some server glitches. Can you hear the hamster wheel squeeking?
• A new, much better server is coming tomorrow, or possibly Tuesday. I think it will be a significant improvement. Whether it is or not, the new setup is much more flexible and expandible.
• I spent the day working out how to disambiguate editions. They’ll be big changes in that area soon.

I just want to put something up to express profound disapointment. Google, a company I’ve believed in for years, has decided to give into censorship in exchange for better access to the Chinese market. Google will get a .cn domain, and searches in China will now exclude anything the Chinese government wants.

Google hasn’t gone as far as Yahoo yet—forking over user information that lands dissidents in jail—but with today’s action I wouldn’t put it past them.

For shame, guys. For shame. I thought better of you.

Links: CNN, Boston Globe/AP, Google Google group.

PS: To think, people were congratulating them for refusing to give anonymized user searches to the US government, as if that were a serious issue. (“I support freedom!” one soon-to-be-duped fan wrote on their support group.) Censoring search results in exchange for a little money? That’s a real issue.

As some of you have noticed, I’ve added a “Fun statistics” link on every profile page. Here’s mine. It presents statistics like:

• Library obscurity (the average number of copies of the books held by others)
• Total tags; tagged books; tags per book
• Cataloging sources
• A histogram of ratings
• A histogram of publication dates

There’s a lot more I’d like to do. Right now these numbers aren’t contextualized. Is my “library obscurity,” 12, high or low (it’s quite low). A percentile, and maybe something on the Zeitgeist page would be a good idea.

I’m very open to other ideas. I can add read-date statistics, for those of you using those fields. Ditto BCID fields. I could do tag-obscurity, for what that’s worth. I’d like to work with page counts; I have the data for some books and can get it for others. In theory I should be able to extract publishers from ISBNs (I can do it from the publishers field too, but that’s too messy). But extracting publishers from ISBNs requires a big database, and I think it’s one I need to pay for, from Bowker I think.

Some other ideas would require a more serious integration with library data—language, date of original publication, etc. Nicole on the LibraryThing discussion group suggested “some kind of histogram of the distribution of your books on the LoC classification
system.” That’s a good idea too.

LibraryThing has a handful of competitors. I’m not too worried about them. LibraryThing wins on features, as many blog comparisons have shown (eg., the Powells blog). And LibraryThing certainly wins on size and traffic. The most frequent comparandum started the same week as LibraryThing but has 4% of the books. Another has 20% as many, but it’s been going for more than four years; it hasn’t broken the web’s top 100,000 sites in six months.

I’ve written to most of LibraryThing’s competitors, seeking a deal that would combine our sites, giving all their users lifetime accounts and bringing the developers on board as partners. So far, no dice. I guess it’s more fun to do your own thing.

A few days ago I came across a new site, Bokhyllan.com, a sort of Swedish LibraryThing. Between guessing-in-context (“LOGGA IN”), bad German and a second-degree connection with someone who knows Swedish, I managed to get the jist of it.

What a perfect situation! They’re just starting, and the site doesn’t have all the features it needs. LibraryThing has those features—even integration with the Swedish library system! But I have no desire to develop a Swedish LibraryThing, and no ability even if it did. It seems silly to do all this development multiple times. Why not combine forces and write the code once? Take the code, keep the domain, charge for it or make it free—I don’t care. But share your improvements and we all win.

I wrote. No answer so far. Maybe this plug will get their attention.

If anyone knows of any other non-English LibraryThings, let me know. If you want to start one, let me know. More generally, if you want to help LibraryThing grow, let me know.

There’s a more general conversation here about open-sourcing LibraryThing. I guess I’m a sceptic that it could work. At a minimum, a whole separate site would need to be set up, with fake data. Security would also be a big issue. If it didn’t work, it would be a big waste of energy. If it caught on, I feel like my job would become cat-herding programmers. In the worst case, LibraryThing would end up fragmented into a half-dozen forks.

Maybe we could discuss open sourcing on the LibraryThing Google Groups group. The group is growing in greatness—must keep using words starting with g. If possible, I’d like to keep blog comments on the blog.

All requests eventually come true. Searching has been improved, replacing the old “relevance” search with a more Google-ish search.

The search page now look like:

The main improvements are:

• Searching for “Greek history” now gives you all the books that have the phrase “greek history,” not some lame weighing of books that have just one or maybe just have a word close to “greek.”
• Book searches can use syntax like +greek -history
• You can now search LibraryThing for authors and tags. Instead of dropping you into a catalog with X-thousand books by J. K. Rowling or tagged “science fiction,” these take will take you to the LibraryThing page for J. K. Rowling and science fiction.

Still to be improved:

• Book searches are still ordered by relevance, which means they can’t be sorted by author, title, etc. I’ll fix this soon.
• You cannot currently search the entire 1.5 million books for a single book. I’m going to add this, but add it in a way that you get the book-page, not a catalog page loaded 10,000 copies of Harry Potter. The main problem right now is speed; I don’t want to introduce any more speed-hogging features.

Of course, you can search individual user’s libraries. To go their catalog and click the “search” link. But this doesn’t seem enough to me. Perhaps profiles should include a search box, or the search tab should have some way to select a user to search. I’ll tell you that I don’t want a menu with 21,000 people in it! Ideas taken.

Update: LibraryThing is still mostly up, but sometimes slow. I am working on the server issue on a number of levels and hope to show some progress soon. This is one issue I will lick, no matter how large it gets. That said, if you’re David Pogue and writing up an article on LibraryThing for the New York Times, please don’t publish it this week!

LibraryThing has hit 1.5 million books cataloged and 2 million tags, all since August 29.

What used to be a straight line of 9,000/day has become a bending curve—gulp! Even so the site pretty-much stayed up today. The database isn’t out of the woods yet, but it’s in better shape. You’ll see some stutters, but downtime should be minimal and I’ll be getting a second server for some of the “thinking.”

Some other updates:

• Profile pages now show ten random books from the user’s library. People were doing this with widgets, but that presented some security problems and also looked bad with the new design. I will probably add the ability to customize what people see (most recently-read books, and so forth).
• Acquisition, started and finished dates have been added. This feature hasn’t been extended everywhere—eg., you can’t use it in a widget. That will come soon. You can enter dates in North American or European format, but the catalog currently only shows them in the righ—um, North American—way.
• I added a statistic to the Zeigeist page listing the “50 ‘completist’ authors.” These are authors that, if you have one of their books, you have a lot. Think people who collect every single Agatha Christie, although the actual authors may surprise you. To qualify the author needs to be read by at least 100 users. Come to think about it, I could do a statistic for whose library is the most “completist.” I am quite sure that a science fiction fan would win.

It’s up again. Knock on wood, I think it will stay that way.

In general, the system is stronger and more disaster-resistant. I am working with not one but two competent database administrators. There will be some glitches, but also significant improvements. I am confident that there is a clear path to scaling this thing up very large.

There are some changes, some temporary, some permanent. Most of the heavy-duty processing has been moved away from the page-display. So it does not calculate the book suggestions when you enter a page, as it did before—although it then cached them. Now it gives you the cached suggestions if there are any, and no suggestions if there aren’t. The suggestions are being generated behind the scenes when nothing else is going on. Soon enough all books will have recommendations again. Since they don’t do much “thinking” the book pages are much faster.

You’ll also notice that the profile page looks different. If it’s your profile you get a lengthy list of who shares books with you. If another person’s it fronts the list of shared books, something that was pushed down the page before. It doesn’t give you a list of that person’s list of top sharers. I’d plan to add these other-profile statistics back in soon, but they will be subordinated to the shared books list, which is, I think, the most interesting piece of information.

The profile page currently lacks links to user reviews and the more intensive similar-libraries calculation. These will be back as soon as I decide where to put them. I plan to make the profile a multi-page affair, moving the comments off the main page. Those of you who have dozens of comments will be glad of that.

That’s the news. Thank you for all your support. It was so supportive that I plan to make downtime a weekly event. 😉

I’m sorry about the downtime. I have reached something of an impasse. Something is just different, and I haven’t been able to figure it out what, nor my various helpers. Suggestions to increase normalization or change to a different db engine are really beside the point. There is some reason it worked two weeks ago and won’t work now. The number of books is not much higher. No new feature has been added. In fact, they’ve been taken away.

If there is anyone out there who is confident they could help—it’s a PHP, MySQL site—give me a ring. I’m quite willing to pay $65/hour for competent help.

10:06 – Widget requests are now going through.

UPDATE: Okay. It’s up. Let’s see if it stays up. I’ll blog about various changes soon. You’ll notice some missing recommendations. The site will go down at 3am EST.

I’ve got good news to report on the database front. I’ve made something of a break-through. On the nastiest queries—for example, determining what 20 users share the most books with you and how many they share—it now performs 500 times faster than before. You probably won’t see that sort of increase, but it will be noticeable. If you have more than 1,000 books, visiting your profile will become fun again, not an invitation to wait. And the changes will allow LibraryThing to grow far beyond its current 1,498,000 books.

Although theoretically simple, the changes need to be made dozens of places. And tested. I’m behind on the sleep department, so I may not be able to finish it this evening. It will almost certainly be up by 5pm tomorrow, spiffier than ever. Within an hour or so I expect LibraryThing will surpass both 1.5 million books and 2 million tags.

Thanks to everyone who send letters of sympathy, encouragement and offers of assistance. Incidentally, I love the idea for the down page to sport a falllen pile repair-themed books.

PS: Is it just lack of sleep that makes the phrase downtime update seem linguistically funny?

I’m taking LibraryThing down for an hour or so this morning, when you should be asleep anyway—you’ve got work tomorrow. I’ve made some changes that improve the speed issues; I hope this proves another step in the right direction.