Thursday, August 25th, 2011

New tag-based recommendations algorithm

Short version. I’ve just finished up a new algorithm for calculating book recommendations based on tags. You can see them on the “Recommendations” sub-page for every work page, under a special “Tags” heading (for a limited time only!), as shown to the right. When a work doesn’t have a recommendation, it will make it. (Expect to wait 2-10 seconds.) Recommendations will be propagating through the system, and, combined with the four other recommendations algorithms we use, going into your personal recommendations over time.

Come discuss it on talk here.

Long version. Recommendation algorithms are always tricky, but doing it based on tags alone is particularly difficult. Do you consider total overlap? Overlaps by tag? How do you rate a book missing an important tag? How do you factor up meaty and meaningful tags, and downgrade meaningless, over obvious or ephemeral ones? LibraryThing has long had tag-based work-to-work recommendations, but they were of uneven quality. I haven’t been making new ones for a while now, and letting them die out of the system; all the old tag-based recommendations have been removed.

The new algorithm approaches the issue afresh, looking at all of a work’s tags, and taking into account various factors that can trip it up. It thinks about factors like tag salience on a work and generally, degree of agreement between tags and low-value tags. It also attends to similar levels of work popularity. After deciding on the basic algorithm, I toyed with various “knobs” for days, myself and with Jeremy, trying to get the best results for a set of sample and problem works. In my experience, you can’t do an algorithm like this without a sense of appropriateness, taste and proportion, and (I hope) that this is one reason LibraryThing recommendations are generally so good.

Once a tag-recommendation is generated, it takes a while for it to be incorporated into the “Combo recommendations” above. “Combo recommendations” incorporate tag recommendations to greater or lesser degrees, depending on its assessment of their quality and contribution. Your personal recommendations are based mostly on these “Combo recommendations.”

The tag recommendations are going to take a while to build for all works that need them. After that, we plan to do some sort of “Pepsi Challenge” test. We think LibraryThing recommendations are as good as any out there, and are eager to prove it.

Some examples. Tag recommendations work absolutely best on non-fiction titles about something very simple and clear cut:

Works really “about” two or more things are harder, as are books with a specific point of view, which might seem separate to some extent from the tags on it. Some examples:

  • PHP and MySQL for dummies by Janet Valade — A successful example, where most of the books deal with both PHP and MySQL, with an orientation toward “entry level” programmers (eg., the “Visual Quickstart” books)
  • Born Fundamentalist, Born Again Catholic by David B. Currie — A successful example that mostly recommends other Protestant-to-Catholic conversion stories (rather than Catholic-to-Protestant ones)
  • Goldwater by Barry Goldwater — A mixed but mostly decent result, surfacing some Goldwater-specific material, but also showing other biographies, especially from the period and involving senators. Ideally it wouldn’t have quite so many contender-bios, as I’m not sure the potential Goldwater reader is eager to dig into Edward’s bio.
  • Freakonomics by Steven D. Levitt and Stephen J. Dubner — Decent list, starting with a short-shelf of popular economics-in-life books, followed by popular introductions to economics and economics-oriented thinky-think books

Fiction, especially non-“genre” fiction, is the big problem. Literary fiction isn’t “about” what it’s about in quite the same way that non-fiction is. Creating some separation between adult and youth titles is also hard. But we’ve made significant progress.

  • Watership Down by Richard Adams — A decent list of animal-centered chapter books, with classics like Redwall and The Wind in the Willows and no The Runaway Bunny or Knuffle Bunny.
  • The Book of Three by Lloyd Alexander — A decent list of magical fantasy books, angled to youth and toward Welsh mythology.
  • The Hobbit by J. R. R. Tolkien — A good list, but obviously centered quite strongly on Tolkien. Works with significant secondary literatures (cf., Harry Potter, Narnia, etc.) tend to be dominated by guides, atlases and so forth.
  • Little Women by Louisa May Alcott — Starts out well, putting March second, but then goes off the rails somewhat after a dozen. The Mother-Daughter Book Club wins because it apparently takes place in Concord, Massachusetts and involves mothers and daughters. The Secret Life of Bees is also about mothers and daughters, coming of age, sisterhood, and was made into a movie. Meh.
  • The Book Thief by Markus Zusak — Decent list, with other books, mostly fiction, about children during the Holocaust.
  • The Shack by William P. Young — I haven’t read it, but I think have a sense of it. There are some winners here, like The Christmas List, Redeeming Love. The Year of Fog and The Deep End of The Ocean cover some of the same issues from a non-religious standpoint, and Where Is God When It Hurts covers them from a non-fiction, evangelical perspective, which might or might not be wanted. But C. S. Lewis and Paul Bunyan(!) aren’t winners, being rather different sorts of fictions. Tim LaHaye is winning almost solely on being “christian fiction,” and James Redfield for “religious fiction,” “inspirational,” etc.
  • Patient Zero by Jonathan Maberry — Zombies? We got zombies, focusing on plague-based zombie terror (no Pride and Prejudice and Zombies). Straight-up plauge titles, like The White Plague are also included. Preston’s The Demon in the Freezer probably shouldn’t be there, but it’s a great book.
  • Earth Abides by George R. Stewart — More apocalypse, this time with few zombies. The Brief History of the Dead is rather different, though I’m not sure how LibraryThing could know that. At least it doesn’t attempt to recommend other boring books, like Earth Abides.
  • Blueberry Girl by Neil Gaiman — Some good stuff. Sound lousy. The tags are dragging the recommendations all around—children, picture book, Gaiman, mothers and daughters, charles vess. Twinkle, Twinkle, Little Star is winning on “poetry,” “rhymes” and child-associated tags.
  • Love in the asylum : a novel by Lisa Carey — My wife’s book. A decent-list of insane-asylum fiction, with some memoirs.
  • Twilight by Stephenie Meyer — Largely okay, so far as I can tell, although with less secondary literature than I would have guessed.
  • Illyria by Elizabeth Hand is winning on tags like “forbidden love,” “teen romance,” “teen lit,” “romantic” and “contemporary fantasy.” I have no idea if the books are similar.

Come discuss it on talk here.

Labels: recommendations, tagging, tags

Thursday, August 25th, 2011

LibraryThing in embassy libraries

Did you know that the U.S. State Department helps organize and maintain libraries around the world? They are set up as Information Resource Centers at embassies and consulates, and as American Corners (partnerships between embassy Public Affairs sections and local host institutions). In Afghanistan, they’re known as Lincoln Learning Centers.

LibraryThing offers free lifetime status for these accounts, and so far more than a hundred diplomatic libraries have begun to catalog their resources using LT. Just a few of those include:

Other American Corner libraries getting their LT catalogs underway recently include a whole bunch in places like Khazakhstan, Serbia, and Fiji.

Just one of the many different uses being made of LibraryThing around the world!

Labels: cultural library, libraries

Monday, August 15th, 2011

Welcome Brian!

Welcome Brian Stinson (LT member tabashco), our new systems administrator: the person who keeps the servers running, plans expansions, monitors performance and protects your data.

Brian describes himself as a city kid from Witchita, KS (and writes “Before you ask, I’ve never met Dorothy, and I couldn’t grow some wheat to save my life but the Sunflower State will always be home”). He earned his BS in Computer Science from Kansas State University, where he’s soon to begin a graduate program in Political Science. Brian will be supported by the rest of the LibraryThing staff, who have become much more familiar with the systems side of LibraryThing since John informed us of his departure.

Brian likes C-Span, running, reading, college football, sledding, and listening to campus radio. His favorite authors include Ernest Hemingway, Cory Doctorow, Arthur Conan Doyle, and Mark Twain. You can follow him on Twitter at @tabashco.

Labels: employees, servers, sysadmin

Wednesday, August 10th, 2011

New: Search your groups and connections

We’ve added a new cross-library search feature. You can now search:

This opens up all sorts of possibilities: you and your family members or friends can create a group together and easily search across the all the books in your libraries, or start a neighborhood group*. You can look for interesting books within a given group. For example, Tim enjoys searching for “Alexander the Great” in the Alexander the Great group.

Be creative, and if you do something really nifty with this feature, make sure and tell us about it!

Come discuss it on Talk. Many thanks to members of the Board for Extreme Thing Advances for help developing this feature.


* I’m already seeing Tim combining this new feature with the “what should you borrow?” recommendations so that he can plunder my bookshelves!

Labels: connection news, features, groups, new feature, new features, search

Tuesday, August 9th, 2011

LibraryThing still interviewing great PHP hackers

We’ve managed to fill our systems-administrator position. (We’ll tell you about him when he starts on Friday, but I can report he hails from the the Sunflower State.) But we’re still actively looking for 1-2 great PHP hackers. Applicants can be from anywhere, though Portland, Maine applicants get free coffee during their interview.

If you’re interested, check out this blog post. System administration skills are no longer a primary need, but they can’t hurt either.

If you want to test your brain a bit, check out the LibraryThing Programming Quiz. We’re asking all applicants to take it, and tell us how they did.

Labels: employees, employment

Monday, August 8th, 2011

August Early Reviewers Batch is up!

The August 2011 batch of Early Reviewer books is up! We’ve got 97 books this month, and a grand total of 2,219 copies to give out. This month’s batch includes new books by Matthew Pearl, Neal Stephenson, Charles Frazier, and Alison Weir!

First, make sure to sign up for Early Reviewers. If you’ve already signed up, please check your mailing address and make sure it’s correct.

Then request away! The list of available books is here:
http://www.librarything.com/er/list

The deadline to request a copy is Tuesday, August 30th at 6 p.m. EDT.

Eligiblity: Publishers do things country-by-country. This month we have publishers who can send books to the US, Canada, the UK, Israel, and a whole bunch more. Make sure to check the flags by each book to see if it can be sent to your country.

Thanks to all the publishers participating this month!

Bloomsbury Orbit Books Taylor Trade Publishing
Henry Holt and Company Double Day Religion Putnam Books
Wave Books Mulholland Books Telegram Books
Saqi Books Ballantine Books The Permanent Press
Chronicle Books Orca Book Publishers Sterling Publishing
Divine Design Tundra Books South Dakota State Historical Society Press
WaterBrook Press Atomic Fez Publishing Sagebrush Press Bookstore
Eirini Press Gray & Company, Publishers Penguin Young Readers Group
The Writer’s Coffee Shop Eerdmans Books for Young Readers Quirk Books
St. Martin’s Griffin Skyhorse Publishing Crossway
Nolo Kensington Publishing Pomegranate
Scribner Indigo Ink Press Barbour Books
William Morrow Wilderness Press Menasha Ridge Press
Beacon Press JournalStone Prufrock Press
Gotham Books Totem Books Penguin Group (USA)
PediNatural (TLS Kids Books) Human Kinetics Avon
Alder Hill Press Cinco Vidas Random House
Spiegel & Grau Coffeetown Press Lee & Low Books

Labels: early reviewers, LTER

Wednesday, August 3rd, 2011

Import your LivingSocial/Visual Bookshelf books and save your reviews

LivingSocial announced last week that they’ll be shuttering their Books section, formerly called “Visual Bookshelf,” as of August 10.

Living Social recommends migrating to another site, but if you do it you’ll lose all your reviews. Living Social splits your data up into multiple files and only the main one can be imported, the “collections” file. We’ve cooked up a better option that preserves your reviews as well.

As a bonus, anyone importing 25 books or more will get a lifetime account on LibraryThing!

Here’s how to do it:

  1. Go to http://www.librarything.com and click the “Join Now” button to create a LibraryThing account.
  2. Sign into your Living Social books account; you’ll see the large black box in the center of the page with three export buttons (Collections, Ratings*, Reviews).
  3. Click the “Export Collections” button and save the file to your desktop; then click the “Export Reviews” button and save that one too (they’ll be called “collection_books_….csv” and “review_books_…..csv”).
  4. On LibraryThing, go to the Import Books page (via the “More” tab at the top of any page on LibraryThing).
  5. Click the “Choose File” button and choose the “collection_books” file from your desktop, then click “Upload.”
  6. On the Import Options page, select the options you want and then click the “Import books” link at the bottom of the page.

To add your reviews:

  1. To add your reviews: once your titles have finished importing, return to the Import Books page and click “Choose File”; select the “review_books” file and click “Upload.”
  2. Under “Handle duplicates” at the bottom of the Import Options page, select the radio button for “Sync ISBN duplicates, importing user data,” and tick the “Replace Reviews with imported info.” Then click the “Import books” button and your reviews should import right into your LibraryThing catalog.

Come talk about it on Talk. And if you have any problems email tim@librarything.com.


*Ratings: The Living Social ratings export option doesn’t seem to be working at the moment. If it starts working in the future, we can work through adding it too.

Labels: features, import

Tuesday, August 2nd, 2011

Find your friends on LibraryThing

LibraryThing had a “friend finder” back before they were ubiquitous. But we’ve lacked one for a while. So we’ve just released a handy new Friend Finder for LibraryThing.

Friend Finder allows members to connect easily with their Facebook and Twitter contacts who also use LibraryThing, or invite their friends to join the site.

New members will see this as an (optional) step as they create an account, while current members can access the Friend Finder on the Edit profile and settings.

For people already on the site, Friend Finder gives you a one-step way to add them to your friends, interesting libraries, contacts and so forth. These work just like adding them from their profile page. For invites, we take a typically respectful approach: no invitations will be sent without your explicit consent and you have to send them one at a time—no spamming everyone you know. You can change the wording of the invitation before you send it. Twitter messages are posted @ your friend. Facebook messages are posted to your friend’s wall.

Come talk about it.

Labels: new feature, new features

Thursday, July 28th, 2011

Goodbye John!

Goodbye John (Felius), LibraryThing’s long-time sysadmin.

John’s been great to us. He took on a system under severe scaling strain, going down all the time and held together with string, and he sized it up and made it reliable. He moved the whole system from Portland to Boston, and made it both safer and faster (example, example). After almost four years with LibraryThing, John is moving on to Engine Yard, a Ruby-on-Rails cloud-hosting provider. His work and his company—John was a lot of fun to chat with at night—will be sorely missed. John promises to hang around as a member. He’s been one since 2005—long before we hired him.

We finished hiring John’s successor. More news soon.


PS: John managed to time his exit to System Administrator Appreciation Day. Believe me, we appreciate ‘em.

Labels: sysadmin, systems adminitration

Thursday, July 28th, 2011

Feature: Where did you get your books?

By popular demand, I’ve added a feature for members to keep track of where they got their books. “From where?” takes two answers, either venues from LibraryThing Local, including bookstores and libraries, or “free text.”

You can find the feature:

By default, members’ “From where?” data is public—unless their account itself is set to “private.” If you want to record the information but not share it, you can do so. The option is available on your account page and when you add a “From where?” location under “Privacy.”

Come talk about it here.

Labels: new feature, new features