Wednesday, November 1st, 2006

Recommendations refreshed, improved

I just completed a rather extensive regeneration of the book recommendations—the “people who own X also own Y” recommendations and the “similarly tagged” ones. (The “Special Sauce” is next.) Recommendations, in turn, affect the “Pssst!” system (recommendations based on your whole library) and involved recreating all the work-to-tag clouds. Quality has, I think, risen significantly. We’ve improved the algorithms and the non-stop growth of the LibraryThing data set—now at 6.8 million books—continually improve the results anyway.

Scope has also improved. The system only makes recommendations for works with more than ten copies. That comes to juse 93,000 out of 1.3 million works. But these works account for over 60% of the books in LibraryThing. Before the current re-do, only 50,000 works were covered.

The recommendations are better, but hardly perfect! We’ve made progress toward better algorithms, and have big plans for future improvements. On the UI side, we’re going to introduce users-feedback on recommendations, including both thumbs-up and thumbs-down buttons, and an “obscurity knob” for Pssst! (People love that on Last.fm.)

The system works best in the “middle,” on books with 25-500 copies, non-fiction, genre fiction, books with a well-defined readership, and books that are “about” something—books like Touching the Void, Prozac Nation, The Historical Figure of Jesus and An encyclopedia of fairies. It has the most trouble with bestsellers, “obligatory books” (think high-school classics), and literary fiction—books like the Da Vinci Code and Great Expectations. To some extent, the problem is almost philosophical. What would be good recommendations for the Da Vinci Code? Statistically-speaking not much stands out. At the high-obscurity end, there are of course books on the book and its themes—Cracking Da Vinci’s code or Baigen’s Holy blood, holy grail.—but most Da Vinci Code readers aren’t interested in that stuff. Ideally, we’d like a better mix of suggestions—some obscure stuff, and some of the quick, high-popularity reads it correlates with.

Burying the lead? We figure we’re a month or so away from getting the algorithms where we want them. Once they are, we’re going to start making them available to libraires, to spice up their online catalogs with top-notch readers’ advisory—and for much less than they’re currently paying for inferior services. We think it will make quite a splash!

NEWS: Tag-combining is back!

Labels: 1

0 Comments:

Leave a Reply