Introducing SocialCatalogers, a Ning-based social network for people who make social cataloging sites. No, this is not a joke. It’s funny, but it’s not a joke.

If you make social cataloging sites, or have a deep interest in them, join up here: http://socialcatalogers.ning.com.

Social cataloging has exploded. Today LibraryThing’s list of competitors—very broadly defined—hit forty.* That’s not counting the dozens or even hundreds of sites in other niches—movies, games, comics, programs, wine, beer, recipes. It’s not counting the swap sites, which catalog as a means to something else. Or the list sites like 43Things and Wordie, which catalog intangibles.** Or projects like John Blyber’s “Social OPAC,” which bolts social cataloging onto “unsocial cataloging” (a service LibraryThing will offering soon too).

For a while the social cataloging social network was me. I started LibraryThing by trying to get Bibliophil to join forces with me—they would provide the social, I’d provide the cataloging. No dice. Since then I’ve emailed or met with half the social cataloging developers out there, looking for synergies or just to talk shop. Sometimes something came out of it; LibraryThing has “also on” integration with a few, like Cork’d, “LibraryThing for wine.” (Some day, if we have enough shared users, LibraryThing can recommend books based on the wines you drink!) There should be more of that.

The time has come for a socal cataloging watering hole. We’re an industry now, or a dwarf industry anyway. Some of us compete, but that doesn’t prevent automakers from getting together. We too can conspire to fix prices! Seriously, we ought to have some things to talk about. At the very least we can keep an eye on the competition.

I made the social network on Ning, which relaunched today. Ning*** is a “social network maker,” started and mostly funded by Mark Andreeson. I wasn’t that impressed with it, until the relaunch. It’s really something now. I was able to create a basic social network in about ten minutes. It’s not what I would have designed, but I would have taken a month to do it, at least. 60% in ten minutes beats 100% in a month every time.

Also, by getting in early, SocialCatalogers hopes to become the dominant social network for people who make social cataloging applications. Take that CatalogingSocial.com, SocietyofSocialCatalogers.com, SocialCatalogingThing.com, ThingSocialCatalogers.com, SocialCatalogersList.com, SocialCatalogersster.com, Joptwix, Flipto, Gropo and Fhtagn****.

*I found StashMatic, which is similar to Squirl and iTaggit. (Squirl is my pick, and I’m not just saying that because half the development team now works for LibraryThing.) And I found JunkLog, which brings minimalism in social cataloging to a new level. That’s not really a knock. It’s kind of cool to strip it down. I can’t tell if it’s developing or defunct.
**Early on in LibraryThing I was at my parents house for the weekend, and my dad came into my room at 5am, fresh from bed. He had an idea he was dying to tell me about. He had an idea—LibraryThing, but for people! Instead of cataloging your books, you list your friends. I let him down easy. (Still, a more catalogy social network would be an interesting project.)
***Mostly because it got so much press but has LibraryThing-level traffic. Then again, I’m Dan Quayle to Andreeson’s John Kennedy. I expect the relaunch to kick Ning into the clouds.
****As every lover of H. P. Lovecraft knows, Fhtagn comes from Ph’nglui mglw’nafh Cthulhu R’lyeh wgah’nagl fhtagn! (“In his house at R’lyeh dead Cthulhu lies dreaming”). Fhtagn.com, .net., .org., .de and .edu are taken. However, phngluimglwnafhcthulhurlyehwgahnaglfhtagn.com is still available, even if phngluimglwnafhcthulhurlyehwgahnaglfhtagn.net is not.

Over on Next Generation Catalogs for Libraries, NCSU‘s Emily Lynema, asked me:

“Do you have any idea of the coverage of non-fiction, research materials in LT? Have you done any projects to look at overlap with a research institution (or with WorldCat)?”

No, we haven’t. And I’m dying to find out, both for academic and non-academic libraries.

So I put together a feed of all unique LibraryThing’s ISBNs. With a little work, library programmers should be able to compare them against their holdings.

If you’re not up to the task, but still want to find out how LibraryThing compares to your library, you can send me a file with ISBNs—just ISBNs or a more detailed dump—and I’ll do the comparison.

See our Feeds and APIs page for the file, AllLibraryThingISBNs.xml.gz.

Complications and opportunities:

• I included only valid ISBNs.
• It’s a week or two old.
• About 20% of LibraryThing books have invalid or no ISBN. Many of these have LCCNs. I suspect a high percentage are library-ish books.
• I have turned all ISBN-13s in 978 format into ISBN-10s. There are a few bogus ones too, including the valid but numerically absurd 0000000000. (Bowker should auction that one off!)
• There can be little doubt that LibraryThing is stronger in paperbacks and weaker in the formats libraries collect. It would therefore be very useful to run all ISBNs through OCLC’s xISBN service*. (By definition, they’re not going to be improved by running them through xISBN’s chief competitor alternative service provider, thingISBN.) Unfortunately, I can’t run them through xISBN on my own.
• The feed is available for non-commercial use only. That basically means libraries and hobbyists. Other use is expressly prohibited.
• I am guessing the overlap won’t always be that impressive as a percentage. But these are the books people think enough of to own. They’re going to move more than other library books.

I’m looking forward to what people find out!

*Which is moving, but will not break.

I parsed the English Wikipedia looking for ISBNs and came up with a public feed of ISBNs to articles.

Over on the LibraryThing blog, I posted about it. LibraryThing is now showing the results on work pages, but others should feel free to use the feed for their own ends. (Cause trouble. Put Wikipedia in your OPAC!)

Enjoy.

The O’Reilly Radar Blog does a nice post on my When tags work and when they don’t: Amazon and LibraryThing.

O’Reilly blogger Brady Forrest notes that much of what I said echoed Joshua Schachter of Del.icio.us:

“You have to understand the selfish user – user #1 has to find the system useful or you won’t get user #2. Systems that only become useful when lots of people are using them usually fail, because there’s no incentive for people to contribute themselves.”

So now would a good time for me to say I’m not claiming my insights are all my own. Most of this stuff has been in the air for a while. While I’d never seen that piece by Schachter, I’ve seen similar statements by him and others. (And there’s a good short panel taped with him and David Weinberger.) I’d love to talk tags with these guys. No doubt like Wikipedia founder Jimmy Wales, Schachter’s so swamped he’s not even answering Bono’s email (source: net@night).*

A few corrections:

• AllConsuming is already owned by Amazon. That is, Amazon now owns two of LibraryThing’s competitors!
• Shelfari allows tagging. Honesty hurts.
• As good as “CEO” sounds, I’m only the president. LibraryThing is an LLC.

I’m loathe to take the last post, When tags work and when they don’t: Amazon and LibraryThing, off the top. It got a *lot* of attention*, and I owe the commenters a follow up.

The Shifted Librarian found this YouTube video “put together by Steven Reed’s students at Wilmington High School.”

It’s a fun video, no question. It’s an *amazing* demonstration of what kids can do these days. My highschool had the best Super8 program in Massachusetts, and this level of professionalism would have been way past our capabilities. The book-throwing is great. The editing is quick and professional. The kids get an A. They rock.

But I can’t leave it at that. The kids are rock stars, but the message is all wrong—and it’s wrong in a very telling way.

The situation is completely false. I don’t mean the ninja—they’re increasingly common in libraries of all sizes—but the contest and its results.

Type Capital of Russia into Google and you get this:

You don’t even need to GO to a page—the answers are in the page titles themselves. Face it, the Web is *great* for this sort of thing. You’re not going to “defend” books by claiming they’re better for looking up trivial facts. They’re not. Breathe deep and repeat after me: They’re worse.**

The second false idea is that libraries and the web are rivals, two competing ways to get the same thing (which is mostly factoids). This is all too often how popular culture sees libraries, and it’s a disaster. If libraries are just low-tech search engines, they are bad ones. They should be defunded and closed.***

I’m not going to launch into a defense of libraries and books. Of course I love them. I started LibraryThing because I loved them so much. But I don’t love them because I hate computers, or because books are better than computers.**** I don’t see them as rivals. The web has supplanted a few things that books used to do, but not the important ones. And libraries can do things with computers they are only just starting to explore.

People who love books need to fight against these ideas. They’re a trap. They’re wrong, and they’re very dangerous to the things we love.*****

Yeah, I know, lighten up, Tim!

*Alexa is under the impression LibraryThing’s traffic doubled the day the post hit. That’s total nonsense and a great proof of Alexa’s failings. It makes me wonder how much they rely on new link creation, not traffic.
**Where did http://www2.blogger.com/img/gl.link.gifthat guy find the books? Does he have the shelving memorized, or did he consult and OPAC to find Countries of the World?
***And don’t tell me it’s about not everyone having a computer. If so, libraries should be just computer centers.
****For starters, books are worse at email, worse at social networking, and they are hands-down a lousy way to blunder upon shocking new types of pornography.
*****Can anyone help me find a quote? I think it was from random sci-fi movie or show, taking place in the near future. The quote was something like “Would you have wanted to shut down the internet just to keep the libraries open?” Don’t even try to Google it. (And I recommend not going to the library either.)

Update: This ninja movie has a good message.

This is an extensive post, revealing the results of a statistical comparison between Amazon and LibraryThing tags, and exploring why tagging has turned out relatively poorly for Amazon. I end by making concrete recommendations for ecommerce sites interested in making tagging work.

Both LibraryThing and Amazon allow users to tag books. But with a tiny fraction of Amazon’s traffic, LibraryThing appears to have accumulated *ten times* as many book tags as Amazon—13 million tags on LibraryThing to about 1.3 million on Amazon. (See below for the method I used to find this out.)

Something is going on here—something with broad implications for tagging, classification and “Web 2.0” commerce. There are a couple of lessons, but the most important is this: Tagging works well when people tag “their” stuff, but it fails when they’re asked to do it to “someone else’s” stuff. You can’t get your customers to organize your products, unless you give them a very good incentive. We all make our beds, but nobody volunteers to fluff pillows at the local Sheraton.

A tale of two tagging sites.

LibraryThing began on August 30, 2005. From the start, we allowed members to tag their books. We showed that people could embrace book tagging, much as they had photo and website tagging. But LibraryThing was a marginal player.

Three months later, Amazon unveiled its tagging feature. This was big deal in certain quarters. To many, Amaon’s move signalled that tagging had “arrived.” As CNet blogger Daniel Terdima wrote:

“[This] may well prove to be the most visible example of a company incorporating tags as a way to bring order to information. Outfits like Flickr are big and have tremendous followings, but nothing compared to Amazon’s.”

Amazon’s size was key. With something like 60 million registered customers, and one of the highest traffic sites out there, tagging at Amazon must have seemed like a sure bet. It’s visitors were a firehose. Point them at tagging and KABOOM!

Amazon’s tagging was quick and easy—but would it work?

It didn’t work out that way. Amazon visitors have not taken to tagging Amazon’s books in significant numbers. With thousands of times the traffic, Amazon produced a tenth as many tags as LibraryThing. What’s going on?

In fairness, Amazon didn’t give tagging a lot of prominence. Tags were stuck in the middle of their ever-lengthening book page—one section for adding your own tags, another for showing others’ tags. They didn’t push them very hard.

It’s likely Amazon could have done better. A higher profile could have increased familiarity and comfort with the feature. Some user-interface tweaks could have enhanced its appeal. Maybe Amazon will make changes, and Amazon tags will get some traction.

But there’s a general message in this: If Amazon with its unsurpassed traffic is having trouble, can other ecommerce sites hope to make tagging work?

Numbers matter

Amazon’s shortfall matters. To do anything useful with tags, you need numbers. With only a few tags, you can’t conclude much. The tags could just be “noise.”

A web of meaning: LibraryThing’s tag cloud for Guns, Germs and Steel.

Take one example: LibraryThing users have applied over 3,900 tags to Jared Diamond‘s Guns, Germs and Steel, including “apples,” “office” and “quite boring.” With just a few tags, it might be thought a desert cookbook, a business book or—worst of all—a boring one. But these are all single-instance tags. With a larger number of tags, clear patterns emerge, with high-level descriptors like “history” (755 times) and “anthropology” (293 times) standing out clearly against the noise. Even lower-frequency tags, like “social evolution” (25 times) and “pulitzer prize” (20 times) can be trusted as relevant.

Large numbers are particularly important when looking for best examples for a given tag. Go by numbers alone and you just get what’s popular. By the numbers Guns, Germs and Steel, tagged “evolution” 39 times, is the number ten book on evolution. That’s crazy. By looking at “tag share,” LibraryThing can understand that Ernst Mayr‘s What Evolution Is is a better choice. Although tagged “evolution” only 25 times, those constitute a much larger percentage of its tags. (See the LibraryThing tag page for “evolution.”)

Critical mass is important, even if we can’t pinpoint the line. Ten tags are never enough; a thousand almost always is. Unfortunately, Amazon’s low numbers translate into a broader failure to reach critical mass. With ten times as many tags overall, LibraryThing has fifteen times as many books with 100 tags, and 35 times as many with over 200 tags.

ISBN tag distribution for A Farewell to Arms. Doubles as an example of my Excel-fu.

The “problem of small numbers” is compounded by Amazon’s failure to aggregate tags accross editions. In Amazon, the tags for the various paperback, hardback, British, French and German editions of a work are all in separate “buckets.” LibraryThing’s unique user-generated “works” concept combines editions and their data, compounding tag statistics. Thus, Amazon’s top edition of A Farewell to Arms has 28 tags, where LibraryThing’s has 716. But when all of LibraryThing’s editions are combined together under a work, LibraryThing has 1,914 tags—68 times as many! A Farewell to Arms is a very well-known book, but Amazon’s 28 tags can’t mean much. With 1,914 tags, LibraryThing has a truly extensive “web of meaning,” created by its members. You can do a lot more when the data is so rich.

Why Ecommerce Tagging Fails

Amazon’s tagging suffers a failure of incentive. The causes are multiple:

• Tags work best when they’re about memory, so tagging makes the most sense when you have a lot of something to remember. On LibraryThing, members with under 50 books seldom tag, but users with 200 or more usually do. When you get right down to it, few of us need to remember 200 books on Amazon. For most of us, the “wishlist” feature is good enough. We don’t need to sub-segment out the “anthropology” books.
• When you tag on LibraryThing, you’re putting your library in order. The pleasure and use is not unlike reshelving your books the way you want them, except that tags can draw together books that must otherwise reside separately on the shelves. And tagging on LibraryThing is connected to a social system
  —tag something “anthropology” and you’re connected to all the other anthropology buffs out there.
• Amazon is a store, not a personal library or even a club. Organizing its data is as fun as straightening items at the supermarket. It’s not your stuff and it’s not your job.
• Amazon underplays the social. Tagging really kicks into high gear when the personal blooms into the social, when organizing your web pages or your books turns into an hours-long exploration of others’ web pages and books. But Amazon doesn’t want you to hang out—they want you to buy! Tags on book pages do not list their taggers. You need to click around a lot before the tags turn into people. (The failure is particularly surprising in light of Amazon’s clear grasp of social software. Amazon got “social” years before it was trendy. What are reviews and Listmania but social sharing and user-generated content?)
• Users don’t “own” their tags. There is no way to export them. Considering how central APIs are to Amazon—and to it’s success—this comes as a surprise. (I’m guessing they’ll add this eventually.)

The problem of opinion tags

Some of the tags from Ann Coulter’s Treason. But what is it about? (Compare with LibraryThing’s page.

The limited utility of tagging on Amazon produces an unintended consequence—a surfeit of “opinion tags.” So, Daniel Silva’s The Unlikely Spy gets “wow what a book” and Nick Hornby’s High Fidelity gets “good” and “good book.” Not infrequently, opinion outnumbers other types of tags. Five of the seven tags applied to Bette Green’s Summer of My German Soldier are opinion tags, incluing “aweful” (sic), “obnoxious,” “pathetic,” “stupid” and “wonderful story.”

The takeover is total with political books. Ann Coulter’s Treason gets a lot of tags like “craptacular,” “evil” and “brain dead.” Coulter’s tag-defenders weigh in with “you won’t disprove the facts,” “you can’t disprove the facts,” “no one has proven this book wrong” and “try and disprove this book.” (Well, I guess that settles it!) Finally, Coulter has also received “dildo” (elsewhere applied mostly to Bill O’Reilly books*), “vibrator,” “lunesta” and “xanax.” It seems the naughty teenagers and the pharmaceutical spammers have discovered Amazon tags!**

Tag-spam on Amazon

Amazon’s all-items tag cloud shows the impact of partisan tagging. After “DVD,” “Music” and “fiction,” the largest tag is “Defectivebydesign,” applied by a small, pitchfork-weilding mob of Microsoft DRM haters.***

Ultimately, I don’t care about the commercial side of things, but “opinion tagging” in a low-numbers environment holds commercial risks. The Summer of my German Soldier is actually a pretty good book (I hear). Although Amazon won’t let me confirm it, one suspects all five negative tags come from one user. Is it fair to let one anonymous reader shape a book’s tag cloud so completely?

How to make Ecommerce Tagging Work

Big suggestions:

• Figure out why your customers would want to tag your stuff. Don’t fool yourself.
• Make tagging as easy as possible. (Amazon’s are quite easy to add, although registration is a pain.)
• Understand that commercial tagging can turn people off. Avoid crass commercialism. Respect your taggers—these people are helping you out!
• Make taggers feel like it’s “their” thing. Encourage users to give out their tag URLs—people love to show off—and let them export their tags any way they want.
• Keep tagging social. Stop selling and start connecting. If you connect people up right, the selling will follow. Think Tupperware!
• Consider whether a non-commerce site has the data you need. Back when LibraryThing had a million tags, Amazon could have bought our data for the price of a cup of coffee. Now, that we’re big and important and have three employees, that’ll be THREE cups of coffee, buster!

Small suggestions:

• Put methods in place to fight spamming and tag-bombing. LibraryThing does this by considering both the number of times a tag has been applied and the number of users who use it. A single angry user can’t make a tag really big on the tag cloud.
• Have logical URLs. Amazons tag URLs are full of junk, much of it rather crass attempts at search engine optimization (eg., the book title is inserted into the tag URL, but it works without it). It seems getting a little search engine help trumped providing users with easy-to-remember URLs.

Methods

To my knowledge, Amazon doesn’t release any total tag statistics. So I tried a statistical sample:

• I picked 1000 random entries from LibraryThing libraries, and retrieved their ISBNs.
• I ran the ISBNs through LibraryThing and Amazon, counting tag numbers. I did it by hand through 100 before I decided to write a quick scraper.
• I compiled the results and did some simple math. You can find my Excel file to the right.

The final results were 56,185 tags on LibraryThing, 5,528 on Amazon. Extrapolating on the sample, I conclude that Amazon has something like 1,337,388 tags in total, to LibraryThing’s 13,593,069.

If anyone wants to duplicate the test, let me know. By default, LibraryThing doesn’t think of tags ISBN-by-ISBN, so I’d need to give you an API to that data.

Problems with my method

• It only covers books. Maybe DVD tagging is a different phenomenon. I note, however, that Amazon’s page for bananas—yes, Amazon sells bananas—is overrun with Borat-themed tags.
• The random books were drawn from LibraryThing. Maybe LibraryThing’s ISBNs are unrepresentative of Amazon’s ISBNs as a whole—that the sort of books that are tagged on LibraryThing are not tagged on Amazon. There may be some truth to this insofar as LibraryThing includes a lot of older books, while Amazon focuses on the new and in-print.
• I only sampled Amazon’s US site. LibraryThing has a fair number of non-US editions.

Let me know what you think

As usual, I’m dying to hear what people think about this post. I know it’s imperfect—I bit off more than I could chew. But it says a lot of things I’ve been keeping in my head for months. Leave comments here. If enough interest develops, we can start something on Talk.

*Shouldn’t it be “falafel”? And YES! O’Reilly’s Culture Warrior IS tagged “falafel”! I swear I did NOT do it.
**Out of 60 unique tags applied to the book, I can spy only four that read like subject tags.
***Small numbers also mean Amazon is open to manipulation. One of the larger tags on their tag cloud is for “bards and minstrels,” applied to 4,200 products by six taggers. The tag has never been used on LibraryThing, Flickr or Del.icio.us. I suspect a conspiracy.

OCLC has introduced WorldCat Registry, a one-stop place for libraries, library consortia, library vendors, funders and suchnot to put contact info, link URLs and other “identity” data. Every institution gets its own page—it’s like MySpace but with libraries and minus friends, comments, tacky background images and all the drunken photos. Here’s LibraryThing’s page. We hate being a “vendor,” but there was no category for “The OCLC of Lilliput.”

OCLC is being generous with the entry requirements. Personal libraries* are out, but small institutional ones are not. Their FAQs note “no restrictions prevent a smaller physical entity such as a church library, or a ‘virtual’ entity such as a digital library, from representing itself in the Registry.” So, if you’re on an institutional membership, go ahead and take them at their word—join up!

When you join up, you can give your catalog URL as:

http://www.librarything.com/catalog/YOURUSERNAME

Your ISBN and ISSN URLs are:

http://www.librarything.com/catalog.php?view=YOURUSERNAME
&searchbox=ISBNORISSN&searchType=Books

LibraryThing is not currently listed among their vendors. Until they do, select “other.”

*Which reminds me, we recently had an application by a coven. They were uncertain if they were a family or an institution. OCLC is silent on the coven issue.

An alert LibraryThing member sent me Amazon email, a textbook case of collaborative filtering gone wrong. (LibraryThing makes these kinds of mistakes too.) The fact that it’s Borges, who has such fun things to say about how books relate to each other, is just icing on the cake.

Dear Amazon.com Customer,

We’ve noticed that customers who have expressed interest in Borges: A Life by Edwin Williamson have also ordered How She Does It: How Women Entrepreneurs Are Changing the Rules of Business Success by Margaret Heffernan. For this reason, you might like to know that Margaret Heffernan’s How She Does It: How Women Entrepreneurs Are Changing the Rules of Business Success is now available. You can order your copy for just $17.13 ($8.82 off the list price) by following the link below.

How She Does It: How Women Entrepreneurs Are Changing the Rules of Business Success Margaret Heffernan List Price: $25.95 Price: $17.13 You Save: $8.82 (34%)

This isn’t breaking news at this point, but it’s still cool. John Blyberg has announced what he calls “SOPAC”:

It’s basically a set of social networking tools integrated into the AADL catalog. It gives users the ability to rate, review, comment-on, and tag items.

Tags, ratings, and reviews in an OPAC! I think it’s great that he’s done it— it’s no surprise that we’d love to put some of LT’s features into OPACs, and to see a big library like AADL take on social stuff legitimates the point.*

Anyway, you should check out his post about it, which even has a nifty screencast.

*Tim has blogged before about putting LT and OPACs… We’re thinking it’ll be a sort of OPAC widget, so hold onto your hats.

So begins the PDF announcing and detailing a major new development for the library-data world. Simon Spero, library-geek extraordinaire, has released a nearly complete copy of the Library of Congress Authority Files.

Get them here:
http://www.ibiblio.org/fred2.0/authorities/

Simon assembled the files, available in MarcXML, by querying the Library of Congress’ Authorities website one-by-one over months. He’s a patient man.

As I’ve discussed before, Library of Congress data is both free and unfree. As a work of the US government, it cannot be copyrighted.* But the LC has traditionally restricted access, offering small amounts through public interfaces**, and selling larger amounts through its Cataloging Distribution Service. A small industry has developed where the CDS’s buyers resell it commercially. Until now, nobody has decided to just… let it go.

I anticipate that Simon’s action will draw some criticism. If the LC can’t make money selling its cataloging, how will it support this vital work? This sentiment will grow stronger when Casey Bisson releases the full LC Marc data, but whether for authorities or other cataloging data I think this is short sighted.

As I see it, the failure of the LC and other libraries to get their data “out there” on the open web has hurt them far more deeply than their catalog sales could ever recoup. It has made them seem irrelevant, standing silent and apart from the great conversation, which grows more interesting with each passing year.

The first culprits are the online catalogs***, ugly, backward things lamed with session-based URLs. If you want to link to the LC, you can’t. The URL you get will only work for you, for ten minutes. Linking–the very soul of the Web–is impossible.

The second culprit is how libraries have distributed the data itself. Amazon makes its book data accessible to all in a handy, universally-understood XML format. It’s so easy and appealing, over 140,000 developers have signed up to receive it. Libraries by contrast generally make their data available—if they make it available—over a tricky and obscure protocol know as z39.50. And the data itself is in MARC, a rich but impenetrable spectrum of formats—eg., DanMARC, the Danish MARC format!—used by and largely only understood by librarians.

With wretched web sites and unretrievable, unparseable data, libraries have lost vital ground. If the world worked right, Googling a book should turn up a library within the first few results. But libraries seldom make the top 100, and despite being the largest library on the planet and producing the lion’s share of original cataloging, the Library of Congress is completely absent. In its place are Amazon, its peers and sites that use Amazon data.**** Libraries may know a lot, but simplicity, attractiveness and ubiquitous data have won out.

It’s time to fight back. Libraries and library data can change the book web for the better. Three cheers to Simon for making a critical first step. Viva La Revolución, my brother.*****

*The LC reserves the right to copyright it outside of the United States. It’s unclear if they ever have.
**In LibraryThing’s case, through a z39.50 connection. Although the limits are not clearly specified, we’ve been given to understand that large-scale mining will not be tolerated.
***What library-techs called OPACs—Online Public Access Catalog. The fact that someone still needs to to add “Public Access” to “Online” is the problem in miniature. Does Google call itself a Public Access Search Engine?
****Don’t get me wrong; Amazon is a great site, and should be up in the top results too.
*****In so far as both Simon and I blogged the death of Milton Friedman, I suspect we’re equally uneasy with revolutionary Spanish.

Unless you’ve been on Mars, you have seen this. Chris Anderson put it best: “This is why I do what I do.”

I wrote this up on the plane from San Francisco. (I was there on a secret, unbloggable mission!*) It’s a bit involved and it doesn’t “arrive” anywhere, but, if you’re interested in subjects and relevancy ranking, it might be worth thinking about.

There are a couple differences between user tagging (“free tagging,” “social tagging,” etc.) and traditional library classification. “Who does it?” is the most obvious difference, followed by whether or not the labeling action takes place within a predefined ontology, or is made up on the fly.

It’s easy to ignore a third, and very critical difference. Subject classifications, like the Library of Congress Subject Headings (LCSH), are essentially binary. It’s non-overlapping buckets. Something either does or does no belong in a subject. There are no gradations of belonging.

The idea is, as Clay Shirky and David Weinberger have reminded us, rooted in the physical world. Subject classification escapes the physicality of shelf-order classification, in which a book must be shelved in a single place, but is still restrained by the physicality of the catalog card. A catalog card can only reference a certain number of subjects. Nobody wants a book to take up twenty cards. And the subject cards can only reference so many books. About 90% of all literature could fall under the LCSH subject Man-woman relationships. But it would make no sense to slot this 90% under that heading in a physical card catalog–the card catalog would instantly grow by 90%! And there seem to be very real differences in relevancy and “what-the-heck”-ness between real-life members of the “Man-woman relationships” LCSH: High Fidelity, Great expectations, The Fountainhead, I Kissed Dating Goodbye, and The Official Hottie Hunting Guide.

If you’re very selective, you can keep the numbers down. But, apart from the rule that the first subject is generally the primary one, there’s no good way to relevancy rank the books belonging to a subject.

Tags can do it, because tens, hundreds or thousands of users applying tags creates a “statistics of meaning.” So, 1984 is tagged dytopia 549 times, torture six times and Great Britain two times. The numbers can be turned into ranking, so 1984 shows up high on a list of books about “dystopia,” lower under “torture” and near the end of a list of books about Great Britain.

This is all well-worn territory. My question is this: Is there any way to relevancy-rank books within subjects?

I was reminded of the question when checking out OCLC’s new project, FictionFinder. I’ll blog about the whole later, but for now know that you can search for a LCSH subject and get back a list of books belonging to it. (I can’t link to the results, which are session based.**) Check out the LCSH “City and Town Life” and the top book is Red Badge of Courage. Lacking a better method, FictionFinder let popularity (the number of OCLC libraries with a copy) stand in for relevance. LibraryThing does the same, using our popularity numbers instead. The results are not systemmatically better (in this case Ulysses wins).

I tried two solutions:

The first was to tie into LibraryThing’s tags. So, figure out what tags are most characteristic of books with the subject “Man-Woman Relationships,” and then use the presence and number of these tags to rank the subject results. So, for example, “Man-Woman Relationships” has a global correlation with “relationships,” “dating” and “romance,” none of which are very prominent among the tags applied to Great Expectations, so it can fall low on the list.

I got far enough down this road to know it was going to help.

The second and more interesting algorithm was to see if books can be ranked within subjects without any other information. This would help OCLC, who are unlikely to pay for LibraryThing data, and to any library that employs LCSH, most of which would have no “popularity” data to use either.

I hit upon the idea that subjects “reinforce” each other, and that this must leave a statistical signature. For example, it seems that “Love stories” and “Psychological fiction” are commonly applied to books about “Man-Woman Relationships,” but that “Androgynous robot alone on an island — Stories” is not. (Okay, that’s not real, but the point stands.) Can these “related subjects” relevancy rank the subject itself?

I wish so, but I can’t get it to work well enough. It works for some topics, but falls down for others, laughably.

Some ideas I’ve considered:

• Treating subjects as links, and running some sort of “page-rank” style connection algorithm against them. Maybe this would bring out coincidences that simple statistics misses.
• Using other library data, such as LCC and Dewey. This would be reminiscent of how I made LibraryThing’s LCSH/LCC/Dewey recommendations.
• Doing statistics on other fields, such as the title. So, for example, there’s probably a statistical correlation between “Man-woman relationships” and books with “dating,” “men and women” and “proposal” in the title.

None strike me as the silver bullet.

Anyway, my plane has landed–allowing me to do real work again–so I end in aporia. Ideas?

*I’m itching to blog it, but I have to hold off for now. I’ll throw some pictures up soon, however. I’d never been to San Francisco before. What a wonderful wonderful town.
**One can understand why OPACs made in 1996 are session based. How frustrating to see a new product with them.