Archive for the ‘apis’ Category

Thursday, September 17th, 2009

The Amazon policy change, and how we’re responding.

“Amazon Cardboard Boxes” by Flickr member Akira Ohgaki (Attribution 2.0 Generic)

Summary: Amazon is requiring us remove links to other booksellers on work pages. We’re creating a new “Get it Now” page, with links to other booksellers, especially local bookstores and libraries, and a host of new features. Talk about it here.

The challenge. We’re days away from releasing a series of changes to our book pages, both forced and intentional. Amazon is requiring all websites, as a condition of getting any data from them, to have the primary page link to Amazon alone. Links to other booksellers are prohibited. Secondary pages—pages you go to from the primary page—can have non-Amazon links.

Everyone at LibraryThing disagrees with this decision. LibraryThing is not a social cataloging and social networking site for Amazon customers but for book lovers. Most of us are Amazon customers on Tuesday, and buy from a local bookstore or get from a library on Wednesday and Thursday! We recognize Amazon’s value, but we certainly value options.

Importanly, the decision is probably not even good for Amazon. Together with a new request-monitoring system, banning iPhone applications that use Amazon data, and much of their work on the Kindle, Amazon is retreating from its historic commitment to simplicity, flexibility and openness. They won through openness. Their data is all over the web, and with it millions of links to Amazon. They won’t benefit from a retreat here.

But agree or not, we have to follow their terms. We thought long and hard about giving up Amazon data entirely, converting to library data only, in concert with a commercial provider, like Bowker or Ingram, and with help from publishers and members. Unlike our competitors, who are exclusively based on Amazon and who don’t “catalog” so much as keep track of which Amazon items you have, that option is available to us. But we’d lose a lot, particularly book covers. Ultimately, we’ve decided the disadvantages outweigh the benefits.

The Response. Most of all, we think we’ve found a way to give Amazon what they require, and continue to provide members with options: We’re going cut back our primary-page links to Amazon alone, and give people the best, most diverse secondary pages we can make. We are allowed to link to other booksellers, like IndieBound and Barnes and Noble on secondary pages, and we’re going to do it far better than we ever have. We’re going to take something away, but also make something better—something that goes way past what we did before, in features and in diversity of options.

The upcoming “Get it Now” page will go far beyond our current “Buy, borrow, swap” links, with a live new and used price-comparison engine, as well as sections for ebooks, audiobooks and swap sites. The page will be edition-aware, and draw on feeds or live data (so the links work). Many members have wanted live pricing data for the books they already own and these features can be used for that purpose too. We’ll also be doing some stuff with libraries nobody else has, or can, do.

Key to the upcoming Get it Now page is a “Local” module, drawing on LibraryThing Local, showing all the libraries and bookstores near you. Where possible, this list will incorporate holdings data and links to buy—the sort of information you never get from a Google search on a book. If not, we’ll give you their telephone numbers and show you where they are on a map. We’ll make the page customizable, and let members add sources to it.

We think the new page will make a lot of members happy. For one thing, LibraryThing has never been about buying books, so having all these links on a separate page won’t be a great loss. And if the new format doesn’t make members happy, we’ll listen, and together we can plan to take LibraryThing on a truly independent course.

Post your comment here, or come talk about this on Site Talk.

Labels: amazon, apis, google, open data

Thursday, August 7th, 2008

A million free covers from LibraryThing

A few days ago, just before hitting thirty million books, we hit one million user-uploaded covers. So, we’ve decided to give them away—to libraries, to bookstores, to everyone.

The basics. The process, patterned after the cover service, is simplicity itself:

  1. Take an ISBN, like 0545010225
  2. Put your Developer Key and the ISBN into a URL, like so:
  3. Put that in an image tag, like so:
    <img src="">
  4. And your website, library catalog or bookstore has a cover.

Easy details. Each cover comes in three sizes. Just replace “medium” with “small” or “large.”

As with Amazon, if we don’t have a cover for the book, we return a transparent 1×1 pixel GIF image. So you can put the cover-image on OPAC pages without knowing if we have the image. If we have it, it shows; if we don’t, it doesn’t.

The Catch? To get covers, you’ll need a LibraryThing Developer Key—any member can get one. This puts a top limit on the number of covers you can retrieve per day—currently 1,000 covers. In fact, we only count it when a cover is made from the original, o our actual limit will be much higher. We encourage you to cache the files locally.

You also agree to some very limited terms:

  • You do not make LibraryThing cover images available to others in bulk. But you may cache bulk quantities of covers.
  • Use does not involve or promote a LibraryThing competitor.
  • If covers are fetched through an automatic process (eg., not by people hitting a web page), you may not fetch more than one cover per second.

You will note that unlike the new API to our Common Knowledge data, you are not required to link back to LibraryThing. But we would certainly appreciate it.

Caveats. Some caveats:

  • At present only about 913,000 covers are accessible, the others being non-ISBN covers.
  • Accuracy isn’t guaranteed–this is user data–and coverage varies.
  • Some covers are blurrier than we’d like, particularly at the “large” size. This is sometimes about original files and sometimes about our resizing routines. We’re working on the latter.

Why are you doing this? The goal is half promotional and half humanitarian.

First, some background. This service “competes” with Amazons cover service, now part of Amazon Web Services. Amazon’s service is, quite simply, better. They have far more covers, and no limit on the number of requests. By changing the URL you can do amazing things to Amazon covers.

The catch is that Amazon’s Terms of Service require a link-back. If you’re trying to make money from Amazon Affiliates, this is a good thing. But libraries and small bookstores have been understandably wary about linking to Amazon. Recent changes in Amazon’s Terms of Service have deepened this worry.

Meanwhile, there are a number of commercial cover providers. They too are probably, on average, better. But they cost money. Not surprisingly many libraries and bookstores skip covers, or paste them in manually from publisher sites.

That’s too bad. Publishers and authors want libraries and bookstores to show their covers. Under U.S. law showing covers to show off books for sale, rental or commentary falls under Fair Use in most circumstances. (We are not lawyers and make no warrant that your use will be legal.) We’ve felt for years that selling covers was a fading business. Serving the files is cheap and getting cheaper. It was time for someone to step up.*

So we’re stepping up. We’re hoping that by encouraging caching and limiting requests, we can keep our bandwidth charges under control. (If it really spikes, we’ll limit new developer keys for a while; if you submit this to Slashdot, we will be Slashdotted for sure!) And it will be good for LibraryThing—another example of our open approach to data. Although none of our competitors do anything like this—indeed our Facebook competitors don’t even allow export although, of course, they import LibraryThing files!—we think LibraryThing has always grown, in part, because we were the good guys—more “Do occasional good” than “Do no evil.”

If we build it, they will come. If the service really pick up, we’re going to add a way for publishers, bookstores and authors to get in on it. We’d be happy to trade some bandwidth out for what publishers know—high-quality covers, author photos, release dates and so forth. We’ve already worked with some publisher data, but we’d love to do more with it.

*In the past, we had been talking to the Open Libary project about a joint effort. We even sent them all our covers and a key to the identifiers that linked them. But nothing came of it. To some extent that was our fault, and to some extent not. (I think them and us would differ on the blame here.) In any case, I was tired of the time and transactional friction, and wanted to try a different approach.

Labels: apis, book covers, covers, open data

Friday, August 1st, 2008

Free Web Services API to Common Knowledge

Introducing the LibraryThing Web Services API.

The API will eventually do many things.

For starters it includes all of the data in LibraryThing’s Common Knowledge project, our groundbreaking “fielded wiki” for interesting book information (see original blog post). It includes fields like series, important characters, important places, author dates, author burial places, agents, edits, etc. If you’re interested in building or enhancing book-data applications, this should be very interesting.

Common Knowledge is always in progress, but the results so far have been quite impressive. Members have made over 500,000 edits, and certain data types have become exceedingly useful and comprehensive. I’m particularly proud of our Series coverage (eg., Star Wars), better—we think—than any commercial series data. 

Oh, and it’s free! The data is made available under the highly permissive Creative Commons Attribution Share Alike license.

Architecturally, the Web Services API is a straightforward REST XML-based API.  The back-end is modular, allowing us to easily expand the available methods in the future. It’s request and response styles were modeled closely on Flickr’s API—Chris is a big fan—so it should make it easier to find similar sample code. The documentation resembles theirs too.

Kudos to Chris for his work on this and let us know what you think (here).

Update: The other big announcement—another data release—won’t be happening today. Too much to do!

Labels: api, apis, common knowledge, web services

Sunday, July 6th, 2008

Introducing the LibraryThing books API

See bottom of post for updates…

I’ve just finished a first draft of a JSON-based API for book data, created a test page and typed up some basic documentation.

What is this for? The API gives you Javascript access to your book data. The most obvious use of this would be to create new, much better widgets. At first, we expect this to interest programmers, but as new widgets are developed, non-programmers will get cool things. I started by redoing our traditional widgets in a new way here. That’s the base, not the ceiling!

How does it work? Every user can retrieve their data, in JSON format—basically as a ready-made JavaScript data structure. You control what is returned—books, tags, ratings, etc.—how it’s sorted and so forth. By default we give you a standard library of functions to parse and display the data. You can use it, build on it or start from scratch. Find out more here.

What’s great? All our code for processing the JSON API has been and will be released as open source—available for use, reuse and modification. Better—since we’re not the best programmers, particularly in JavaScript!—we are requiring any software that builds upon the API to be released under similar terms, so everyone can take advantage of improvements and advances. 

Does this make code look sexy?

What’s the catch? The API is not intended for making backups or exporting your data to other programs. For that, use our CSV and TSV export functions, from the Tools tab. We are licensing the JSON API for browser-use only. This is about our data licenses. In-browser widgets have never drawn ire from our data providers.

Where can this go? This is just getting started. Everything can be expanded and improved. As members want new or different data, I will be only too happy to add it to the API. But the most interesting development will probably come from members, not LibraryThing employees.

I have created a LibraryThing API Development group to discuss the API, work through code and come up with new ideas.

At a minimum, I can see:

  • New widget types, like widgets showing your most recent reviews.
  • Widgets that take you to libraries, and other places other than LibraryThing. (Libraries have been clamoring for this for ages. Many use LibraryThing to feature new books on the website, and want the links to go to their catalog, not LibraryThing.)
  • New result sets, for your tags or authors (separate from our books), your book’s works, series info, etc.
  • Integration with other JS-based APIs, like Google Book Search.

What if I’m not a programmer? No problem. Come and LibraryThing API Developmenttell us what you want. We’ll help you, or maybe someone else will.

UPDATE: I’ve made some changes to the programming, changing how the code is structured and adding result sets for reading dates. We also have the first outside use of the API, a very promising—if not perfect—cover flip test by MMcM (here). Follow what’s going on in the LibraryThing API Development group.

Labels: api, apis, books api, javascript, json, member projects

Tuesday, March 25th, 2008


I’ve just blogged about a new Javascript/JSON API for work info over on Thingology, LibraryThing’s blog for ideas, issues, libraries and labs.

It’s mostly designed to make it easy for people to link to LibraryThing only when we have the book. You can also dress up the link with copy- and review-counts, and an average rating.

I think regular members will be more excited by a JSON API to your own books. This will allow us and members to write new widgets—widget for reviews, for example—and better widgets. I’m want to write them so that all the JavaScript code that comes out it is automatically shared between members, both legally and technically.

The work-info API is a first step. Let’s talk about this and what should come.

Labels: apis, json, widgets