Archive for the ‘servers’ Category

Tuesday, February 11th, 2014

LibraryThing adds SSL


LibraryThing has added SSL encryption to all pages that ask for private data.

That means the data you submit for signing in—signing up, changing your password, changing your email, etc.—is securely encrypted between you and LibraryThing. Depending on your browser, this will show up as a “lock” symbol, or just a change in the LibraryThing URL from http:// to https://.

Is LibraryThing going all-SSL?

We have decided on this as a first step, with the intention of going to all-SSL, or all-SSL for signed-in members only, as soon as practicable.

Going all-SSL is going to require considerable work, sifting through all the non-http URLs to avoid “mixed content” messages. Although these vary in their obtrusiveness browser-by-browser, going all-SSL without extensive testing is likely to lead to a lot more in confusion that it solves in potential problems.

As a result of this change, if you previously chose to browse LibraryThing using SSL, ignoring the warnings, you will no longer be able to do so. Rather, if you’re on one of the selected, user-data pages, it now forces you to use https. If you’re not on one of these pages, it forces you to use http.

At present, the solution covers and all its subdomains, like (Danish), (Brazilian Portuguese). It is not installed on separate domains, like (Germany) and (Holland). We will be weighing our options there, as SSL certificates are expensive.

Come discuss this on Talk, if you like.

Labels: new features, security, servers

Monday, August 15th, 2011

Welcome Brian!

Welcome Brian Stinson (LT member tabashco), our new systems administrator: the person who keeps the servers running, plans expansions, monitors performance and protects your data.

Brian describes himself as a city kid from Witchita, KS (and writes “Before you ask, I’ve never met Dorothy, and I couldn’t grow some wheat to save my life but the Sunflower State will always be home”). He earned his BS in Computer Science from Kansas State University, where he’s soon to begin a graduate program in Political Science. Brian will be supported by the rest of the LibraryThing staff, who have become much more familiar with the systems side of LibraryThing since John informed us of his departure.

Brian likes C-Span, running, reading, college football, sledding, and listening to campus radio. His favorite authors include Ernest Hemingway, Cory Doctorow, Arthur Conan Doyle, and Mark Twain. You can follow him on Twitter at @tabashco.

Labels: employees, servers, sysadmin

Tuesday, July 5th, 2011

Job: Be LibraryThing’s Systems Administrator

John chatting with Abby and Jeremy at the colo.

After four happy years at LibraryThing I’ve decided to take up an offer to move on to a new role. This means we need to a find a sysadmin to replace me!

LibraryThing is the labour of love of a small group of smart, dedicated people who work hard to do a lot with a little.

Qualifications: We’re looking for someone with broad systems administration experience, who can quickly pick up unfamiliar technologies, diagnose problems and keep everything running smoothly. You need to be calm under pressure, cautious and an excellent communicator. We’re a small team, so when things break at 4am, you need to be available.

LibraryThing is “headquartered” in Portland, Maine, but the servers are in Boston and many employees are in neither. You can be anywhere—I’m in Tasmania!

Experience: Applicants need considerable experience running websites. Experience in Linux systems administration is essential; we use RHEL and CentOS, but you’ve probably got professional experience with at least half a dozen distros. Experience with MySQL is also important, including replication, monitoring and tuning. You will need to be able to demonstrate experience with remote server administration including lights-out management techniques and equipment.

Technologies. Here’s a partial list of the technologies we use.

  • Apache
  • Nginx
  • Memcache
  • Solr
  • Subversion
  • PHP
  • Python
  • Bash shell scripting
  • Munin
  • rrdtool
  • Xen virtualisation
  • NFS
  • LVM
  • iscsi

Abby looking at the space where the new servers should go.

How We Work. LibraryThing has a somewhat unusual development culture. It is not for everyone. We develop quickly, knocking out features in hours or days, not weeks.

We develop incrementally and opportunistically, assuming that member feedback will sometimes overturn our plans in mid-course, and that some projects will fail. Everyone who works for LibraryThing must interact directly with members.

LibraryThing is more than a job for us. We work long, hard and usually sober, but not necessarily during “regular” hours. We love what we do. We want someone who will feel the same way.

Compensation. Salary plus gold-plated health insurance. This is a full-time job.

How to Apply.

Email: Tim and I will read your applications.

Send an email with your resume. In your email, go through the sections of the blog post above, and indicate how you match up with the job. Be specific. Do not send a cover letter.

Labels: employment, jobs, servers

Tuesday, May 10th, 2011

LibraryThing is Faster, part II

It’s not a new “feature,” but speed and reliability are a key component of the appeal of a site. A few weeks ago we reported on a new server configuration that cut page-generation times in half (see LibraryThing is Faster). Now we’re reporting on some database tweaks that have made the process of finding, adding and editing data faster.

Like all large database-driven sites, LibraryThing can’t rely on a single database. Instead, we have a single “master” database which replicates its changes to a number of “slave” databases. (See Wikipedia: Database Replication.) Because sites “read” a lot more than they write, scalability is achieved doing most “reads” from the slave machines, which can be multiplied almost indefinitely to deal with increased traffic. Unfortunately, writes still need to move from the master to the slave, which necessarily involves a slight lag. If the lag becomes too great you get stale data or processes that pause (and pile up!) waiting for fresh information to pass from the master to the slaves. You also get bugs. And annoyances, like Talk posts not appearing right away. Replication lag also degrades query speed and therefore site speed generally.

As a heavy database-driven site running on relatively cheap hardware we’ve sometimes struggled to keep replication delay down. The problem is particularly acute on our weaker slaves. Fortunately, our ongoing review of performance issues has disclosed series of code and “schema” changes that have substantially improved replication speed. Here’s a chart of the average replication delay on one of our database servers over the last ten days. As you can see, two changes have made a big difference!

We’re excited about the progress we’ve made so far. It exceeded our expectations. Our performance review is continuing. We won’t stop until LibraryThing is as fast and reliable as it is powerful, rich in data and fun to use!

Come talk about this.

Labels: servers, speed

Wednesday, April 20th, 2011

LibraryThing is faster

I’m happy to report that LibraryThing’s servers have undergone a considerable improvement. LibraryThing’s server administrator, John Dalton (member Felius), carried through an ambitious restructuring of how LibraryThing’s considerable traffic was distributed across our web servers. And the restructuring worked out.

Across the site “pages” have been sped up by about 100%, dropping from a median of about 1 second to about 0.5 seconds. Catalog, or “Your books,” pages have dropped from a median of 1.6 seconds to 0.8 seconds. Response has also become more predictable, with much a lower standard deviation of response times.

Good, but not everything. While server-rendering speed is important, it isn’t the only factor in perceived speed–the other two being transfer and rendering by the web browser. Server improvements also hide the fact that rendering times also include database actions, which were not improved by this change. The truly bothersome pages on LibraryThing are hindered by this, not by web server load per se. So, this change hasn’t done much to improve response time on catalogs with thousands or tens of thousands of books, hit for the first time that day, or on work-combination requests that require recalculating thousands of items of data. Basically, the improvement speeds up every page by an average of 0.5 seconds, but a 10 second page still takes 9.5 seconds.

Here are some graphs of the effect on different page types. The white band is a period for which we don’t have numbers.

All pages (includes widgets)

Catalog. Savings have been dramatic but, as noted above, mostly on the vast majority of “normal” requests, not on the rare but painful ones.

Talk topic pages. These have gotten much faster, because the data is easy to get from the database but also very copious, so it took a lot of server work to render it. This improvement has a perverse side-effect, however–the faster the page is made the more the Talk page can get ahead of master-slave replication. This issue will be addressed in an upcoming improvement.

Work pages haven’t improved because they were already well-distributed across our servers.

Labels: servers, speed

Monday, June 29th, 2009

More servers, less sleep

We just finished moving about a dozen of our original servers from our “colo” in Maine to one in Somerville, where another dozen were waiting. In the process, we’ve basically doubled our server power.

We’re still waiting to get all our metrics back up, and we have a few weeks of retasking servers (a lot will be wearing different hats), but, so far, the results have been very encouraging. We are faster today, and will be getting faster tomorrow. We have a lot more memory, disk space and system redundancy too—so keep adding books.

We’ve collected all our pictures on a Flickr tag Great LibraryThing Server Schlepp. Here are some of the better ones.

The move was a group affair. Abby, Sonya, Mike, Dan and I did all the physical work. Our Australian systems administrator directed us by video chat—and burned through his monthly bandwidth doing it. Abby and I did a trial run with one server on Wednesday. The rest pulled an all-nighter, except Sonya who arrived like a well-rested and showered cavalry at 7am. When we were done, Mike and I went back to my parents’ in Cambridge and slept like logs. Abby, who was taking care of a toddler, stayed up the whole day. Ouch—and kudos to her.

Labels: new features, servers

Sunday, December 21st, 2008

We’re faster (but not resting)

Last Wednesday John brought live two new database servers, Alexander and Hannibal*.

Together, they more than doubled our database heft. Put another way, our servers, which were operating at near full capacity all day long, can finally rest a bit. They can do everything as fast as they’re able, unencumbered by unsupportable amounts of work.

Performance. The effect on site performance has been positive. But problems remain. Profile pages are dramatically faster. Author, work, subject are faster and no longer slow down at peak times. Talk pages are essentially unchanged.

The catalog is faster. The page-generation averages now hover just over one second, not around two seconds. But I was hoping for more. The standard deviation of page-creation times remains high—people with huge libraries get hurt. Last night we I made a series of improvements which I hope will pay off. (The standard deviation is down, but will it stay down?)

The future. We will continue to improve. Until Wednesday the situation was desperate. When a box got behind, we had to turn off access to interior pages to all but signed-in members. That day is over, thank God**. And we can finally tease apart what was is itself slow, versus what was just slow because everything else was slowing it down. Lastly, John has long wanted to try out some low-level tweaks, but with no spare capacity, couldn’t. I expect he will find ways to wring more out of what we have.

Whether he can or not, we are going to keep improving. We have laid aside the money to buy a number of other servers—up to ten, if needed. One or two will be database servers, probably removing administration and caching traffic from the live servers. A number will be memory machines—low-end boxes with tiny disk drives and obscene amounts of RAM. They’ll help us use memory caching more effectively, reducing database load. The balance will be tasked in other ways—supporting LibraryThing for Libraries, serving secondary resources (covers, APIs, widgets) and providing redundancy, so we won’t be skating along a cliff anymore.

Thanks to John for getting the new servers racked and running. Thanks to the members for hanging in with us as we grow, and grew and grew!

*Yes, I named them. Cliche, I know. But Alexander was my research interest in grad school, so I’m allowed! Anyway, at least they’re consistent, and set a pattern we can follow (next up, Mithridates and Shapur). I’m still bothered that a previous sysadmin named our twin MyISAM databases Apollo and Athena, not Apollo and Artemis (who were twins). Then there’s Plato and his bigger twin Mongo, which makes no sense, but feels right, and the one everyone hates, our backup machine, Mnemosyne.
** John adds “the upgrade has given our database servers more horsepower rather than more raw speed. While the new servers are faster, the biggest initial gain is in the amount of load we can take on without starting to slow down.”

Labels: servers, sysadmin