Archive for the ‘sysadmin’ Category

Tuesday, January 5th, 2016

Job: Remote Sysadmin for LibraryThing

We’ll let you out from time to time.

Work with a great team, without meeting them!

LibraryThing is looking for a full-time systems administrator, starting soon. The job can be remote or local to Portland, Maine.

Why? Seth Ryder, LibraryThing’s sysadmin is moving on to an exciting new job at HarperCollins. This is bad for us—Seth was a fantastic shepherd of the LibraryThing systems. The good news is, thanks to Seth, our systems have never been stronger, more organized or better documented!


Hours: In the past, we’ve listed the job as full- or part-time. This time we’re listing it as full-time, expecting the new sysadmin to take on various systems projects. We remain open to considering part-time applicants who are a particularly good fit.

Qualifications: We’re looking for someone with broad systems administration experience, who can quickly pick up unfamiliar technologies, diagnose problems and keep everything running smoothly. You need to be calm under pressure, cautious and an excellent communicator. We’re a small team, so when things break at 4am, you need to be available.

Work Anywhere. LibraryThing is “headquartered” in Portland, Maine, but the servers are in Massachusetts and most employees are in neither.

Experience: Applicants need considerable experience running websites. Experience in Linux systems administration is essential; we use RHEL and CentOS, but you’ve probably got professional experience with at least half a dozen distros. Experience with MySQL is also important, including replication, monitoring and tuning. You will need to be able to demonstrate experience with remote server administration including lights-out management techniques and equipment.

Technologies: Here’s a partial list of the technologies we use.

  • Apache
  • Nginx
  • MySQL, Master-Slave replication
  • Memcached
  • Solr, Elasticsearch
  • Subversion
  • PHP
  • Python
  • Bash shell scripting
  • Munin, Graphite, Logstash (ELK)
  • Xen and KVM virtualization
  • rrdtool
  • NFS
  • LVM
  • iscsi

Compensations: Salary plus great health insurance.

How to Apply: Email Send an email with your resume. In your email, review the blog post above, and indicate how you match up with the job. Be specific.(1) Please do not send a separate cover letter.

If you want to stand out, go ahead and take the LibraryThing Programming Test. If programming is part of your skills, we’ll ask you to take it before we interview you.

We aren’t considering head-hunters or companies.

1. This job is going to be posted lots of places, and that means we’ll get a lot of people “rolling the dice.” If you don’t seem like you’re applying for this job, we’ll ignore your email. If you want us to KNOW you read the job post–and are therefore a detail-oriented person–please put “banana” in the subject line, as in “Sysadmin Job (Banana).” Really.

Labels: employees, employment, sysadmin, systems adminitration, Uncategorized

Tuesday, February 4th, 2014

Security Notice and LibraryThing Password Reset

As a security precaution, we are requiring ALL members to change their passwords, here:

A security review and search of our records has determined that LibraryThing suffered a data breach in November of 2011. The breach was narrow. We have found no evidence that any catalog or other book data was accessed, changed or lost. The breach did not include member names, so it is unlikely the hacker(s) were after LibraryThing accounts.

Unfortunately, the hacker(s) did assemble and retrieve two key pieces of data–email addresses and encrypted passwords of members who had listed an email and joined before that date. Our passwords are stored as a one-way encyption (in technical terms, a salted hash). Such hashes are difficult, but not impossible, to break, especially for simple passwords.

Although a minority of accounts were affected, we are requiring all members to change their password to take advantage of increased account security features.

The breach. The hacker(s) gained partial access to our system through a flaw in our WordPress blogging software. Read more in “The Full Details” below.

All evidence points to this being an email-hacking attack. We have every reason to believe no other LibraryThing data was taken, not even user names. The intent was probably to grab the emails for spam, and break the password hashes, if possible. When broken, the passwords could be used against members who used the same password for their email, or email-based services, as they used on LibraryThing. Using the same password across many services is bad practice, but not uncommon. No financial data could have been taken. We do not get or store credit card numbers or any other financial information.

Our response. Security has been tightened significantly since late 2011, and has been further improved across the board since we discovered the event during a security review on January 21st, 2014. We have now moved our WordPress blog off our servers entirely, so a successful hack leads nowhere. Our password and account-recovery systems have been upgraded to meet the highest industry standards, and we have implemented a slate of additional security measures.

Email notices are being sent out to all members with email addresses. You can change your password any time; you don’t need to wait for the email.

Our apology. The hack may come as a shock; it certainly was to us. Although events of this sort–and far worse–have become numbingly frequent, they are failures indeed. I regret and apologize that any such event could happen on my watch, and the rest of the team feels the same way. We are all committed to ensuring that LibraryThing is as secure as possible going forward.

We hope this failure will not sour you on our service or community. LibraryThing members are a dedicated and passionate bunch, and a pleasure and honor to serve. After years of getting by, the company has significant profits to sink back into development of the main site; we will meet this event with renewed dedication and resources. (Please see, and spread, our recent job ad.)

Because the hack undermines a customer relationship, we have chosen to upgrade to “lifetime” accounts all members who joined before November 20th, 2011. We included those who did not have email addresses listed.

Come ask questions and discuss on Talk. You are also welcome to email


Tim Spalding
Founder and President

The Full Details

What happened:

  • The hacker broke into the system through a flaw in WordPress, the blogging software that we use. This gave them only partial access to the system, but was sufficient to query the user database and save the results.
  • The breach occurred on November 19th, 2011.
  • We have no evidence of further data breaches. They are not impossible. We are confident no similar attacks could have taken place since at least January 2013, when we added some specific security features.
  • We discovered the breach on January 21st, 2014. As it happened so long ago, we believe whatever damage could be done, has already been done.
  • We waited two weeks in order to understand the attack and to implement a new password system and a series of other security steps before going public, and potentially drawing hacker interest.

What was taken:

  • The hacker(s) exported three fields: email address, password hash and the IP address at sign-up. (The IP would not be of much use to them.)
  • Only members with accounts opened before November 20th, 2011, with email addresses, were affected. In total, 685,259 emails were exported.
  • We have no indication that other LibraryThing data was accessed or taken. It is significant that the hacker didn’t even export LibraryThing user ids or user names. They were surely after emails and passwords, not book data.
  • LibraryThing does not receive or store credit card information or any other financial details. If you registered for a paid account via PayPal, PayPal has your credit card information, but they do not send the numbers to us.
  • We have reasonable suspicion that someone has used the data as a list of live email addresses, and sent spam to them. We have no evidence that any password hashes have been broken, or LibraryThing accounts compromised.

How passwords work:

  • Systems like LibraryThing do not store passwords per se. Rather, we store complicated cryptographic transformations, called hashes, which are “salted” for increased security.
  • In theory, you cannot get from the hash to the password. In practice, hackers with powerful computers can break hashed passwords, especially if the underlying password is simple (e.g., “book” rather than “mypencilbreaks71” or “xyA1!oG3g”).
  • Hacked passwords are dangerous when someone uses the same password across multiple online services, so failure at any one service opens up the rest.
  • Members should change their password at LibraryThing and any other service on which they used the same password. Here and elsewhere, members should also choose longer, hard-to-guess passwords. We encourage you to read safe-password advice from Google or Twitter.

Security improvements:

We can’t go into detail about security improvements. (If we did, we’d be compromising security.) But we can say what you can see:

  • We have moved our WordPress blogs off LibraryThing servers entirely, and onto a separate subdomain, This insulates us from potential WordPress problems.
  • We have upgraded our password system to the highest industry standards. Users who joined in the last week or so, or changed their passwords, are already on the new system. But for simplicity’s sake, we’re requiring everyone to change their password.
  • We have a new system for password resets and changes, including password-strength indicators.
  • Our password recovery system has been changed from one involving sending out a temporary password to one employing quick-expiring tokens.
  • We are now sending out emails whenever a password has been changed. When a member changes their email address, change notices go out to both the new and old email addresses.
  • To discourage spamming of public emails—something that happened recently to some members—we have added an option to show your public email to friends, to signed-in members or everyone. By default, everyone who formerly chose to display their email publicly will now be set to friends-only.

Free accounts:

  • All members with accounts opened before November 20th, 2011 have been upgraded to lifetime accounts.

Labels: security, sysadmin, systems adminitration

Wednesday, December 12th, 2012

Welcome Seth, LT’s new sysadmin!

We are delighted to welcome Seth Ryder (LT member sryder) to the LibraryThing staff. Seth is our new systems administrator, and aside from all the usual system-administrator-type stuff, we threw him into the deep end last week by having him help us out with SantaThing ordering on his fourth day on the job!

Seth grew up in a small town outside Grand Rapids, Michigan. He worked for a large hosting company in the midwest where he spent his time as a Systems Administrator and even dabbled with a bit of quality assurance work for their internal development team. He has also has been doing freelance development for a few small companies over the past three years. At LibraryThing, Seth steps in to succeed Brian Stinson, who left LT for a great job at Kansas State University, where he’s working on his graduate degree in political science.

When he’s not busy keeping LibraryThing up and running, Seth says he enjoys attending concerts, learning about new technologies, reading fantasy books, exploring local breweries with friends, and hacking on personal projects. His favorite authors include J.R.R. Tolkien, George R.R. Martin, J.K. Rowling, and Brandon Sanderson. You can follow him on Twitter at @sethryder.

Labels: employees, sysadmin

Monday, August 15th, 2011

Welcome Brian!

Welcome Brian Stinson (LT member tabashco), our new systems administrator: the person who keeps the servers running, plans expansions, monitors performance and protects your data.

Brian describes himself as a city kid from Witchita, KS (and writes “Before you ask, I’ve never met Dorothy, and I couldn’t grow some wheat to save my life but the Sunflower State will always be home”). He earned his BS in Computer Science from Kansas State University, where he’s soon to begin a graduate program in Political Science. Brian will be supported by the rest of the LibraryThing staff, who have become much more familiar with the systems side of LibraryThing since John informed us of his departure.

Brian likes C-Span, running, reading, college football, sledding, and listening to campus radio. His favorite authors include Ernest Hemingway, Cory Doctorow, Arthur Conan Doyle, and Mark Twain. You can follow him on Twitter at @tabashco.

Labels: employees, servers, sysadmin

Thursday, July 28th, 2011

Goodbye John!

Goodbye John (Felius), LibraryThing’s long-time sysadmin.

John’s been great to us. He took on a system under severe scaling strain, going down all the time and held together with string, and he sized it up and made it reliable. He moved the whole system from Portland to Boston, and made it both safer and faster (example, example). After almost four years with LibraryThing, John is moving on to Engine Yard, a Ruby-on-Rails cloud-hosting provider. His work and his company—John was a lot of fun to chat with at night—will be sorely missed. John promises to hang around as a member. He’s been one since 2005—long before we hired him.

We finished hiring John’s successor. More news soon.

PS: John managed to time his exit to System Administrator Appreciation Day. Believe me, we appreciate ’em.

Labels: sysadmin, systems adminitration

Sunday, December 21st, 2008

We’re faster (but not resting)

Last Wednesday John brought live two new database servers, Alexander and Hannibal*.

Together, they more than doubled our database heft. Put another way, our servers, which were operating at near full capacity all day long, can finally rest a bit. They can do everything as fast as they’re able, unencumbered by unsupportable amounts of work.

Performance. The effect on site performance has been positive. But problems remain. Profile pages are dramatically faster. Author, work, subject are faster and no longer slow down at peak times. Talk pages are essentially unchanged.

The catalog is faster. The page-generation averages now hover just over one second, not around two seconds. But I was hoping for more. The standard deviation of page-creation times remains high—people with huge libraries get hurt. Last night we I made a series of improvements which I hope will pay off. (The standard deviation is down, but will it stay down?)

The future. We will continue to improve. Until Wednesday the situation was desperate. When a box got behind, we had to turn off access to interior pages to all but signed-in members. That day is over, thank God**. And we can finally tease apart what was is itself slow, versus what was just slow because everything else was slowing it down. Lastly, John has long wanted to try out some low-level tweaks, but with no spare capacity, couldn’t. I expect he will find ways to wring more out of what we have.

Whether he can or not, we are going to keep improving. We have laid aside the money to buy a number of other servers—up to ten, if needed. One or two will be database servers, probably removing administration and caching traffic from the live servers. A number will be memory machines—low-end boxes with tiny disk drives and obscene amounts of RAM. They’ll help us use memory caching more effectively, reducing database load. The balance will be tasked in other ways—supporting LibraryThing for Libraries, serving secondary resources (covers, APIs, widgets) and providing redundancy, so we won’t be skating along a cliff anymore.

Thanks to John for getting the new servers racked and running. Thanks to the members for hanging in with us as we grow, and grew and grew!

*Yes, I named them. Cliche, I know. But Alexander was my research interest in grad school, so I’m allowed! Anyway, at least they’re consistent, and set a pattern we can follow (next up, Mithridates and Shapur). I’m still bothered that a previous sysadmin named our twin MyISAM databases Apollo and Athena, not Apollo and Artemis (who were twins). Then there’s Plato and his bigger twin Mongo, which makes no sense, but feels right, and the one everyone hates, our backup machine, Mnemosyne.
** John adds “the upgrade has given our database servers more horsepower rather than more raw speed. While the new servers are faster, the biggest initial gain is in the amount of load we can take on without starting to slow down.”

Labels: servers, sysadmin

Monday, October 8th, 2007

Welcome Felius!

We’ve gone ahead and hired our first full-time, dedicated systems administrator. His name is John Dalton, but you know him as Felius, a LibraryThing member since September 14, 2005—two weeks after we launched! When he bleeds, he bleeds LibraryThing.

John’s mission at LibraryThing is simple:

  • Make things stable
  • Make things fast

John isn’t a miracle worker. A lot of our problems are in code, not systems (ie., blame me)*. Being without a dedicated, full-time “sysadmin” for so long has given him a lot of work to do. And our continued growth is scary. But we’re overjoyed to have him on board, and expect great things.

A few more things:

  • John lives in Tasmania, Australia. Seriously. This presents fewer problems than you might think. Although he’s fifteen hours ahead, everyone at LT works like a maniac, so our work days overlap a lot. And what is to our US and European members late-night maintenance and downtime takes place during his lunch hour.**
  • As we promised when we advertised for the job, whoever discovered our next employee would get a $1,000 book spree. We allowed people to find themself, which is what John did. Don’t you wish you worked for LibraryThing, or at least sent me a note about this guy Felius? He promises to be the first user of our upcoming wishlist feature. Then he’ll get his wish.
  • Favourite authors include Neal Stephenson, Arthur C. Clarke, Neil Gaiman, Bill Bryson and Simon Winchester.***
  • When not watching a dozen terminals or poring over columns of sar output, or reading, John’s interests include spending time with his wife and two young boys, gaming, playing cricket (badly) and occasionally performing in the Tenor section of the Tasmanian Symphony Orchestra Chorus.

*John is also a programmer, but we’re not going to be calling on these skills regularly. There’s enough pure systems stuff to do.
**Between John in Tasmania, Casey is Seattle and Giovanni in Germany, we can now officially claim that the “sun never sets on LibraryThing.” We can also claim some really complex accounting. John is even paid in Australian dollars, which fluctuate rather wildly against the dollar.
***He and I share Alfred Bester, Clifford Stoll and Paul Graham—right on.

Labels: employees, felius, john dalton, sysadmin, systems adminitration