Archive for the ‘systems adminitration’ Category

Tuesday, January 5th, 2016

Job: Remote Sysadmin for LibraryThing

We’ll let you out from time to time.

Work with a great team, without meeting them!

LibraryThing is looking for a full-time systems administrator, starting soon. The job can be remote or local to Portland, Maine.

Why? Seth Ryder, LibraryThing’s sysadmin is moving on to an exciting new job at HarperCollins. This is bad for us—Seth was a fantastic shepherd of the LibraryThing systems. The good news is, thanks to Seth, our systems have never been stronger, more organized or better documented!

Specifics

Hours: In the past, we’ve listed the job as full- or part-time. This time we’re listing it as full-time, expecting the new sysadmin to take on various systems projects. We remain open to considering part-time applicants who are a particularly good fit.

Qualifications: We’re looking for someone with broad systems administration experience, who can quickly pick up unfamiliar technologies, diagnose problems and keep everything running smoothly. You need to be calm under pressure, cautious and an excellent communicator. We’re a small team, so when things break at 4am, you need to be available.

Work Anywhere. LibraryThing is “headquartered” in Portland, Maine, but the servers are in Massachusetts and most employees are in neither.

Experience: Applicants need considerable experience running websites. Experience in Linux systems administration is essential; we use RHEL and CentOS, but you’ve probably got professional experience with at least half a dozen distros. Experience with MySQL is also important, including replication, monitoring and tuning. You will need to be able to demonstrate experience with remote server administration including lights-out management techniques and equipment.

Technologies: Here’s a partial list of the technologies we use.

  • Apache
  • Nginx
  • MySQL, Master-Slave replication
  • Memcached
  • Solr, Elasticsearch
  • Subversion
  • PHP
  • Python
  • Bash shell scripting
  • Munin, Graphite, Logstash (ELK)
  • Xen and KVM virtualization
  • rrdtool
  • NFS
  • LVM
  • iscsi

Compensations: Salary plus great health insurance.

How to Apply: Email sysadminjob@librarything.com. Send an email with your resume. In your email, review the blog post above, and indicate how you match up with the job. Be specific.(1) Please do not send a separate cover letter.

If you want to stand out, go ahead and take the LibraryThing Programming Test. If programming is part of your skills, we’ll ask you to take it before we interview you.

We aren’t considering head-hunters or companies.


1. This job is going to be posted lots of places, and that means we’ll get a lot of people “rolling the dice.” If you don’t seem like you’re applying for this job, we’ll ignore your email. If you want us to KNOW you read the job post–and are therefore a detail-oriented person–please put “banana” in the subject line, as in “Sysadmin Job (Banana).” Really.

Labels: employees, employment, sysadmin, systems adminitration, Uncategorized

Tuesday, February 4th, 2014

Security Notice and LibraryThing Password Reset

As a security precaution, we are requiring ALL members to change their passwords, here: http://www.librarything.com/changepassword.php

A security review and search of our records has determined that LibraryThing suffered a data breach in November of 2011. The breach was narrow. We have found no evidence that any catalog or other book data was accessed, changed or lost. The breach did not include member names, so it is unlikely the hacker(s) were after LibraryThing accounts.

Unfortunately, the hacker(s) did assemble and retrieve two key pieces of data–email addresses and encrypted passwords of members who had listed an email and joined before that date. Our passwords are stored as a one-way encyption (in technical terms, a salted hash). Such hashes are difficult, but not impossible, to break, especially for simple passwords.

Although a minority of accounts were affected, we are requiring all members to change their password to take advantage of increased account security features.

The breach. The hacker(s) gained partial access to our system through a flaw in our WordPress blogging software. Read more in “The Full Details” below.

All evidence points to this being an email-hacking attack. We have every reason to believe no other LibraryThing data was taken, not even user names. The intent was probably to grab the emails for spam, and break the password hashes, if possible. When broken, the passwords could be used against members who used the same password for their email, or email-based services, as they used on LibraryThing. Using the same password across many services is bad practice, but not uncommon. No financial data could have been taken. We do not get or store credit card numbers or any other financial information.

Our response. Security has been tightened significantly since late 2011, and has been further improved across the board since we discovered the event during a security review on January 21st, 2014. We have now moved our WordPress blog off our servers entirely, so a successful hack leads nowhere. Our password and account-recovery systems have been upgraded to meet the highest industry standards, and we have implemented a slate of additional security measures.

Email notices are being sent out to all members with email addresses. You can change your password any time; you don’t need to wait for the email.

Our apology. The hack may come as a shock; it certainly was to us. Although events of this sort–and far worse–have become numbingly frequent, they are failures indeed. I regret and apologize that any such event could happen on my watch, and the rest of the team feels the same way. We are all committed to ensuring that LibraryThing is as secure as possible going forward.

We hope this failure will not sour you on our service or community. LibraryThing members are a dedicated and passionate bunch, and a pleasure and honor to serve. After years of getting by, the company has significant profits to sink back into development of the main site; we will meet this event with renewed dedication and resources. (Please see, and spread, our recent job ad.)

Because the hack undermines a customer relationship, we have chosen to upgrade to “lifetime” accounts all members who joined before November 20th, 2011. We included those who did not have email addresses listed.

Come ask questions and discuss on Talk. You are also welcome to email tim@librarything.com.

Sincerely,

Tim Spalding
Founder and President


The Full Details

What happened:

  • The hacker broke into the system through a flaw in WordPress, the blogging software that we use. This gave them only partial access to the system, but was sufficient to query the user database and save the results.
  • The breach occurred on November 19th, 2011.
  • We have no evidence of further data breaches. They are not impossible. We are confident no similar attacks could have taken place since at least January 2013, when we added some specific security features.
  • We discovered the breach on January 21st, 2014. As it happened so long ago, we believe whatever damage could be done, has already been done.
  • We waited two weeks in order to understand the attack and to implement a new password system and a series of other security steps before going public, and potentially drawing hacker interest.

What was taken:

  • The hacker(s) exported three fields: email address, password hash and the IP address at sign-up. (The IP would not be of much use to them.)
  • Only members with accounts opened before November 20th, 2011, with email addresses, were affected. In total, 685,259 emails were exported.
  • We have no indication that other LibraryThing data was accessed or taken. It is significant that the hacker didn’t even export LibraryThing user ids or user names. They were surely after emails and passwords, not book data.
  • LibraryThing does not receive or store credit card information or any other financial details. If you registered for a paid account via PayPal, PayPal has your credit card information, but they do not send the numbers to us.
  • We have reasonable suspicion that someone has used the data as a list of live email addresses, and sent spam to them. We have no evidence that any password hashes have been broken, or LibraryThing accounts compromised.

How passwords work:

  • Systems like LibraryThing do not store passwords per se. Rather, we store complicated cryptographic transformations, called hashes, which are “salted” for increased security.
  • In theory, you cannot get from the hash to the password. In practice, hackers with powerful computers can break hashed passwords, especially if the underlying password is simple (e.g., “book” rather than “mypencilbreaks71” or “xyA1!oG3g”).
  • Hacked passwords are dangerous when someone uses the same password across multiple online services, so failure at any one service opens up the rest.
  • Members should change their password at LibraryThing and any other service on which they used the same password. Here and elsewhere, members should also choose longer, hard-to-guess passwords. We encourage you to read safe-password advice from Google or Twitter.

Security improvements:

We can’t go into detail about security improvements. (If we did, we’d be compromising security.) But we can say what you can see:

  • We have moved our WordPress blogs off LibraryThing servers entirely, and onto a separate subdomain, blog.librarything.com. This insulates us from potential WordPress problems.
  • We have upgraded our password system to the highest industry standards. Users who joined in the last week or so, or changed their passwords, are already on the new system. But for simplicity’s sake, we’re requiring everyone to change their password.
  • We have a new system for password resets and changes, including password-strength indicators.
  • Our password recovery system has been changed from one involving sending out a temporary password to one employing quick-expiring tokens.
  • We are now sending out emails whenever a password has been changed. When a member changes their email address, change notices go out to both the new and old email addresses.
  • To discourage spamming of public emails—something that happened recently to some members—we have added an option to show your public email to friends, to signed-in members or everyone. By default, everyone who formerly chose to display their email publicly will now be set to friends-only.

Free accounts:

  • All members with accounts opened before November 20th, 2011 have been upgraded to lifetime accounts.

Labels: security, sysadmin, systems adminitration

Thursday, July 28th, 2011

Goodbye John!

Goodbye John (Felius), LibraryThing’s long-time sysadmin.

John’s been great to us. He took on a system under severe scaling strain, going down all the time and held together with string, and he sized it up and made it reliable. He moved the whole system from Portland to Boston, and made it both safer and faster (example, example). After almost four years with LibraryThing, John is moving on to Engine Yard, a Ruby-on-Rails cloud-hosting provider. His work and his company—John was a lot of fun to chat with at night—will be sorely missed. John promises to hang around as a member. He’s been one since 2005—long before we hired him.

We finished hiring John’s successor. More news soon.


PS: John managed to time his exit to System Administrator Appreciation Day. Believe me, we appreciate ’em.

Labels: sysadmin, systems adminitration

Monday, October 8th, 2007

Welcome Felius!

We’ve gone ahead and hired our first full-time, dedicated systems administrator. His name is John Dalton, but you know him as Felius, a LibraryThing member since September 14, 2005—two weeks after we launched! When he bleeds, he bleeds LibraryThing.

John’s mission at LibraryThing is simple:

  • Make things stable
  • Make things fast

John isn’t a miracle worker. A lot of our problems are in code, not systems (ie., blame me)*. Being without a dedicated, full-time “sysadmin” for so long has given him a lot of work to do. And our continued growth is scary. But we’re overjoyed to have him on board, and expect great things.

A few more things:

  • John lives in Tasmania, Australia. Seriously. This presents fewer problems than you might think. Although he’s fifteen hours ahead, everyone at LT works like a maniac, so our work days overlap a lot. And what is to our US and European members late-night maintenance and downtime takes place during his lunch hour.**
  • As we promised when we advertised for the job, whoever discovered our next employee would get a $1,000 book spree. We allowed people to find themself, which is what John did. Don’t you wish you worked for LibraryThing, or at least sent me a note about this guy Felius? He promises to be the first user of our upcoming wishlist feature. Then he’ll get his wish.
  • Favourite authors include Neal Stephenson, Arthur C. Clarke, Neil Gaiman, Bill Bryson and Simon Winchester.***
  • When not watching a dozen terminals or poring over columns of sar output, or reading, John’s interests include spending time with his wife and two young boys, gaming, playing cricket (badly) and occasionally performing in the Tenor section of the Tasmanian Symphony Orchestra Chorus.

*John is also a programmer, but we’re not going to be calling on these skills regularly. There’s enough pure systems stuff to do.
**Between John in Tasmania, Casey is Seattle and Giovanni in Germany, we can now officially claim that the “sun never sets on LibraryThing.” We can also claim some really complex accounting. John is even paid in Australian dollars, which fluctuate rather wildly against the dollar.
***He and I share Alfred Bester, Clifford Stoll and Paul Graham—right on.

Labels: employees, felius, john dalton, sysadmin, systems adminitration