Archive for the ‘sysadmin’ Category

Tuesday, January 5th, 2016

Job: Remote Sysadmin for LibraryThing

We’ll let you out from time to time.

Work with a great team, without meeting them!

LibraryThing is looking for a full-time systems administrator, starting soon. The job can be remote or local to Portland, Maine.

Why? Seth Ryder, LibraryThing’s sysadmin is moving on to an exciting new job at HarperCollins. This is bad for us—Seth was a fantastic shepherd of the LibraryThing systems. The good news is, thanks to Seth, our systems have never been stronger, more organized or better documented!

Specifics

Hours: In the past, we’ve listed the job as full- or part-time. This time we’re listing it as full-time, expecting the new sysadmin to take on various systems projects. We remain open to considering part-time applicants who are a particularly good fit.

Qualifications: We’re looking for someone with broad systems administration experience, who can quickly pick up unfamiliar technologies, diagnose problems and keep everything running smoothly. You need to be calm under pressure, cautious and an excellent communicator. We’re a small team, so when things break at 4am, you need to be available.

Work Anywhere. LibraryThing is “headquartered” in Portland, Maine, but the servers are in Massachusetts and most employees are in neither.

Experience: Applicants need considerable experience running websites. Experience in Linux systems administration is essential; we use RHEL and CentOS, but you’ve probably got professional experience with at least half a dozen distros. Experience with MySQL is also important, including replication, monitoring and tuning. You will need to be able to demonstrate experience with remote server administration including lights-out management techniques and equipment.

Technologies: Here’s a partial list of the technologies we use.

  • Apache
  • Nginx
  • MySQL, Master-Slave replication
  • Memcached
  • Solr, Elasticsearch
  • Subversion
  • PHP
  • Python
  • Bash shell scripting
  • Munin, Graphite, Logstash (ELK)
  • Xen and KVM virtualization
  • rrdtool
  • NFS
  • LVM
  • iscsi

Compensations: Salary plus great health insurance.

How to Apply: Email sysadminjob@librarything.com. Send an email with your resume. In your email, review the blog post above, and indicate how you match up with the job. Be specific.(1) Please do not send a separate cover letter.

If you want to stand out, go ahead and take the LibraryThing Programming Test. If programming is part of your skills, we’ll ask you to take it before we interview you.

We aren’t considering head-hunters or companies.


1. This job is going to be posted lots of places, and that means we’ll get a lot of people “rolling the dice.” If you don’t seem like you’re applying for this job, we’ll ignore your email. If you want us to KNOW you read the job post–and are therefore a detail-oriented person–please put “banana” in the subject line, as in “Sysadmin Job (Banana).” Really.

Labels: employees, employment, sysadmin, Uncategorized

Tuesday, February 4th, 2014

Security Notice and LibraryThing Password Reset

As a security precaution, we are requiring ALL members to change their passwords, here: http://www.librarything.com/changepassword.php

A security review and search of our records has determined that LibraryThing suffered a data breach in November of 2011. The breach was narrow. We have found no evidence that any catalog or other book data was accessed, changed or lost. The breach did not include member names, so it is unlikely the hacker(s) were after LibraryThing accounts.

Unfortunately, the hacker(s) did assemble and retrieve two key pieces of data–email addresses and encrypted passwords of members who had listed an email and joined before that date. Our passwords are stored as a one-way encyption (in technical terms, a salted hash). Such hashes are difficult, but not impossible, to break, especially for simple passwords.

Although a minority of accounts were affected, we are requiring all members to change their password to take advantage of increased account security features.

The breach. The hacker(s) gained partial access to our system through a flaw in our WordPress blogging software. Read more in “The Full Details” below.

All evidence points to this being an email-hacking attack. We have every reason to believe no other LibraryThing data was taken, not even user names. The intent was probably to grab the emails for spam, and break the password hashes, if possible. When broken, the passwords could be used against members who used the same password for their email, or email-based services, as they used on LibraryThing. Using the same password across many services is bad practice, but not uncommon. No financial data could have been taken. We do not get or store credit card numbers or any other financial information.

Our response. Security has been tightened significantly since late 2011, and has been further improved across the board since we discovered the event during a security review on January 21st, 2014. We have now moved our WordPress blog off our servers entirely, so a successful hack leads nowhere. Our password and account-recovery systems have been upgraded to meet the highest industry standards, and we have implemented a slate of additional security measures.

Email notices are being sent out to all members with email addresses. You can change your password any time; you don’t need to wait for the email.

Our apology. The hack may come as a shock; it certainly was to us. Although events of this sort–and far worse–have become numbingly frequent, they are failures indeed. I regret and apologize that any such event could happen on my watch, and the rest of the team feels the same way. We are all committed to ensuring that LibraryThing is as secure as possible going forward.

We hope this failure will not sour you on our service or community. LibraryThing members are a dedicated and passionate bunch, and a pleasure and honor to serve. After years of getting by, the company has significant profits to sink back into development of the main site; we will meet this event with renewed dedication and resources. (Please see, and spread, our recent job ad.)

Because the hack undermines a customer relationship, we have chosen to upgrade to “lifetime” accounts all members who joined before November 20th, 2011. We included those who did not have email addresses listed.

Come ask questions and discuss on Talk. You are also welcome to email tim@librarything.com.

Sincerely,

Tim Spalding
Founder and President


The Full Details

What happened:

  • The hacker broke into the system through a flaw in WordPress, the blogging software that we use. This gave them only partial access to the system, but was sufficient to query the user database and save the results.
  • The breach occurred on November 19th, 2011.
  • We have no evidence of further data breaches. They are not impossible. We are confident no similar attacks could have taken place since at least January 2013, when we added some specific security features.
  • We discovered the breach on January 21st, 2014. As it happened so long ago, we believe whatever damage could be done, has already been done.
  • We waited two weeks in order to understand the attack and to implement a new password system and a series of other security steps before going public, and potentially drawing hacker interest.

What was taken:

  • The hacker(s) exported three fields: email address, password hash and the IP address at sign-up. (The IP would not be of much use to them.)
  • Only members with accounts opened before November 20th, 2011, with email addresses, were affected. In total, 685,259 emails were exported.
  • We have no indication that other LibraryThing data was accessed or taken. It is significant that the hacker didn’t even export LibraryThing user ids or user names. They were surely after emails and passwords, not book data.
  • LibraryThing does not receive or store credit card information or any other financial details. If you registered for a paid account via PayPal, PayPal has your credit card information, but they do not send the numbers to us.
  • We have reasonable suspicion that someone has used the data as a list of live email addresses, and sent spam to them. We have no evidence that any password hashes have been broken, or LibraryThing accounts compromised.

How passwords work:

  • Systems like LibraryThing do not store passwords per se. Rather, we store complicated cryptographic transformations, called hashes, which are “salted” for increased security.
  • In theory, you cannot get from the hash to the password. In practice, hackers with powerful computers can break hashed passwords, especially if the underlying password is simple (e.g., “book” rather than “mypencilbreaks71” or “xyA1!oG3g”).
  • Hacked passwords are dangerous when someone uses the same password across multiple online services, so failure at any one service opens up the rest.
  • Members should change their password at LibraryThing and any other service on which they used the same password. Here and elsewhere, members should also choose longer, hard-to-guess passwords. We encourage you to read safe-password advice from Google or Twitter.

Security improvements:

We can’t go into detail about security improvements. (If we did, we’d be compromising security.) But we can say what you can see:

  • We have moved our WordPress blogs off LibraryThing servers entirely, and onto a separate subdomain, blog.librarything.com. This insulates us from potential WordPress problems.
  • We have upgraded our password system to the highest industry standards. Users who joined in the last week or so, or changed their passwords, are already on the new system. But for simplicity’s sake, we’re requiring everyone to change their password.
  • We have a new system for password resets and changes, including password-strength indicators.
  • Our password recovery system has been changed from one involving sending out a temporary password to one employing quick-expiring tokens.
  • We are now sending out emails whenever a password has been changed. When a member changes their email address, change notices go out to both the new and old email addresses.
  • To discourage spamming of public emails—something that happened recently to some members—we have added an option to show your public email to friends, to signed-in members or everyone. By default, everyone who formerly chose to display their email publicly will now be set to friends-only.

Free accounts:

  • All members with accounts opened before November 20th, 2011 have been upgraded to lifetime accounts.

Labels: security, sysadmin

Wednesday, December 12th, 2012

Welcome Seth, LT’s new sysadmin!

We are delighted to welcome Seth Ryder (LT member sryder) to the LibraryThing staff. Seth is our new systems administrator, and aside from all the usual system-administrator-type stuff, we threw him into the deep end last week by having him help us out with SantaThing ordering on his fourth day on the job!

Seth grew up in a small town outside Grand Rapids, Michigan. He worked for a large hosting company in the midwest where he spent his time as a Systems Administrator and even dabbled with a bit of quality assurance work for their internal development team. He has also has been doing freelance development for a few small companies over the past three years. At LibraryThing, Seth steps in to succeed Brian Stinson, who left LT for a great job at Kansas State University, where he’s working on his graduate degree in political science.

When he’s not busy keeping LibraryThing up and running, Seth says he enjoys attending concerts, learning about new technologies, reading fantasy books, exploring local breweries with friends, and hacking on personal projects. His favorite authors include J.R.R. Tolkien, George R.R. Martin, J.K. Rowling, and Brandon Sanderson. You can follow him on Twitter at @sethryder.

Labels: employees, sysadmin

Monday, August 15th, 2011

Welcome Brian!

Welcome Brian Stinson (LT member tabashco), our new systems administrator: the person who keeps the servers running, plans expansions, monitors performance and protects your data.

Brian describes himself as a city kid from Witchita, KS (and writes “Before you ask, I’ve never met Dorothy, and I couldn’t grow some wheat to save my life but the Sunflower State will always be home”). He earned his BS in Computer Science from Kansas State University, where he’s soon to begin a graduate program in Political Science. Brian will be supported by the rest of the LibraryThing staff, who have become much more familiar with the systems side of LibraryThing since John informed us of his departure.

Brian likes C-Span, running, reading, college football, sledding, and listening to campus radio. His favorite authors include Ernest Hemingway, Cory Doctorow, Arthur Conan Doyle, and Mark Twain. You can follow him on Twitter at @tabashco.

Labels: employees, servers, sysadmin

Thursday, July 28th, 2011

Goodbye John!

Goodbye John (Felius), LibraryThing’s long-time sysadmin.

John’s been great to us. He took on a system under severe scaling strain, going down all the time and held together with string, and he sized it up and made it reliable. He moved the whole system from Portland to Boston, and made it both safer and faster (example, example). After almost four years with LibraryThing, John is moving on to Engine Yard, a Ruby-on-Rails cloud-hosting provider. His work and his company—John was a lot of fun to chat with at night—will be sorely missed. John promises to hang around as a member. He’s been one since 2005—long before we hired him.

We finished hiring John’s successor. More news soon.


PS: John managed to time his exit to System Administrator Appreciation Day. Believe me, we appreciate ’em.

Labels: sysadmin