We just finished moving about a dozen of our original servers from our “colo” in Maine to one in Somerville, where another dozen were waiting. In the process, we’ve basically doubled our server power.
We’re still waiting to get all our metrics back up, and we have a few weeks of retasking servers (a lot will be wearing different hats), but, so far, the results have been very encouraging. We are faster today, and will be getting faster tomorrow. We have a lot more memory, disk space and system redundancy too—so keep adding books.
We’ve collected all our pictures on a Flickr tag Great LibraryThing Server Schlepp. Here are some of the better ones.
The move was a group affair. Abby, Sonya, Mike, Dan and I did all the physical work. Our Australian systems administrator directed us by video chat—and burned through his monthly bandwidth doing it. Abby and I did a trial run with one server on Wednesday. The rest pulled an all-nighter, except Sonya who arrived like a well-rested and showered cavalry at 7am. When we were done, Mike and I went back to my parents’ in Cambridge and slept like logs. Abby, who was taking care of a toddler, stayed up the whole day. Ouch—and kudos to her.
Labels: new features, servers
Nice rack(s)!
(Somebody had to say it.)
Thanks for all the work. I just joined up a few weeks ago, and I love the depth and flexibility of the site.
Keep it up!
All of you deserve kudo's and R&R. I'm new to your site. I jumped in head-first immediately. When asked what my latest addiction is (and it's always something), I simply say "LibraryThing.com.
Thank you very much for all the hard work and the sleepless time, we just take it by granted. But yes, I say THANK YOU!!!
It is nice geeky fun to handle bare metal of course. I wonder if you ever considered using Amazon's Web Services. You'd be able to concentrate all your energies on the site's features, instead of building, and managing a data center.
And since I assume that you have considered, and discarded it, the next question is 'Why did you choose not to use AWS?"
Thank you for this amazing place. Whenever I need a break I come to this cyberspace library and find my sanity.
Just watched the entire slide show. I especially liked the last few pictures with John lurking in the corner looking by turns intense and jovial.
Looks like an amazing undertaking.
@David
The biggest problem for us is our database servers. I tested a small EC2 instance running MySQL on an EBS volume and found that it couldn't keep up with our replication load (let alone serve any real queries).
There are other tests I could try (bigger instances, different IO setups) but they'd be much more expensive and time consuming for me to attempt. One day, perhaps.
On the other hand, architectural changes on our end could make us fit better into the EC2 model, but that's not a quick fix either!
I *am* a fan of EC2, however, and I still dream of shifting some batch processing load or even web front-end requests off into the cloud. Unfortunately we have more than twice as many DB servers as we do web servers, so LibraryThing will be running on real hardware for a while longer yet!
Thank you everyone for your hard work! I love this website.