Dive into the archives.
- A Good Day for Hadoop
Yesterday was a very good day for the Hadoop project.
Yahoo! announced they used a roughly 3800 node cluster to sort thru a Petabyte of data in a little over 16 hours. It’s an amazing feat for any project but especially one with so much potential as Hadoop.
The other good news was [...]
- Testing with Redis
Long time, no blog… But enough about that.
On the side, I’ve been working on a new aggregator, Aggir, which allows me to test various things. I started off using SQLite and Sequel for storage, put Solr behind the scenes for search and added a very simple Web UI using Sinatra and HAML. [...]
- Getting Down to the Metal
Rails Metal looks pretty darn awesome. It allows you to specify specific URI paths which will bypass the normal Rails stack, shaving precious milliseconds off your responses and not making the Baby Jesus cry.
As a byproduct with a simple config item, you can start using Rack::Cache which is a very good HTTP cache that [...]
- Looking into HBase
HBase is the open source implementation of Google’s Bigtable. I’ve been keeping my eye on it in combination with Hadoop. I had some extra time today so I decided to see how easy it would be to hook it up with the aggregator we built for things like Topics.
One of the nice things [...]
- Starling
Blaine Cook from Twitter released Starling last week. He describes it as:
Starling is a light-weight persistent queue server that speaks the MemCache protocol. It was built to drive Twitter’s backend, and is in production across Twitter’s cluster.
I’m always a sucker for new infrastructure code but the problem is trying to figure out how to [...]
