Blaine Cook from Twitter released Starling last week. He describes it as:

Starling is a light-weight persistent queue server that speaks the MemCache protocol. It was built to drive Twitter’s backend, and is in production across Twitter’s cluster.

I’m always a sucker for new infrastructure code but the problem is trying to figure out how to see if it will fit within whatever current architecture is already in place. I think though I might have found somewhere.

We have an aggregator grabbing various RSS feeds both internally and externally. Right now, during the parsing / adding to the database, we send off the information to our indexer for later searching. This is basically how the Topics pages are built dynamically.

Instead of sending the data during run-time, maybe it would be better to just add it to a queue with Starling for later processing. A job in the indexer could check the queue and index anything that’s there. This also would allow other applications to add things ready for indexing.

• BROWSE / IN TIMELINE
• « topics.latimes.com
• » Changes at the Times

• BROWSE / IN Code Development Ruby
• « topics.latimes.com
• » No Architects Need Apply