On the side, I’ve been working on a new aggregator, Aggir, which allows me to test various things. I started off using SQLite and Sequel for storage, put Solr behind the scenes for search and added a very simple Web UI using Sinatra and HAML. Yeah, I think I pretty much used all the necessary hot projects. It was fun to build and it works pretty well right now.

I have more to do on the Solr front since I’m just using the defaults for relevance searching. I’d like to dig more into the Solr internals for additional query parsing and classification at index time. It’s some of the stuff I’ve been doing at work but wanted to use a different type of data set.

Of course, now that I had things somewhat stable, I decided to blow it all up and try something new. That something new is Redis using Ezra’s client library.

I started down the path of updating everything, ripping out the database storage to use Redis instead. So far so good, I have the start of this on a branch. One issue I found though was testing my code. It was simple with Sequel since I could create a different database without any worry of overwriting real data. With Redis, I can easily delete keys in between tests but with the keys were the same that a real update would use so non-test data would be deleted.

I think I’ve come up with a solution that at least is working for me. I’ve made each key combine a prefix with other data. The prefixes are defined as class variables. I only set them in the library code if they haven’t already been defined elsewhere. In my test code, I set them with an additional test-specific prefix so that I can easily delete all of the testing keys by using the keys(‘test_*’) method. This will allow me to walk thru all of the test keys created during a test and delete them before running the next test. This mirrors what is done with the database.

I’m now able to test on the same instance that I’ve loaded with posts from various blogs. I have more to say about the mindset change from a relational db to key-value storage but I wanted to get this post out.

What this means for managing projects—including any project that relies on the Internet to deliver its value proposition—is simple: The longer your project timeline, the greater the risk that what you deliver will not be what you or your customers need when you deliver it. Not only are longer-term projects more likely to fail due to changes in requirements or conditions during the project, they’re more expensive. This increases the cost of failure. And because we can only do a few of them in a year, the impact of any one failure is huge.

Since I worked from home, both were behind a router but I didn’t want to pony up money for a static IP so you never knew quite was the IP was if things were reconnected. Obviously this wasn’t a big deal when I was sitting at my desk home but became an issue when I was up at HQ in San Francisco.

My solution was to have a cron job run every hour on the server and compare its IP with the one it had an hour ago. If it changed, it emailed me so I would know.

I mention all of this because Jeremy Zawodny has done the same thing though he has made it much more Web 2.0-compliant by using Twitter.

I imagine I would do exactly that as well if I needed to right now.

Cloud Architectures address key difficulties surrounding large-scale data processing. In traditional data processing it is difficult to get as many machines as an application needs. Second, it is difficult to get the machines when one needs them. Third, it is difficult to distribute and co-ordinate a large-scale job on different machines, run processes on them, and provision another machine to recover if one machine fails. Fourth, it is difficult to auto-scale up and down based on dynamic workloads. Fifth, it is difficult to get rid of all those machines when the job is done. Cloud Architectures solve such difficulties.

Applications built on Cloud Architectures run in-the-cloud where the physical location of the infrastructure is determined by the provider. They take advantage of simple APIs of Internet-accessible services that scale on-demand, that are industrial-strength, where the complex reliability and scalability logic of the underlying services remains implemented and hidden inside-the-cloud. The usage of resources in Cloud Architectures is as needed, sometimes ephemeral or seasonal, thereby providing the highest utilization and optimum bang for the buck.

Definitely check the tips for building cloud applications since they are very relevant no matter if you are deploying on Amazon or your own system.

Yesterday, Tara Hunt posted a great intro for companies to use Twitter. It has great advice plus some good tools for keeping track of who is talking about you.

One was Tweetburner which shortens URLs and keeps track of how many clicks you get. The other is Tweetscan which allows you to search thru all tweets for a keyword.

I’ve found some interesting ones which weren’t complimentary to the Times but are a great look at people’s opinions.

• http://twitter.com/kirkskodis/statuses/794637391
• http://twitter.com/rowast/statuses/787411161
• http://twitter.com/ctcrowell/statuses/780154897

And there is the response from Giles. I work with Giles, respect him a ton but I think he’s wrong here or at least it’s too early to see if he is right or not.

Google App Engine is a joke. You can’t create apps in Rails, Haskell, Erlang, Seaside, Perl, PHP, or Lisp. You can’t even leverage Unix. You get all the restrictive corporate BS you’d have to deal with if you worked at Google, but you don’t get the stock options, the salary, the free food, or even a parking space for your trouble. Packing Google with partisans who favored a particular language to unreasonable degrees resulted in a product which just isn’t even worth considering. This is why polyglot programming matters. The age of one dominant language is over, and Google’s attempt to make Python numero uno just means wasted time and wasted money.

I think the focus on Python-only is missing a couple of things that seem fairly obviously. One, the main config app.yaml file has an option for saying which runtime is being used and states that other languages / runtimes may be supported in the future. And yes, I do see that it isn’t for sure but come on, do you really think Google will give that kind of additional momentum to Amazon or any other competitors. The second thing is that most developers know that getting something out the door is more important that trying to get all possible features included before launch. Why should this be considered any different?

We know that Google has the Python expertise, why is it shocking that Python is the first language supported? Why would they want to offer a Rails environment and get slammed for it not being totally ready. Why not get the bugs shaken out with now before opening it up to other languages?

Of course, I could totally be wrong and this is an end-game play by Google but it seems way too early for that.

Starling is a light-weight persistent queue server that speaks the MemCache protocol. It was built to drive Twitter’s backend, and is in production across Twitter’s cluster.

I’m always a sucker for new infrastructure code but the problem is trying to figure out how to see if it will fit within whatever current architecture is already in place. I think though I might have found somewhere.

We have an aggregator grabbing various RSS feeds both internally and externally. Right now, during the parsing / adding to the database, we send off the information to our indexer for later searching. This is basically how the Topics pages are built dynamically.

Instead of sending the data during run-time, maybe it would be better to just add it to a queue with Starling for later processing. A job in the indexer could check the queue and index anything that’s there. This also would allow other applications to add things ready for indexing.