With Web 2.0 applications facing growing audiences with data hungry appetites MOM has suddenly had a resurgence.
Messaging systems are a great way of scaling your site by placing operations/messages on a queue and processing them asynchronously. This allows you to get quite imaginative with your architecture and distribute worker processes in geographically disparate locations if you so wished and also distribute/delay the actual processing needs as and when you can.
This can be used with great effect with services such as Amazons own simple messaging offering known as SQS and EC2. As soon as load starts peaking on one EC2 node you can have a script that calls out to a mother-ship to instantiate another EC2 instance and as soon as it’s ‘born’ starts processing messages off the SQS queue. If you wanted you could even have the original load stricken node contact the mother-ship via SQS (probably a more reliable solution). This is one of the architectures I’m currently advocating as EC2 is stateless and rather than worry about database replication across transient nodes, I try and share as much information as I can using messaging and web-services. I’m currently backing off exposing REST style interfaces for non public facing services purely for performance and scaling reasons. Don’t get me wrong, I love REST and for public a public facing API it makes life a lot easier. But as Zed Shaw says, HTTP isn’t exactly the most performant protocol.
I’ve seen people try and achieve similar architectures using a database and their own cobbled together mesh of scripts and while this may work great on small sites, you’re just pushing your scaling issues and problems onto your poor DB. And as anyone who’s had the misfortune to be stuck in a room with me would tell you, I hate relational databases on multiple fronts, they’re so 1980’s and I’ll tell you why in another blog post.
It’s seems these days that we’re currently quite spoiled with the cheap/open messaging solutions that are available. A few years ago most of what I had to play with were proprietary and expensive offerings from various large vendors. A few that I’ve played around with lately have been:
- ActiveMQ – An open source solution utilising STOMP. Check out ActiveMessaging for Ruby/Rails.
- Jabber – While it’s not really intended for reliable delivery of messages, it does have a lot of support and with quite a few Ruby clients available.
- SQS – Amazons offering that is easy to plugin solution if you’re using EC2 as you don’t need to worrying about setting up the messaging infrastructure between distributed nodes that aren’t necessarily sharing a VPN. ActiveMessaging has support for SQS though I kinda feel it’s bit of a chain saw to a bunny rabbit as SQS really is a very simple service as it’s name suggests. I wrote my own lightweight SQS Rails plugin a while ago and will probably dust it off in the next week or so.
- RabbitMQ – Now this is the matrons knockers when it comes to kick ass, ultra fast and scalable messaging. It simply rocks, with performance off the hook. It’s written in Erlang and supports the AMPQ protocol. Unfortunately I haven’t seen much in the way of Ruby support. I’ve played with QPid’s (Apaches own open source AMPQ offering) Ruby client library but haven’t actually tested this against RabbitMQ, so you’re mileage may vary.
So there you have it, messaging. It rocks, it’s sexy and what’s more it’ll make you sexy… Honest guv’nor.
