Blog of Jonathan Jaikoo Conway, web technologist, entrepreneur and daddy.

After a failed attempt to move away from Mephisto to MT earlier this year I was determined to find a blogging engine that I could easily adapt to my evil and wicked ways. I’d looked at a few ‘toy’ blogging engines written in Ruby, I’d even looked at some PHP based blogging apps but I’m a busy busy busy man. I don’t have time to swim through a mess of PHP (Wordpress) or write yet another blogging engine.

What I needed was something that gave me enough to build upon and yet was easily configurable… Hopefully through Ruby as that’s what I use for my day job. A while ago I’d stumbled across Feather. At first I wasn’t impressed, but recently I’d seen more and more Ruby based blogs convert from Typo/Mephisto to Feather. This peeked my interest enough to give it a try and here’s a few of the things I found:

i) Built on Merb (bonus for me as 70% of what I do these days utilizes Merb).

ii) Flexible plugin architecture based upon Merb slices. I groked this a lot faster than Mephisto plugins which had more of a learning curve.

iii) A whole plethora of plugins such as Mephisto importing, theming, Twitter and more.iv) Ability to have my url structure impersonate that of my old Mephisto blog. No more mod_rewrite hackery to make sure there aren’t any dead links.

Now onto the bad news. As Feather is a new project there are a few ‘rough’ edges. In order to get this blog up I did have to roll my sleeves up but with the code in GitHub it didn’t take me long to get my new blog up and running. I do miss the sweet admin from Mephisto but it’s only a matter of time before Feather too has this functionality.

For those of you interested in the technical details; this blog is running on a modified version of Feather, Merb 0.9.4(edge), Datamapper 0.9.4(edge), Apache 2.2 and Phusion Passenger with a Rack Up file found on Merbivore.

The theme was designed by me and is called Tweetish. I know for a fact this theme doesn’t work under IE 6 and I have neither the time or interest in supporting such an old and insecure browser. So for the best experience use Safari, Firefox or even Internet Explorer 7. The themes been enhanced with some pseudo CSS3 magic which you won’t see in IE 7.

Oh and the last thing. After much deliberation I’ve dumped the old comments for now and the whole blog is using Disqus. Let me know what you think, now time for me to get some sleep.

This is really exciting news. Aman Gupta has released a very comprehensive Ruby AMQP library . The API feels very natural and show’s off the power of RabbitMQ.

For more information, Alexis sent me a link to Ben Hood’s detailed explanation of the Ruby API who also gave an excellent presentation at LRUG last week on RabbitMQ. One of the things Ben showed off last week was the Jabber/RabbitMQ gateway which with all this talk of scalable messaging architectures lately (see Twitter) has definitely peeked my interest.

Well after a bit of waiting it looks like my talk proposal was accepted last week. So it looks like I’ll be giving an in depth talk at RailsConf EU about the different architecture and tools required to integrate video into a web application. I’ll also be covering the various gotchas and things to look out for so it will definitely aimed more at the intermediate to advanced audience.

On a personal note I’m really excited that I’ll be going back to Berlin as I still dream about the wonderful beer to be drunk there! Yummy!

UPDATE 2008-08-12
There’s a new patch for Varnish if you’re running on Open Solaris which can be found here

It’s been a long while since my last blog post. Things have been pretty crazy over at vzaar so apologies for lack of content.

I finally had a few minutes of time tonight to write up something that I’ve been meaning to share for a while now.

I’ve been talking with a few startups over the past few years who basically have a lot in terms of potential but little in the way of cash. I’ve seen many a startup fail not because they weren’t popular but because the gap between becoming popular and getting enough investment/paying customers through the door was too large. So lately I’ve been spending most of my waking hours thinking how best to squeeze every last bit of performance and penny from software to hardware. Even if you work for a cash rich company a little bit of prudence can save hundreds of thousands of pound over time and can be the difference between that nice little bonus cheque or a boot up the ass.

Storing terabytes of data soon becomes expensive and a headache to manage. Sure, ZFS eases things in terms of management but you’ve got to ask yourself. Do you have the time and resources to spend? Also as your site becomes ever more popular do you have enough cash keep the bandwidth fires going?

One of the things I decided to talk about was making a poor mans storage + CDN that can scale to quite large demands, while keeping costs as low as possible. Now before I go further on the CDN front, if you go to a CDN company and have enough cash/bargaining power then it will be cheaper and more performant for you to go with a proper content delivery network like Limelight. I make no false claims.

However, in the first stages for boot strapping a lot of companies simply can’t take the risk on bargaining a good rate with a CDN if they simply don’t know how popular the site will be. You maybe the most unpopular site on the web. On the other had you could be the next Facebook and there’s no way of knowing in the beginning.

The following solution I present merely simplifies and reduces the cost of content storage and delivery via everyones favourite Amazon web service that is S3. It isn’t a true CDN as you’ll probably realise, but if you’re reading thus far then you probably don’t care and this alternative is good enough.

My solution first started out as a combination of just Nginx, Merb and memcached with some rather complicated scripts in between sweeping the cached content from the file system.

After a while I came to realise that my solution was just way too complicated and that I was yak shaving. It was a few weeks later that I happened to read how Wordpress basically do the same thing but instead use Varnish to handle the caching of content.

They have a setup where Nginx proxies to Varnish. If the requested object is found in the cache then it’s served otherwise it’s forwarded by Varnish onto a PHP script that retrieves the requested object via S3 and serves it. They also have a ‘hawtness’ algorithm that decides whether to put the object into the cache but my solution doesn’t cover that. But it wouldn’t be hard at all to add it to the Merb app.

Our solution is very similar to Wordpresses setup except that we use a small Merb app to do the serving of assets from S3.

Why Merb and not Rails? I needed something with high concurrency and plus Merb can do some pretty funky stuff with regards to rendering which you’ll see in a bit.

First thing we need to do is setup Nginx. I’ll be setting this up on an Ubuntu server so change these instructions as you see fit. We’ll also be installing the fair proxy load balancer Nginx module (although not using it for now).

Download, unpack and build Nginx and the load balancer module

cd /usr/local/src
curl -O http://sysoev.ru/nginx/nginx-0.6.30.tar.gz
git clone git://github.com/gnosek/nginx-upstream-fair.git

tar xvzf nginx-0.6.30.tar.gz
cd nginx-0.6.30
./configure --prefix=/opt/nginx  --with-openssl=/usr/lib \
--with-sha1=/usr/lib --with-http_flv_module \
--with-http_ssl_module --with-http_gzip_static_module \
--add-module=/usr/local/src/nginx-upstream-fair

make
sudo make install

Next up is Varnish. Again we’ll be building from source.

curl -O http://heanet.dl.sourceforge.net/sourceforge/varnish/varnish-1.1.2.tar.gz
tar xvzf varnish-1.1.2.tar.gz
cd varnish-1.1.2
./configure
make
sudo make install

If you get any errors while building make sure you install any related lib’s. If you don’t know the lib name then doing a ‘aptitude search NAME_OF_LIB’ should bring up the package name.

Now to configure Varnish. Here’s a quick VCL file I knocked up. It’s not perfect and is designed to be tweaked, but is good enough to get started with. I’ve annotated the file to give you a quick heads up.

//We'll be using 2 merb instances b1 and b2
backend b1 {
        set backend.host = "127.0.0.1";
        set backend.port = "4000";
}

backend b2 {
        set backend.host = "127.0.0.1";
        set backend.port = "4001";
}

//If you want to be able to expire stuff from the cache
//from life cycle events in your Merb/Rails/Ramaze/Mack ap
//then you'll need to set which IP's you can do this from
acl purge {
        "localhost";
        "127.0.0.1";
}

sub vcl_recv {
        //We'll be serving 'small files from the first merb instance'
    if (req.request == "GET" && req.url ~ "\.(gif|jpg|swf|css|js)$") {
        set req.backend = b1;
        lookup;
     }else{
               //And everything else such as movies etc from another 
               //instance
        set req.backend = b2;
        lookup;
    }
     if (req.request == "PURGE") {
                if (!client.ip ~ purge) {
                        error 405 "Not allowed.";
                }
                lookup;
     }
}
 //Manual cache expiry stuff
sub vcl_hit {
        if (req.request == "PURGE") {
                set obj.ttl = 0s;
                error 200 "Purged.";
        }
}

sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache.";
        }
        //Just something to stop us serving terabytes of data to 
        //search engines and making is poor:(
        if (req.http.user-agent ~ "spider") { 
            error 503 "Not presently in cache"; 
        }
}

Save the above file in /usr/local/etc/default.net.vcl Now before we configure Nginx, lets download and take a look at the Merb app that does all the retrieval and streaming from S3. I’ve put it on Github and it can be found at http://github.com/jaikoo/pmcdn/tree/master.

git clone git://github.com/jaikoo/pmcdn.git

You’ll need to edit the s3_conf.yml file in the config directory and change the BUCKET_NAME in the assets controller before we can get going. After that’s done cd into the merb apps dir and start up two merb instances.

cd pmcdn
merb -d -c 2

Hopefully you shouldn’t have any errors and so it’s on to configure Nginx.

#/etc/nginx/nginx.conf
#I previously created a user for nginx www-data
user www-data;
worker_processes  1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    access_log  /var/log/nginx/access.log;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;

    #this should really go in include /etc/nginx/sites-enabled/*;
    upstream cache1 {
      server 127.0.0.1:8080;
    }

    server {
     #Change this to your asset server domain
       server_name asset1.jiggahaeyo.com;
        listen 80;
        proxy_max_temp_file_size 0;
        proxy_next_upstream off;
        proxy_read_timeout 60;
        proxy_intercept_errors on;

        error_page 404 @404;
        location @404 {
          rewrite .* /404.html last;
        }

      location / {
        proxy_pass http://cache1;
    }
 }

}

Now we need to start up Varnish on a different port first. It’s default port is port 80 and if your asset server was on a seperate machine from your main app then you could get away with not using Nginx and just use Varnish as is, but for our example we’ll start Varnish up on port 8080.

/usr/local/sbin/varnishd -a 0.0.0.0:8080 -f /usr/local/etc/default.net.vcl

Then it’s Nginx’s turn

/etc/init.d/nginx start

Next make sure you’ve already uploaded an asset to the bucket that you declared in the assets controller. To test this out if you have an image in the namespace images called test1.jpg then the url would look something like http://asset1.jiggahaeyo.com/assets/images/test1.jpg. If you’re tailing the Merb log you’ll see the first time you do this request the Merb app will query S3 and stream the file upstream. Hit the url again and you’ll again be served with the file, but this time the Merb app won’t be hit at all as it’s coming from Varnish.

The Merb app automatically grabs everything that matches /assets* and forwards it to the ‘show’ action in the assets controller. We then remove the ’/assets’ bit from the request path and pass that on as the path to the object in S3 which is retrieved and streamed back without loading the whole file into memory.

OK, this is all cool and stuff but I know you’re thinking that although we’re saving in S3 content serving costs because we’re now caching we are just pushing the cost of bandwidth onto our hosting provider. Bandwidth’s not cheap! Or is it? You’ll find that there are numerous hosts out there who give some fantastic deals when it comes to bandwidth. Joyent for example do a pretty good deal on their accelerators in which they give you 10TB of bandwidth. True the file storage available on those accelerators isn’t much but who cares when we’re using S3!

Also if say our app is partitioned we can have an asset cache on each of our partitioned nodes utilizing any spare bandwidth rather than eating all the bandwidth from just one asset server.

The knowledgeable of you out there are probably now thinking that this is all cool, but what about geographical caching? You get that with a true CDN. Surely the poor mans CDN should do this too? Well indeed it can! However, I’ll leave it to John Buswell’s article in o3 magazine on global load balancing which can be found here as he explains it far better than I could.

Working in a dual Ruby web stack framework has it’s perks. Today I wanted to use Paperclip in my Merb app as well as my Rails app. So ten minutes later I packaged up Paperclip as a gem and tweaked it ever so slightly to work on both Rails and Merb. The hardest thing was getting the manifest right.

It just goes to prove how well designed and extracted plugins like Thinking Sphinx and Paperclip are in order to allow me to do this so quickly.

Gem can be found here if you’re interested. Any bugs let me know.