Overall Architecture

I thought I'd write a bit about the planned architecture for predict.ly - or at least the plan so far.

DNS

The architecture starts at the DNS level, utilising IP Anycast for redundant DNS servers. This will be sourced from an existing vendor with good reviews - still to pick. Utilising round robin and monitoring the DNS system will distribute the traffic for each domain to the 2 servers providing basic traffic load balancing - or more correctly termed, load distribution.

The DNS service can determine if a server is offline and stop sending people to it within 5 minutes. I found some results posted about the effectiveness of this and they suggest 80% of users will fail over within an hour.

While this isn't bullet proof, it's better than nothing and combined with round robin DNS, only 50% of people would need to be fail-over so you would be operating at 90% at the dns level.

IP Level

At the IP level each server will monitor the other and if one becomes unavailable, the other will assume that IP address. This pretty much removes the need for the DNS failover however having it in place will allow for scaling later such as multi geographic location hosting.

This monitoring should be able to detect and recover an unserviced IP within 2 minutes.

Load Balancing

Still looking at solutions, however the likelihood is that this will be in the form of a http proxy distribution requests to http daemons on both servers. If the solutions merely distributed traffic without some intelligent balancing, then there might not be much point given the DNS balancing. However I am hoping to find something that will balance traffic out based on server load, opting for the least loaded.

MySQL

A basic master-master mysql setup using a meter free gigabit vpn should provide failover and high availability. Tests so far have been really encouraging.

Filesystem

I would really like a replicating filesystem but the setup looks like a nightmare. The advice seems to be go with rsync and that is probably what I will do.

Cache System

I am still looking into the cache system. It will likely be a mix of filesystem and memcached. Additionally there will be a small custom java server that will be used to aggregate writes to the database to reduce load.

Web Server

I'm looking at lighty for the web server running php with fastcgi. The question of op-code caches has yet to be resolved, I don't think I can afford the memory requirements.

Budget

Currently looking at $500 annually.