revjim.net

I have a web server. I have a backup web server. I have a database server. I have a backup database server. Everything is hunky-dory. Well… everything WAS hunky-dory. Now we have too many users. What do I do?

I have a few thoughts.

First, an option that will work for now, but may not be the best in the long run: For the web side of things, I can move different applications to different boxes. I can use Apache to redirect users to the proper box in the event that they don't know where to go, or have an old URL. On the same token, I can build database servers for each of the web servers, and alter the applications on those boxes to point to the appropriate database server.

Secondly, I have a better option: Use a load balancer and multiple web servers. The load balancer can be setup to ship requests for different applications to different servers in the event of applications that require lots of non-database data, and can ship other requests evenly amongst a pile of servers. Getting more performance is merely a matter of buying a new box, copying the proper data onto it, and throwing it into the loop. I can use IP address spoofing methods to assist the load balancer from having to rewrite both incoming and outgoing packets. Each server will have 2 NICs in order to handle internal and external communication properly. The problem here comes in with database servers. We have a lot of data, and a lot of applications. Not all of our applications use abstracted database calls, and of those that do, more than one abstraction layer is employed. Moving to a "multiple slave with one master" style setup would be a very time-consuming move as all of our code would have to be rewritten to handle a "read" server and a "write" server, and to take care of instances when a "read" operation should be performed on the "write" server. Additionally, we don't add any scalability because all the writes still occur on ONE server. When that ONE server reaches its peak, we must upgrade that machine. In the corporate environment that I am speaking of, buying new hardware and upgrading an entire machine is not something that is easily done.

LiveJournal used to have a ONE master setup, and, as the userbase grew, the server capacity dwindled. They eventually switched (and just finished recently) to a custom designed solution for distributing the load amongst MANY master servers (known as Clustering in LiveJournal speak). I could attempt a configuration similar to this. The problem is that, LiveJournal had ONE application to convert and think of and take care of. I have MANY applications, with many different types of data that will need to be converted. Coming up with a custom solution that suits all of them will most likely not be an easy nor time-effective task. Additionally, all applications written in the future will have to fit into this custom clustering method. This could prove to be disastrous.

I'm just looking for input from those who might have some experience in this arena. I am using Apache, and MySQL for server software. Any tips or pointers would be appreciated.