Load Testing a cluster of web servers
Ok – so you’ve spent months putting together a shiny new mission-critical web application, complete with multiple web servers, a few application and/or database servers. There is a load balancer in the mix or maybe you are using Microsoft NLB (Network Load Balancing). The system is nearly ready to deploy and it is time to run a load test. The first inclination is to fire up the load testing tool and throw the maximum expected load at the entire cluster. After all, there is no sense in wasting time with the smaller pieces, right? Wrong.
There are 2 very simple reasons:
- How will you interpret the results?
- If the system falls on its face, the political ramifications can be significant.
I’ll start with #2 – Since we became a provider of load testing services, we have seen many systems that fell on their face in a highly publicized first test of a system. After spending a lot of money on a new system, the project sponsors are now very worried. The opponents are lobbing tomatoes from the cheap seats. The project manager is taking a lot of heat. This doesn’t always happen, but when it does, the project can be derailed, along with the careers of those responsible for it.
#1 might require more explanation. If the system doesn’t perform perfectly on the first try (and in my experience they seldom do), you will have no idea where the problem lies. Is the web application optimized for good performance? Is there a problem with the load balancer? Is the web application not tuned properly for operating in a cluster? Are the web servers configured properly? etc.
In our experience, a cluster should never be tested until a scaled-down version has been tested – preferrably the smallest version that can be constructed without changing the architecture (for example: 1 web server, 1 application server and 1 database server). Ideally, this happens early in the development process.
The simple reasoning is this – until single server system has been tuned for best performance, there is little point in testing the cluster. The odds are very good that you will backtrack and test an individual server anyway. But more importantly, the odds of success for that first big test are much higher if you have already load-tested and tuned the single-server version for maximum performance.