Load Testing Fallacies: It’s Not Ready for Testing
There is a new Load Testing 101 post over on the Dynatrace Blog. Overall, I think it is a pretty good article. However, it illustrates some of the fallacies of load testing that we hear from time to time. In this article, I want to dispel a fallacy that might be inadvertently reinforced by that post. Please note that I don’t want to speak for the authors – so I encourage you to read their article and judge their meaning for yourself.
Here is a partial quote from that post about why load testing projects fail:
“the testing tool reported errors with 2 or 5 concurrent users and response times going through the roof with only a fraction of the desired load. In all these cases the application was not ready for testing as some basic problems in the implementation or in the architecture did not allow any high load on the system.” … “The result of these situations is wasted time by the testers, the test software and hardware.”
I strongly disagree with that last bit, as well as the general gist of the section. If the first test revealed that the system could only handle a fraction of the desired load, that is exactly what you hope to find in early performance tests! When an application is only partially complete and the limited functionality does not meet its performance requirements, it is a strong indication of architectural flaws in the system. As the project moves forward and more features are added, is the software likely to perform better? In our experience, systems do not get faster accidentally. Finding this problem early in the development process gives the developers more time to adjust the implementation for better performance. With less existing code to re-work, the cost of changes will be much lower.
Below are some of the questions we commonly hear. While every situation is unique, I’ve included our general recommendations:
Q: When is a system ready for load testing?
A: As soon as there is something that a user can do in the system.
Q: The only thing a user can do in our system is login and navigate a few menus. Should we really bother load testing that?
A: Yes! A surprisingly large number of performance problems that we see are not caused by problems in the application logic. They are a result of the system architecture or the configuration and integration of various pieces of 3rd party software. The earlier they can be found, the less they cost to fix.
Q: What good are load test results from a system that is only partially complete?
A: It should be used both as a baseline for judging future results and as an early warning indicator. If the system doesn’t perform well at this early stage, it is not likely to perform well when a large number of more complex features have been added.
Q: Our system will need to support 20,000 simultaneous users and we expect to deploy with a small army of quad-core web servers behind a load balancer and a cluster of eight-core database servers with a bazillion gig of RAM each. All we have to test with right now is the web/app server running on our build/test machine and an old database server we found in the broom closet. Can we use that for load-testing?
A: Ideally, of course, you want to have a testing environment that is identical to your production environment – but that is not always practical or cost-effective. So yes, you absolutely can begin load-testing with a scaled-down hardware infrastructure. Naturally, you should only expect the hardware to support a fraction of the intended load – perhaps only a few hundred users in the above example. But if early testing reveals it only supports 10, you will gain much-needed time to explore optimization and hardware options. In fact, with load balanced systems it is especially important to test with a single server first: this provides a baseline for comparison when the full system is tested (I can’t tell you how many times we see customers make this mistake). Be sure that you set the proper expectations for anyone who is likely to see or hear of your test results – ensure they understand this is a scaled-down test and that you expect failures early on.
Q: So what things are important to have in place when we being early load testing?
A: It is important to have all the various software components installed and configured as you plan for your deployment environment. There are obviously exceptions – particularly in relation to clustering and load-balancing with multiple servers. Having the correct versions of the 3rd-party software components is pretty important here.
Q: What kind of problems do you expect to uncover when the system is only partially complete?
A: Misconfiguration of servers and software. 3rd party frameworks that have inherent architectural performance limitations. Connection pools that are too small.
I hope this clarifies why you do not need a fully working system to begin performance testing. If you have any questions you would like to see added on this list, please comment below!
Chris