Load Testing Anti-Patterns: Building the Perfect Test
We run into a wide variety of customers who need load testing software and/or services. While many are working their way through the process on-the-fly, others have toiled long and hard to develop a thorough testing plan, complete with detailed descriptions of exactly how the load test should be performed. On occasion, this testing plan is incredibly specific – detailing exactly how many users should be doing this or doing that, exactly how many users should click a button at the same time, exactly which search result the user should follow, etc.
While I applaud the amount of effort put into conceiving this plan, I fear that in many cases much of this effort was wasted. One of the fundamental truths of load testing is that it is a simulation. It is a simulation of one potential period of usage that hopefully represents an approximation the expected real-world usage. My use of the terms simulation and approximation is intentional – I should probably throw in ‘guess‘ and ‘hand waving‘ as well. No matter how much time and effort goes into the planning and execution of a load test, it is infinitely unlikely that the test will be identical to any traffic period ever experienced by the site.
That’s right – I said it: a load test will never produce exactly the traffic pattern encountered in real-world use. I can hear testers all over the world shouting “Heresy!” Nevertheless, I speak the truth. And I’ll take it one step farther: the same load test executed twice in a row will rarely produce exactly the same result (depending on how precisely you measure).
Why? Different users will traverse the site in different ways. Even when the use case is very limited (there is only one way to accomplish the task) the arrival rate of new users will vary and the speed at which the users progress through the pages will vary. Some users will make mistakes and need to back up. Others will become distracted and abandon the operation. Even with an automated tool attempting to perform exactly the same test, the many layers of technology will introduce unseen effects. Deep down, the operating system kernel will schedule thread execution slightly differently based on the past state of the machine. Disk access time varies based on which disk sector is being accessed. Ethernet has designed-in variability to handle network traffic collisions.
By now you are probably asking – what is my point? I can’t do anything about that! My point is a very simple one, really: Don’t sweat the small stuff.
Load Testing is about finding the most likely breakpoints in the most cost-effective manner. Do this by developing testcases quickly and running lots of tests with lots of variations. Time spent tweaking a testcase over and over to get it perfect is time that is NOT spent adding some other scenario into the mix. The quicker you get test results back to the development team, the less expensive they are to fix.
Trying to build a perfect test is a testing anti-pattern because it delays test results and limits test coverage. Don’t fall into that trap.
Test early, test often!
Chris, Chief Engineer at Web Performance