Load Testing Fallacies: The Test Scenarios Must Be Completely Accurate
Last time, I talked about why it is ok to start testing early in the development process. I’m going to continue that thought process to discuss load testing without complete performance requirements. This Load Testing 101 article says “If the real end user is going to do work with your application in a totally different way than you test you are as good as with no testing at all.” While there is a nugget of truth hidden in there, it is easy to take away the wrong understanding.
One interpretation of that statement would be that “you must have a load test that represents the user activities and data with 100% accuracy. If you achieve less than that, your test results will worthless and many hours of effort will be wasted”. If this was the intended message, then I heartily disagree!
Let me dig out the nugget of truth from the quote: The more accurately that your test scenarios and test data model real-world user behavior, the more valuable the test results will be. This is quite true.
Now, let me put a big caveat on top of that: performance testing (like many other endeavors) follows the law of diminishing returns – at some point, additional effort spent improving the testing will cost more than the additional benefits gained from the effort. In performance testing, this point frequently comes very early in the process.
Why?
Because most performance problems are not limited to a specific usage scenario. Most performance problems are systemic – they affect a wide variety of scenarios. We perform a lot of load testing for our clients and analyze a lot of test results from customers using our Load Tester software. The most common problems are infrastructure, configuration and architecture – not one specific code path.
Am I recommending that you test 1 or 2 scenarios and call it a day? Certainly not.
My point is that testing early in the process, without a 200-page comprehensive test plan, is ok. In fact, it is almost always better than testing 2 days before deployment because it took that long to decide what to test. If early testing reveals a fundamental flaw in the system architecture or a 3rd party package, nobody is going to care that the user scenario that uncovered it does not exactly match real-world usage statistics.
Test early, test often!
Chris Merrill, Chief Engineer