Menu

Load Testing Fallacies: The Test Scenarios Must Be Completely Accurate

Last time, I talked about why it is ok to start testing early in the development process. I’m going to continue that thought process to discuss load testing without complete performance requirements.  This Load Testing 101 article says “If the real end user is going to do work with your application in a totally different way than you test you are as good as with no testing at all.” While there is a nugget of truth hidden in there, it is easy to take away the wrong understanding.

One interpretation of that statement would be that “you must have a load test that represents the user activities and data with 100% accuracy. If you achieve less than that, your test results will worthless and many hours of effort will be wasted”. If this was the intended message, then I heartily disagree!

Let me dig out the nugget of truth from the quote: The more accurately that your test scenarios and test data model real-world user behavior, the more valuable the test results will be. This is quite true.

Now, let me put a big caveat on top of that: performance testing (like many other endeavors) follows the law of diminishing returns – at some point, additional effort spent improving the testing will cost more than the additional benefits gained from the effort. In performance testing, this point frequently comes very early in the process.

Why?

Because most performance problems are not limited to a specific usage scenario. Most performance problems are systemic – they affect a wide variety of scenarios. We perform a lot of load testing for our clients and analyze a lot of test results from customers using our Load Tester software. The most common problems are infrastructure, configuration and architecture – not one specific code path.

Am I recommending that you test 1 or 2 scenarios and call it a day? Certainly not.

My point is that testing early in the process, without a 200-page comprehensive test plan, is ok. In fact, it is almost always better than testing 2 days before deployment because it took that long to decide what to test. If early testing reveals a fundamental flaw in the system architecture or a 3rd party package, nobody is going to care that the user scenario that uncovered it does not exactly match real-world usage statistics.

Test early, test often!

Chris Merrill, Chief Engineer

Add Your Comment

You must be logged in to post a comment.

Resources

Search

Copyright © 2017 Web Performance, Inc.

A Durham web design company

×

For Prices Call (1) 919-845-7601 9AM-5PM EST

After hours? Prefer email? Just complete this form and we will get back to you as soon as possible with a quote. Please note: Technical support questions should be posted to our online support system.

About You
Product Needs
×

Request a Quote

Our experienced performance engineers have tuned hundreds of systems for companies large and small, and know just where to look, saving you time to market and money. We'll run your website through a complete performance evaluation, then tell you exactly how many users your site can support, including such important details as the effects of "the last mile." We'll also pinpoint potential problem areas and give you a full report detailing what needs to be fixed and where.

About You
Product Needs