How to Develop a Robust Load Testing Plan
When I first started writing test cases with Load Tester, I found it easy to fall into the psychological trap of writing functional test cases. But load testing requires a different approach, and inadequate tests can cost you time and money.
Functional test cases (such as the unit tests popularized by JUnit) confirm the correctness of a system. These tests should be highly specific and have excellent code coverage. A good engineer approaches functional testing as though she were designing a jet engine: if any screw, flange, or circuit fails then the entire system is completely worthless.
Load tests should be designed to imitate the swarming behavior of real users. When load testing, we do still want to have good code coverage, but load tests that are inspired by your web application’s internal component structure may not provide the timely insights you need.
For example, the first test case I ever wrote simply logged into a piece of blogging software, and then immediately logged out. That was a very simple test that exercised a specific feature of the application. But it wasn’t representative of what a user would typically do. Load testing should start with the most ordinary behavior that we expect from our users.
A typical user would arrive at a blog either through a google search, a link from another blog, or from his RSS reader. He would likely be interested in a specific piece of information, which he would read and then he might follow any prominent links, such as on a list of “recent comments” or the title link to the front page.
Intuitively, testing only common cases will feel like less than due diligence. But if you spend two weeks writing up a comprehensive suite of test cases for 10,000 users only to discover a server misconfiguration that causes your site to fall over at 40 users, then that’s two weeks during which your engineers weren’t solving a critical problem with your installation.
Once a website passes the first gauntlet, we want to add cases that reveal bottlenecks, namely: bandwidth, TCP connections, server cpu, server memory, and database concurrency limitations. But even at this stage, only test cases that are founded on natural use patterns will reveal the most pressing problems.
In summary, a strong load testing plan follows this rough schedule:
- Start with a very small selection of the most frequently used scenarios. Deliver results to your engineers ASAP.
- After meeting basic performance goals, expand the scope of your test cases to capture a spread of common use cases.
- Add administrative functions or batch-processing jobs that could plausibly have a performance impact.
- Continue adding functions and edge cases as budget and schedule allows.
- Remember to re-test after each change to your system’s configuration.
Lane,
Engineer at Web Performance.