The Purpose of Load Testing
Why am I doing this?
Even a well-executed Load Testing effort may fail if it does not answer the right questions…or answer them in the right way. The “right questions” might appear obvious, such as “How many hits/sec can the server handle?” Such questions, while well-intentioned, may not be the questions that really need to be answered. They may simply be the easiest questions to ask…and answer.
From a higher-level strategic level, the real questions sound something like:
1. Is this system ready to deploy?
2. Can we make this system available to another 350 users?
3. We deployed the system and the performance is terrible. Find out why and fix it!
4. The last time we ran a new advertising campaign, the web server crashed. Is it ready for the next one?
Digging in a little deeper, these can be translated to:
1. Will this new system meet the expected performance goals for the intended deployment?
2. Can this existing system meet the expected performance goals if the user base is increased by NNN users?
3. We already know the system can not meet the expected performance goals. Replicate the problem so the developers can fix it….Then see #1.
If you read those carefully, there is really one question – Will the system meet the expected performance goals? Now, you may be asking: Isn’t that where we started — with a question like “How many hits/sec can the server handle?”. Yes and no. Yes, those questions sound a lot alike. But instead of getting distracted with metrics like hits/sec, we are now thinking in terms of performance goals. “But isn’t hits/sec a performance goal?” Yes, but not a very useful one. It is easy to collect and measure, but with few exceptions, it doesn’t tell us enough about the user experience. And THAT is what most load tests should be trying to measure: the user experience. In the end, the system performance will be defined by the user experience. The number of hits/sec is irrelevant to the users – they care about how the system feels. So, a load test should determine if the performance goals are met…and the performance goals should reflect an acceptable user experience.
Now let’s get back to the subject of this article – “The Purpose of Load Testing”. In most cases, the purpose of load testing should be to determine if the system will meet the performance goals under the expected usage scenarios.
So…if not hits/sec, what are the performance goals? From the users perspective, it is how long it takes to complete a given task. As the user attempts to complete a task, they will divide their time among a few primary activities:
1. Figure out how to accomplish the next step in the task
2. Perform the step
3. Wait for the step to complete
#1 is characterized by reading the page to determine what comes next. In many systems, #2 is data entry and pressing the submit button. Collectively, #1 and #2 are generally referred to as think time. Note that the use of AJAX techniques may appear to blur the lines between 2 and 3 a little – since each step can become a much smaller unit. But the principle still applies — just at a smaller scale.
While #1 and #2 are certainly important to the user experience – they are defined by the design of the user interface and how fast the user is working — and therefore mostly outside of the area of performance testing. When load testing, we are primarily interested in the duration of #3 – where the user is waiting for a result from the server. This is generally referred to as page duration or page load time. This measurement is by far, the most valuable data we can collect to determine the user experience.
The second most valuable data we can collect is the number of errors encountered. An error can be any time a step in the task results in an unexpected, incorrect or undesirable result…such as “connection refused”, “server too busy” or “unexpected application error”.
Returning again to the subject of this article, what is “The Purpose of Load Testing”?
The purpose of Load Testing is to simulate a realistic user load and measure the user experience to determine if the performance goals are met.
That was a really long-winded discussion to arrive at a simple answer for a seemingly simple question. Nonetheless, I’m frequently confronted by testers (and managers) that are focused on some metric, like hits/sec or time to first byte, and miss the bigger picture – what is the user experiencing? So I think it is very important to approach the problem with the right purpose in mind.
Now, determining the numbers that go with the performance goals are…well, that can be a challenge all on it’s own. Hopefully your user representative or project manager is prepared to answer those questions. You should be looking for answers like:
- 90% of the pages should load within 6 seconds
- At 90% of the maximum expected load, there should be 0 errors
- At the maximum expected load, no more than 1% of pages should result in errors
The answers may be simpler or more complex, but there must be specific numbers assigned to the goals…otherwise there is no way to determine if the tests indicate a system that passes or fails to meet the goals.
Now that we know why we are testing, next time I will share some thoughts on choosing what to test.
Test early, test often!
Chris, Engineer at Web Performance
Note: If you’ve made it this far, it probably doesn’t apply to you, but in the interest of completeness I will point out that there certainly are cases where determining specific metrics such as hits/sec and MBytes/sec are the end goal. In these cases, pursuing these metrics may take you down a path that is different from this series of articles. These are, however, less common.