Load Test Configuration – using a stepped ramp
One of the most common load testing mistakes made by beginners is their configuration of the user ramp rate. This is best illustrated with an example configuration that we see frequently. This chart (from the load test report produced by our load testing software) shows the min/max/average page durations (pink, red, tan) along with the number of virtual users (blue) running the test:
What information does this chart give us about the system being tested?
- At 30 users, the average page duration is around 20 seconds and the maximum page duration is around 110 seconds
By contrast, look at this chart from another load test report:
This chart tells us:
- At 100 users, page durations are minimal
- At 200 users, page durations have not degraded much from 100 users
- At 300-400 users, page durations have degraded significantly with average page durations in the 50-100 second range. However, the minimum page durations indicate that some pages are still responding very quickly.
- After 500 users, page durations have started to degrade non-linearly and even the minimum page durations have degraded seriously.
To summarize – the first test tells us only that performance is not good at 30 users. If 30 users was the goal, this test has not provided any information to indicate how far away we are from reaching that goal. If the goal in the second test was to support 500 users, it is easy to see that the system can currently support at least 40% of the goal – and there is probably a lot of optimization work remaining.
Note that the test configuration shown in the first example can be useful. This kind of test is usually referred to as a soak test. The purpose of a soak test is to run for a long time at the anticipated load to see if the performance degrades over time. This is frequently used to check for memory leaks or limits on other resources. However, soak tests are generally not used until the system is meeting the performance requirements.
When using a stepped ramp, it is important to hold at each user level for long enough to allow every page in all the scenarios to be exercised – usually at least several times. If all pages are not measured during the step, conclusions drawn from the results may be faulty.
Chris Merrill, Chief Engineer