Why you should not cheat on VUs
Because load testing tools are usually priced by the number of simultaneous users it can generate, it is common to use fewer VUs (virtual users) than called for by the test plan. To compensate, testers may lower the think time (the time between pages) in order to move VUs more quickly through the testcases and achieve a higher transaction rate than actual users would produce. The transaction rate may also be increased by raising the simulated per-user bandwidth – again to speed the lower number of VUs through the testcases more quickly. The system capacity must then be extrapolated based on these questionable results.
There is a subset of systems for which this practice is less likely to backfire. These are primarily sites serving stateless static content or highly atomic web services. If your system doesn’t fall into one of these categories, then cheating on VUs can have disastrous consequences.
Each user that is logged into a system is consuming resources. The most obvious case is the session state that is associated with that login. It is stored on the server, frequently in-memory, until the user logs out or the session expires. On many systems, this accounts for a significant portion of the total memory load. Testing with a smaller number of users that move more quickly through their testcases will under-estimate this memory load. If the memory usage was near the limits during testing, then real-world loads will experience poor performance due to memory constraints.
In addition, cheating on VUs under-represents the number of simultaneous connections maintained. This can manifest at several layers in the system. At the front end, web server may hit connection limits that were not exposed during testing. Deeper in the system, each simultaneous request may require a database connection from a fixed pool while the request is processed. Simulating fewer users and a higher transaction rate may not exercise the limits of this pool or it’s ability to respond when more are needed. It may not test the peak limits on the number of connections the database will be required to support. It may not exercise the amount of concurrent reads, writes and queries experienced by the database – and that concurrency is a primary factor in many load-related website failures. The result, in production, could be slower access as requests are queued while waiting for a connection from the pool. Or worse, the pools will exceed the number the database can handle, leading to critical failures.
If the system is using non-sticky load balancing, then there is a cost associated with moving sessions from one server to another. If the total number of sessions is artificially cut by a factor of ten (or even two), then the cost of these moves will be drastically under-estimated by the load test. Performance could then suffer when real-world usage requirements swamp the available bandwidth between the servers.
We recognized that everyone has budget constraints. Given the ridiculous price charged for load-testing large numbers of VUs by some load testing software vendors, it is not at all surprising that this practice is common. With the release of Load Tester 5, the price factor is no longer a consideration: Load Tester 5 provides unlimited load testing at an affordable price. No more excuses for cheating!
Chris Merrill, Chief Engineer