Do NOT cheat on VUs!
Because load testing tools are usually priced by the number of simultaneous users it can generate, it is tempting to lower the think time or increase the simulated bandwidth and run the test with a lower number of VUs than needed. Then, estimate the number of users the system can handle based on this scaling this (flawed) data.
There is a subset of systems for which this is acceptable. These are primarily sites serving stateless static content or very atomic web services on systems extensively engineered for scalability. Unless your system falls into one of these very narrow categories, then DO NOT DO THIS! Even if you think your system is in one of those categories, we still recommend against this approach. Why? Read on…
Each user that is logged into a system is consuming resources. As one example, take the session state that is associated with that login. It is stored on the server, frequently in-memory, until the user logs out (or the session expires). On many systems, this accounts for a significant amount of the total memory load. Testing with a smaller number of users that move more quickly through their testcases will under-estimate this memory load, possibly leading to memory shortages in production.
As another example, consider database connections: Each simultaneous request may reserve a database connection from a fixed pool for the duration of the request. Simulating fewer users with a higher transaction rate will not exercise the limits of this pool or it’s ability to respond quickly when more are needed. It will not test the peak limits on the number of connections the database will be required to support. It will not exercise the amount of concurrent reads, writes and queries experienced by the database. The result, in production, could be slower access as requests are queued while waiting for a connection from the pool. Or worse, the pools will exceed the number the database can handle, leading to critical failures.
Even systems engineered for scalability have session-specific costs to account for. If the system is using non-sticky load balancing, then there is a cost associated with moving sessions from one server to another. If the total number of sessions is artificially cut by a factor of ten (or even two), then the cost of these moves will be drastically under-estimated. Performance could then suffer when real-world usage requirements swamp the available bandwidth between the servers.
These are just a few of the hidden problems that are not detected when testers use inflated transaction rates to compensate for not having enough VUs. So please, don’t cheat on VUs!
Chris Merrill, Chief Engineer
p.s. With our recent change to unlimited-VU licenses for Load Tester 5, the financial justification for cheating on VUs has been removed.