Build Realistic Testcases
When you have identified the first group of scenarios to test, it is time to turn them into testcases, which is the term we use for the representation of the scenario in software. In it’s simplest form, a testcase is a list of pages to visit. In the simplest tools, it may simply be a list of URLs. With most tools, the testcase is a program generated by the tool and customized by the tester. More sophisticated tools, such as Load Tester, present the testcase visually.
The testcase represents not only the page visited but also the data that the virtual (simulated) users will enter in form fields, the amount of time they spend reading pages, how to extract dynamic data from pages, rules to validate what a successful page looks like, etc. Regardless of the representation, there are a few crucial rules to follow to ensure the accuracy of the load placed on the system.
- Use real, unique, user identities. Even if the system allows the same user to login to multiple independent sessions, it is not a good idea to simulate more than one session per user identity. Ideally, the ratio of unique of identities used in the test to the total number of identities in the system should be close to what the system will experience in production use.
- Use realistic data. Wherever the simulated users are entering data, it can be important to enter data that mirrors the variety of real-world data. When that data is written, indexed and searched by the system, the performance of those operations can be affected by the variety or conformity of the data. If real users will all be entering distinctly different values in fields, then the virtual users should be doing the same.
- Think time should be realistic. When recording a scenario in the browser, it is natural to speed through the steps very quickly, especially when doing it several times over the course of a few days. Real users, however, would take much longer to enter data and decide what to do next. We call this think time – the time spent between page requests while reading the page or entering data. Take time to find out how long real users spend in each scenario. Having accurate think times (on average) is very important. Think times that are too long can place much lower load on the system than real users would and thus the system may not be able to handle the load in production. Likewise, think times that are too short will simulate a much higher load – possibly resulting in a project delay to over-build the system needlessly.
- Real users have a variety of reading and typing speeds, so the think time should vary as well. We generally vary the think time +/- 25-50% from the average, depending on the complexity of the operation the user is performing. A realistic variation of think times is important, but not nearly as important as getting the average correct.
Chris Merrill, Chief Engineer