Load Testing by Example

Best Practices

Overview

The purpose of this tutorial is to present some of the best practices we have developed at Web Performance. Through our Load Tester product development and load testing service experiences, we have found some easy tips and general strategies that can help make load testing efforts more successful. They are presented here in hopes of making all load testing efforts more successful.

General Strategies

As a tester, our job is to provide the best possible feedback on the readiness of an application as early as possible in the development cycle. Of course in the real world, deadlines are always tight and testing begins later than it should. We've found a few simple rules that have helped make our load testing projects more effective.

1. A reasonably accurate result right now is worth more than a very accurate result later. It is well understood that changes become more expensive later in the project. As a result, the quicker that performance problems are found, the more easily they are fixed.
2. Test broadly, then deeply. Testing a broad range of scenarios in a very simple manner is better than testing a few testcases very deeply. Early in a project, an approximate simulation of the real world is acceptable. Time spent getting each testcase to exactly mimic the predicted real world scenario or testing dozens of slight variations of the scenario is better spent testing a wide range of scenarios.
3. Test in a controlled environment. Testing without dedicated servers and good configuration management will yield test results that are not reproducible. If you cannot reproduce the results, then you cannot measure the improvements accurately when the next version of the system is ready for testing.

Testing Steps

At Web Performance, we generally recommend that beginners follow a series of steps like:

Select the scenarios to be tested
Pick a scenario and configure it
Load test the scenario
Repeat steps 2-3 until each scenario is ready, as schedule permits
Combine the load tests and run a combined test

Scenario Selection

When selecting scenarios to test from the group of all possible scenarios, we rate each scenario by two aspects: Difficulty of Simulation and Importance. Unfortunately for beginners, the Difficulty of Simulation is hard to judge without some experience with the load testing tool and some understanding of the underlying application implementation details. But in general, scenarios that are short and easy to verify are frequently also easy to simulate. Verifying a scenario refers to the ability of confirming that, the scenario was successful and performed the desired action in the system, when simulated in the testing tool. For instance a scenario that sends an e-mail is easy to verify - just check to see if the e-mail was sent. Similarly, anything that performs an add operation that can later be queried is easy to verify.

Once the scenarios have been rated on the Difficulty of Simulation and Importance, we order them. The scenarios that are easy to simulate and very important come first. Then we choose the scenarios that are either easy to simulate but only somewhat important or not as easy to simulate but very important. We usually alternate those - favoring the easy ones when schedules are tight. We favor the easy scenarios because experience has shown that the factors that cause performance problems are rarely related to the importance of the scenario. There is a lot of value in testing as many scenarios as possible very early in the testing effort. This gives the developers more time to troubleshoot and tune the application when problems are found.

Configuration Management

Good configuration management is essential to getting consistent load test results. If the test environment is frequently changing, it will be difficult to compare the result of one test to another. It will also be difficult to troubleshoot problems, since it will not be easy to rule out environmental factors as affecting the test results.

Some pointers:

Ensure the testing hardware is dedicated to the testing effort. Testing on production systems that are being accessed by other users is a common mistake. Since it is generally impossible to tell what other users are doing, it will be difficult to determine if performance anomalies are the fault of the application being tested, the test configuration or the other users on the system.
Ensure the application data (i.e. database) is in the same state for every test. As databases get populated with greater amounts of data, queries get slower. This may mask other improvements over time. In general, the database should be restored to a known state before every test. This can also make the testing effort much quicker - because it allows certain assumptions to be made in the testcase construction and configuration that can save a lot of time and effort.
Change only one thing in the test environment at a time. Changing more than one factor at a time makes it difficult to assess the effectiveness of any of the changes.

Part of managing the application data (database) is populating it. When you do not have good testing tools available, you might consider doing this by injecting data directly into the database. But with a good load testing tool, you have the ability to easily populate the database with large amounts of data with relative ease. Since the data is entered via the same GUI that real users would be entering the data, you have the added assurance that the data has passed all of the cleansing and verification rules of the application. Once you have created and configured a testcase to add users to your application, for instance, you can easy populate the database with hundreds or thousand of users. You can even create different populations, backing up each one, to allow for comparison testing. This can help answer questions like "Will the performance change in 6 months after 50,000 users have been registered?"

Testcase Construction

First, I want to clarify our terminology. When we describe the way a user interacts with a web application, we refer to scenarios. Once we have recorded the scenario and begin configuring it for simulation, we refer to it as a testcase. It is possible to have multiple testcases for the same scenario. For example, minor variations might be considered the same scenario, while requiring separate testcases for Load Testing purposes.

Creating and customizing a testcase generally follows these steps:

Record the scenario using a browser while the testing software records the actions.
Configure the testcase to simulate multiple user identities (for any system with a login)
Customize the testcase to supply different inputs (search keywords, for example)
Replay the testcase to verify correct simulation

Note that when first working with a new application, we will typically perform replays after steps 1 and 2, to ensure the simulation is correct before moving on to the next step.

Testcase complexity

In order to get quick results, we try to keep our testcases as simple as possible while still reflecting the expected usage of the system.

One of the ways we do this is to model variations of a scenario as separate testcases. For example, if some variations of a scenario require a user to visit a confirmation page while others do not, we would model these as two separate testcases. One reason is that modeling if-then logic in a testcase adds complexity and generally requires a script-based testing tool. Maintaining scripts is labor-intensive and increases the overall cost of testing. Second, unless measuring both variations is critical, we will generally only choose one of the scenarios earlier in the testing process to allow us to test more broadly. In later stages, we will come back to address the remaining variations.

Feedback & Comments

Comments about this report may be posted at the company blog post.

Metadata

Version History

v1.0 - 1st public release (10 September 2007) v1.1 - email cleanup (23 Jan 09)

Resources

DesignHammer – A Durham web design company

919.845.7601 Mon – Fri 9am – 5pm ET