Is the last mile relevant to Load Testing?
One of our competitors has recently been emphasizing the use of their monitoring network during load tests. The pitch goes something like this: “With our last-mile monitoring network you can determine how different geographical regions will experience your web pages.” This solution has been implemented by using testing agents in the homes of real consumers to measure performance during a load test.
At first glance, this peaked our interest. However, after some analysis, we started to question the real value of this approach to load testing:
- Will this approach dilute the value of the load test results?
- Can the same information be obtained in other ways? Can you get it earlier in the schedule and at lower cost?
- What actionable results does this provide?
In general, the goal of load testing is to evaluate the performance of the application server(s) and network. The proposed method introduces another variable (the client network) which may reduce the value and/or reliability of the results. For example, imagine a scenario where your organization has completed a load test and found that improvements need to be made in the page load times. A few days later, changes to the application are complete and a second load test is executed. The results contain a surprise – the pages are loading slower, rather than faster. The obvious question is “were the slower pages caused by the network problems on the client end or by the changes to the application? Introducing another variable into the problem space has made the analysis more difficult. Load testing is difficult under the best of conditions – why make it harder?
On the other hand, there is value in seeing how long it takes web pages to load for real users – e.g. users on dial-up internet connections. We have brought this problem to the attention of many of our customers. But we did it without client-side testing agents. How? Simple, really. Our Load Tester software has the ability to show you how a web page will perform at lower network speeds – live in your browser! This can be done early in the development process (when problems are less costly to fix) – long before you deploy the application to real users. Additionally, the Baseline Analysis report in Load Tester can estimate how load times for a page will vary based on different network speeds. It will also show you how much bandwidth you need at your server to handle your expected user load. All of these features are available in Load Tester for free. We recommend using them before load testing begins. Naturally, Load Tester also has the ability to simulate various network speeds during a load test to ensure the load placed on your server reflects real-world users – while still providing consistent, repeatable tests.
Note that with either approach, you will need to know what kind of network connections your users have. There are a number of sources for bandwidth availability information, such as this US Broadband Penetration report. If you do not have this information about your user population then you will need to get it (or estimate it).
Finally – what can you do with the results? In most cases – nothing. Discovering that some users in a particular geographic region are experiencing slower response times than most others is a rarely a problem that your organization can solve. As a website administrator (or even corporate CIO), you have no control over the client networks. You could make use of a CDN (content delivery network) to deliver the content from locations closer to the end-user. In the countries with advanced networks such as the US, this will offer only minimal improvements (unless your system is already bandwidth limited – which is a different problem). When delivering content world-wide, a CDN could be very beneficial. But again, this result would be achieved by knowing your user population – not load testing.
To summarize, while we found the idea of adding real-world user machines into the load-testing scenario to be an interesting approach, we have not found any real benefits to it.
So I’ll put the question out to the readers – have you found this kind of test to be a valuable part of load testing? If yes – what problems did it solve for you that you were unable to solve with a more traditional load-testing approach? We’d like to hear from you!
Chris