Removing Ad-servers and Click-trackers from your Load Test
We field a lot of questions about load testing websites with 3rd party components on the pages containing advertisements or user tracking mechanisms. For many of our users, we recommend leaving these out of the load test entirely. For some users this causes some concern over maintaining the realism of the load test. In this post, I’m going to discuss the advantages and disadvantages of dealing with these in a load test – and then provide some quick instructions for easily keeping these services out of your load test.
I’ll start by looking at the possible advantages of including these services in your load test:
- The entire system is load tested – including these third party systems
- The test is more realistic because it does more of what a real user would do
That first one seems pretty compelling: Surely a thorough test should measure the capabilities of these important 3rd party services, right? There are several reasons this may not be realistic, depending on your situation:
- The results may not be useful. Since you do not control these systems (and you are testing a live system), there is little reason to believe that the performance witnessed during the load test will not change day to day (or minute to minute) as the 3rd-party system(s) experience traffic changes due to their other customers or changes to their system.
- The results may not be actionable – there is little you can do to improve the performance
As for the reality of the test – there are varying degrees of realism in load testing. If your page architecture has been well-optimized, then the end-user should not experience noticeable changes in the page render time even when these 3rd party systems are slow. Note that page render time refers to when enough of the page is visible for the user to start interacting with it. This is distinctly different than the total page load time. Most load-testing tools (including ours) will measure total page load time, rather than the page render time (which can be very subjective) – so in many cases the tests will better reflect reality when these 3rd party servers are excluded from the test.
Now lets talk about the disadvantages:
- The user-tracking metrics collected for your site could be affected by the load tests – artificially inflating the visit rates.
- The ad vendors, if they are paying for each impression on your site, may have fraud-detection algorithms triggered by the load test, possibly resulting in actions that could interrupt the ad revenue for the site.
- The testcases will typically be much harder to configure. Note that this is not unique to our product – all load testing products will require additional customization work to handle these cases.
If you agree with me up to this point, you probably want to know how to easily keep these factors out of your load test. From the Window menu in Load Tester, choose Preferences > Web Performance > Recording > Transaction Blocking. You will see this:
Press the Add Host button to add more hosts to the list. Note that wild-carding is supported, so if the 3rd party system has many hosts, such as server1.adsRus.com, server2.adsRus.com, etc, you can simply enter .adsRus.com to block them all.
After entering these hosts, record your testcase and you will find that they have been automatically left out of the testcase.
If you have any questions about this topic or load-testing in general, please use our support system or give us a call!
Chris