Accuracy of Load Testing with Real Browsers vs Virtual Browsers - Web Performance
Menu

Accuracy of Load Testing with Real Browsers vs Virtual Browsers

Comparing the two directly is not an apples-to-apples comparison, because they measure different things. Actually, Load Tester measures a lot of the same things for each, but what is generally considered the most important metric – Page Duration – is actually measuring a different aspect of performance in each case.

Virtual Browsers

Virtual Browsers work at the HTTP layer – they send the same HTTP messages to the server that real browsers would send. The Page Duration measures the time from the beginning of the first request that is sent to the server to the end of the last response for a resource on that page.

http_duration

Our Virtual Browsers will use the same number of connections to the server as a real browser. And it will distribute the requests amongst those connections in a very similar way: it will use inactive connections first, connections remain open for a while, etc. When done correctly, the target application cannot tell the difference between our Virtual Browser and a human operating a real browser. Doing it correctly varies from easy to challenging, depending on the complexity of the application.

Real Browsers

Real Browsers are, well, REAL browsers that are driven by our virtual user instead of a human user. The driving takes place via APIs into the browser that are designed for automation (courtesy of the WebDriver / Selenium project). For example, a Go To URL step instructs the browser to navigate to a URL. The Duration of that step measures from the time the command is sent to the browser until the browser reports completion (or failure). In the case of the Go To URL command, the command is complete when the browser fires the “On Load” event. This step will include the amount of time to get all the resources from the server – which is what is measured by the Virtual Browsers. It will ALSO include the amount of time the browser takes to render the page on the screen, which is not measured by the Virtual Browsers (since they never render the page).

with_render_duration

Our Real Browser virtual users cannot be distinguished from human users and the complexity of the application has little impact on the ability of the tester to get accurate results.

Which one is more accurate?

In the hands of an experienced tester, both can be equally accurate, even though they measure performance a bit differently. Both methods also exercise the server equally. One includes render time and one does not. The importance of this difference can not be answered in a general way – each project and organization will place a different value on it.

In the hands of an tester that is not experienced in the intricacies of HTTP and web application construction, the Real Browser virtual user may yield more accurate results with less effort.

How can Virtual Browsers be just as accurate…

…if they don’t measure render time? The most common objective of load testing is to measure how the performance of a system degrades under load. It is true that if the browser requires 2 seconds to render page after receiving the page resources, the virtual browser does not report this time. Here is an example…this chart shows (fictional) test results from a load test that measured performance at 10-50 users using virtual browsers:

virtual-browser-chart

This next chart shows test results from a similar test using real browsers. You can see that it includes the rendering time required by the browser.

real-browser-chart

You can also see that the render time is constant – it does not change due to load on the server. That extra second is determined by the page design and the computing power available to the browser. The total duration required may increase as the server slows, but that increase is DIRECTLY proportional to the increase in response time from the server. They both measure the performance degradation under load equally:

real-vs-virtual-browser

How do we know this is true?

We know this to be true because we have been doing this for a long time – we run load tests for customers nearly every day and we research the topic tirelessly. But we don’t expect you to take our word for it. Open up FireBug on a page and you can observe the render time of a page. Hit that page again when the system is experiencing load-related performance problems and you will see that the page render time is constant while degradation occurs in the network time. If you are interested in reading more about how we test our own software to ensure accuracy, you might want to read through another post: How do you know if a load testing tool is accurate?

Chris Merrill
Chief Engineer

Add Your Comment

You must be logged in to post a comment.

Resources

Copyright © 2024 Web Performance, Inc.

A Durham web design company

×

(1) 919-845-7601 9AM-5PM EST

Just complete this form and we will get back to you as soon as possible with a quote. Please note: Technical support questions should be posted to our online support system.

About You
How Many Concurrent Users