Real browsers vs virtual browsers: which to choose for your testing?
Since version 6, Web Performance Tester has supported two different ways to simulate user behavior on a website for testing: real browsers and virtual browsers. These two methods take very different approaches to the problem and each has different advantages and disadvantages. Those are not always obvious at first glance, so I’d like to run through the key difference to help you decide.
But first, a brief description of the two approaches:
Real browsers – When using the real-browser approach, the test is defined in terms of the actions the that a human would take in the browser in order to complete the scenario. For example, clicking on a button, typing text into a field, etc. When executing a real browser testcase, a real browser is started and controlled through automation APIs (including Selenium/WebDriver) and used to perform the actions in the scenario.
Virtual browsers – The virtual browser approach is defined in terms of the HTTP messages exchanged between the browser and the web servers when a human is using the browser to complete the scenario. These messages consist of HTTP request and response pairs that allows the browser to fetch all the content required to render each web page. When executing a virtual browser testcase, the innards of a browser are simulated – making connections to each server, sending requests and processing the responses as needed.
It is worth pointing out: when performed correctly and completely, the server cannot tell the difference between a real browser and a simulated browser, regardless of which approach is used.
So which one is better? It really depends on the specifics of your project requirements. Lets talk about the differences and how they could impact your testing.
Accuracy
The two approaches measure different aspects of the user experience: a real browser testcase measures events that happen in the rendered page and are visible to the user, while a virtual browser testcase measures the response time of network requests that must complete before the page can be rendered.
Real-browser durations tend to be higher than virtual browsers because after the response messages arrive, the browser needs additional time to render the page. However, because that render time is a client-side effect, it has no impact on measurement of site performance under load – that processing time is the same whether the server is responding fast or slow. For each approach, measured durations will increase by the same amount as the server slows.
Real-browser testcases can measure things that happen in the UI and are visible to the user, so it can help you answer questions like: “After the user does X, how long does it take for button Y to become enabled?” Virtual-browser testcases can measure the performance of network requests that are invisible to the user. It can help answer questions like: “This is page is slow. Which resource on the page is causing the slow-down?” Both can be valuable, depending on your needs.
Learning curve
Most activities in Web Performance Tester (WPT) will be the same, regardless of which approach is used. However, building and customizing testcases is very different.
When working with real-browser testcases, the tester will need to know how to locate elements on the page, in order to interact with them. The tester will need a basic understanding of HTML elements and may benefit from knowledge of CSS and javascript.
When working with virtual-browser testcases, testers will need some basic understanding of the HTTP protocol and HTML. For some systems, particularly more complex applications, they may also need to a deep understanding of how task-related data is passed between the browser and the server, regular expressions and strong problem solving skills.
Testcase development time
For simple sites and web applications, there may no difference in the effort required to develop tests. As the sites become more complex, real-browser testcases will tend to be easier for testers to build and customize as required.
Any testing tool that performs HTTP-level simulation of a browser has to deal with a particular problem: correlation of fields in the server requests with data returned in previous responses. If the application under test (AUT) uses atypical methods for exchanging this data, customization of the testcases will be required. Depending on the experience level of the tester, this can be challenging. Our support staff can walk you through the problems very quickly, but it will still slow the testing effort.
Flexibility
Real-browser testcases provide the ability to insert custom code into the testcase or the browser to simulate user actions that would be difficult to achieve any other way. As an example, a simple bit of javascript can be inserted to click every radio button in a particular column of a table that also has particular image in the next column. In this regard, real-browser testcases are very flexible.
Virtual-browser testcases provide very precise control over the network traffic generated by a load test. As a result, they can be much more flexible for certain classes of problems. For example, if there are certain 3rd-party servers required to render a web page but you do not want (or are not allowed) to test them, you can simply delete those requests from a virtual-browser testcase.
Maintenance Effort
Real-browser testcases model the user actions. When the web application changes the workflow or how the elements on the page are identified, real-browser testcases will need to be adjusted. When the application is well designed for testing and the tester chooses robust element locators, maintenance may be minimal.
Virtual-browser testcases model the network traffic. Changes to the web application that changes those network requests and responses may require changes to the testcase, even when those changes are not visible to the end user. Depending on the complexity of the system under test, these testcases may need to be re-recorded and all customizations re-applied.
Testing resources
The sending, receiving and processing of HTTP messages by our virtual browser users is exceptionally efficient. Running real browsers is, by comparison, resource intensive. If your site can be reached from our cloud engines (running in EC2), then access to massive amounts of load generating power is just a few clicks away – as a result either approach will work well for all but the largest tests. However, if the testing must occur in a closed environment, then resource usage should be considered. Feel free to contact us to help you determine how much computing capacity will be needed for your test.
Chris
If you’d like to discuss these differences, or have other advantages/disadvantages to add, please feel free to comment on this post or contact us. Please note that there are exceptions and edge-cases for everything I’ve said above.
1 Comment
27 October 2014 The Week That Was – Week Ending 4th Oct, 2014 | Practical Performance Analyst
[…] Real Browsers v/s Virtual Browsers : Which one to choose for your Performance Testing – Link […]