Load Testing with Virtual vs Real Browsers
Perhaps the most contentious issue in load testing today is whether to test with real or virtual browsers. Read through the issues and then leave a comment to let us know how you feel.
Which is More Real?
Load Tester PRO’s virtual browser recreates the HTTP data stream exactly as a real browser would, so from the server’s point of view it is identical to a real browser. For example, it automatically configures session tracking, cookies, and application state, just to name a few. The server is not aware that the HTML, images, and javascript it’s sending out are not actually rendered, and thus can not tell a virtual from a real browser.
Real browsers, on the other hand, incorporate some or all of the code from a shipping browser, such as Firefox, Chrome, or Internet Explorer. During a test, the real browser will parse and render every page, and for the most part is the same application that the user has.
Testcase Development
The biggest perceived advantage of real browsers is that, because they are real, they can execute AJAX, Flash, and other code executed inside the browser, and thus increase the liklihood that complicated pages will execute correctly without any added configuration. While AJAX support is solid with real browsers, it actually can not handle embedded technologies such as Flex and Silverlight because the GUI is not accessible from the DOM, there is no way to direct user and keyboard clicks into the right part of the browser.
Virtual browsers handle AJAX, Flash, applets, etc, by parsing and configuring the communication stream from the executed code to the server. Load Tester PRO’s virtual browser can be configured to handle all of the AJAX cases, but only some of Flash and applets. If those technologies communicate with the web server using standard protocols, then its possible to parse and configure the data stream. If they chose to use proprietary non-HTTP connections then parsing those communications becomes impossible.
In the embedded browser technology race Active-X, Applets, and Flash are dead or dying, but Silverlight and Flex are gaining ground. The convention wisdom is that embedded technologies will eventually be replaced by HTML5, but if you need to test Silverlight or Flex today, or think you might need to in the future, then a virtual browser specially designed for those technologies are your only bet.
In the case that complicated HTML5 or AJAX implementations such as GMT become predominate, real browsers have the edge. The more complicated the javascript, such as with GMT, the easier the test case configuration will be with real browsers. A skilled tester can always handle configuring complicated javascript with virtual browsers, but real browsers in theory would make this task considerably easier for the novice tester.
Test cases with complicated workflows, for example, where the names of fields and other page elements change dynamically are equally hard on both real and virtual browsers.
Scalability
The biggest advantage of a virtual browser is that it was designed to maximize the number of browsers running on a single computer, while real browsers are optimized to maximize performance for one browser at a time. The virtual browser in Load Tester PRO™ was designed from the ground up for efficiency, even re-writing low-level I/O code at the byte level. The result is a single computer can typically simulate 2,000 concurrent and accurate virtual users. The reason is that the I/O layer is extremely efficient, and freed of the need to completely parse and render the HTML, the computer has more power available to simulate extra browsers.
Because real browsers are optimized for optimal performance when run one at a time on a single computer, running more than one at a time on a computer affects rendering speed. Even the biggest system can’t run that many browsers without having a negative effect on the accuracy of the page load speed. Before the cloud, testing with real browsers was impossible; now, its theoretically possible to create hundreds of virtual computers in the cloud, each with one or more browsers.
Even with the cloud, though, real browser scalability is still an issue. The total number of users that can be simulated with real browsers is limited by the sheer volume of computers needed to generate any substantial load. For example, if a maximum of 5 real browsers could be made to run on a cloud computer, generating 1,000 users from real browsers would take 200 cloud computers, something that is possible, but complicated and pricey. The theoretical maximum load that can be generated from real browsers isn’t much higher than that.
By comparison, the same 200 cloud computers could simulate 400,000 users with virtual browsers.
Performance Statistics
With virtual browsers the statistics from each and every connection from the browser to the server is measured and collected. Real browsers do not have access to this low level information, but on the other hand, theoretically would have access to rendering times. Unfortunately, rendering times for the most part are only useful for measuring performance when not under load, and are not a substitute for a full detailed analysis of the transfer times for a page’s content.
Error Detecion
One area where Load Tester PRO’s virtual browser has an advantage is error detection. During the process of correlating fields and variables it creates a map of what the web pages look like, and automatically configures error detection. With a real browser the user must configure validation by hand on a page-by-page basis.
1 Comment
2 January 2013 testingwhiz
Load Testing with complicated workflows are hard on both real and virtual browsers.The load testing with virtual browser is designed to increase the efficiency from the ground, while testing with real browsers are still possible only theoretically.