Menu

Beta Browser Shootout

Safari 4 Beta, IE 8 Beta, and Chrome Beta 1 are compared to Firefox 3.0.

Matt Drew
©2008 Web Performance Inc, March, 2009; v1.0

Read and post comments

Introduction

The purpose of this test was to compare the page loading performance of the newest versions of three major browsers: Internet Explorer 8 (IE8), Chrome 1, and Safari 4. Firefox 3.0 was included as a baseline. When we conducted a similar real-world browser performance test in 2007 we looked at Safari 3 beta, Firefox 2, and Internet Explorer 7. In that test Safari 3 beta showed a significant advantage in page loading speed. Do Apple’s claims for Safari 4 uphold this tradition? We’ll see.

Which Web Pages?

Which web pages to test is the first question to answer. We chose fifteen of the top sixteen rated english-language sites from Alexa.com (as of February 3, 2009). This approach gave us a good cross-section of sites ranging from very light, small sites to sites loaded with javascript, images, and dynamic content.

We also wanted to test representative content for each site, as well as the sites’ front pages. For example, on search sites, we tested a sample search page using the search term “test”. For Facebook, we used its own Facebook page. We also used a test Ebay auction and sample articles from various sites. We had trouble getting consistent data from Blogger, so we used MySpace (#16) instead. The following is a list of the URLs we tested from March 3 through March 5, 2009:

  1. http://www.google.com/search?hl=en&q=test
  2. http://search.yahoo.com/search?p=test
  3. http://www.youtube.com/watch?v=iBBcibQByNs
  4. http://www.facebook.com/facebook
  5. http://search.live.com/results.aspx?q=test&go=&form=QBLH&qs=n
  6. http://articles.moneycentral.msn.com/taxes/home.aspx
  7. http://cgi.ebay.com/Test-Item-Please-do-not-bid-0749-2-19 05_W0QQitemZ6156056805QQcmdZViewItemQQptZLH_DefaultDomain_0?hash=item6156056805
  8. http://en.wikipedia.org/wiki/Tim_Duncan
  9. http://raleigh.craigslist.org
  10. http://www.aol.com
  11. http://www.amazon.com/Spore-Mac/dp/B000FKBCX4
  12. http://sports.espn.go.com/mlb/news/story?id=3886444
  13. http://www.cnn.com/2009/POLITICS/02/04/voter.anger/index.html
  14. http:///www.microsoft.com/security/default.mspx
  15. http://celebrity.myspace.com/index.cfm?fuseaction=celebrity.risingstar

Testing Considerations

Static vs. Real-World Testing

Static benchmark tests are useful tools for checking web browser performance. They measure browser performance in a highly controlled way with few variables. Unfortunately, when you or I sit down to browse the news or check out the latest Facebook postings, we will not have a fixed set of carefully structured data files. Real world testing gives a much better approximation of the day-to-day user’s experience. Site quirks and flaws are more likely to show up under these conditions.

Network Effects

Bandwidth and latency are key challenges in testing a browser’s performance directly. In our 2007 test, we established that test results were consistent between local network copies of sites and the public sites. Consequently, for this test, we did not repeat the local portion of the test and instead focused on improving the quality of the live data.

Advertising

Advertising is part of the web browsing experience, and we wanted to include it in our test. However, since ads varied widely in size, speed, and latency, we had to block traffic to most advertising sites to achieve consistent results. We emphatically make no claims about the superiority of this approach over any other - it has clear limitations and should be taken as only one of many data points in the overall browser performance picture.

Caching Effects

We knew that there would be a difference between loading a page for the first time and loading it again when the data had already been downloaded, so we wanted to test both. First we would load each page into the browser to trigger any server-side caching and ensure that the server was performing as fast as it reasonably could. We would then purge the cache using the tools available in each browser. Once the cache was purged, we would load the page again, and measure the total time it took from the beginning of the first HTTP request until the last element was downloaded to the browser. The cached page loads were done a few seconds after the uncached full page loads using the refresh button in each browser. Each browser was tested several times across different times and different days to help eliminate network effects. The scores shown in the graphs below represent the mean of the best data points. In some cases, we were unable to get five data points for uncached page loads in the correct sequence, and portions of the tests had to be completed at later times.

Background Processes

All of the browsers have background processes which could (and did) interfere with the test results. These are activities such as updating rss feeds, checking for updated phishing information, or pre-loading search results. Since these processes are part of the browsing experience, we chose to tolerate their effects on the testing process.

Test Setup

The operating system used was Windows XP Professional SP 3, patched to current as of February 3rd, 2009. Browsers tested were Firefox 3.0.6, Internet Explorer 8.0.6001.18372, Chrome 1.0.154.48 and Safari 4 public beta (528.16). All browser configurations were the default as installed on Windows XP, with the exception of the proxy configuration to point the browsers at our recording proxy. All tests were performed remotely via the Microsoft terminal services client.

The test platform was a Dell PowerEdge SC1420, with a single hyper-threaded 2.8Gz Xeon and 4GB of RAM (PAE enabled).

Measurement of web page performance was done using our own Web Performance Load Tester™ version 3.5.6673, which analyzes HTTP traffic. Our software uses a Java HTTP proxy server to record all HTTP traffic between the client browser and the site. The reason this measurement technique is reasonably accurate is that the browsers try to pre-render the pages as much as possible while simultaneously pipelining HTTP image requests on multiple threads. By the time the last byte of the last element arrives, the web page is entirely rendered except for that last element, for which the render time is negligible. The software groups HTTP traffic that happens after the initial load separately, so in general we avoid including later AJAX requests in the initial page load timings. One thing we had to watch out for were pages (such as the AOL main page) that included timers that automatically loaded new content.

Testing Issues

Ideally, we would like to show test results that were unbiased by unanticipated errors in the operation of the browsers. However, the real world is rarely ideal, and we did find a few bugs:

Test Data

The three new browsers had uncached load time averages of approximately two seconds; Firefox 3 was slightly slower at just under three seconds. Cached load times showed slightly more variation but were still within half a second of each other.

Beta Browser Web Page Load Time Comparison

Conclusions

All three of the newer browsers were so close that there was no clear winner. Internet Explorer 8 had a small advantage, but not one that would exceed the margin of error for the test.

All three of the browsers tested were significantly faster at retrieving uncached data than the baseline Firefox browser. They were roughly equivalent in retrieving cached data.

Safari did not maintain its advantage, despite Apple’s claims that Safari 4 loads pages almost three times faster than Firefox 3. Their claims are more than likely valid in static benchmark tests, but in real-world applications, Safari 4 has less than one second advantage over Firefox 3.0.6 on uncached live page loads, and none at all over browsers of its own generation.

Version History

v1.0 - 1st public release (March 18, 2009)


Resources

Copyright © 2019 Web Performance, Inc.

A Durham web design company

×

(1) 919-845-7601 9AM-5PM EST

Just complete this form and we will get back to you as soon as possible with a quote. Please note: Technical support questions should be posted to our online support system.

About You
What you Need