One of the opening sessions of the Velocity 2009 conference was a joint session with engineers from Microsoft Live Search (Bing) and Google Search…yes, they were together on the same stage :>
Both organizations wanted to measure the effects of performance on the business – i.e. on revenue. Revenue for search sites is based on the number of searches and on the number of clicks on results. Both of the search giants have experimentation platforms which allow them to send a small portion of real users to a different version of the search platform, where they can modify the user experience in a controlled way and measure the differences in user metrics (number of clicks, abandonment rate, user satisfaction, etc). Here are some of the results from their testing:
Microsoft performed experiments using chunked transfer encoding, which allows the beginning of a page to be delivered quickly (e.g. page headers), while the server was still working on the search results. They found that implementing this resulted in:
0.7% may not sound like a lot, but the Microsoft engineer noted that many features they have added to the site resulted in lesser improvements in user satisfaction.
The conclusions that Google and Microsoft drew from these experiments is that (1) speed does matter and (2) users do notice even small changes in site performance.
Chris
When his dad brought home a Commodore PET computer, Chris was drawn into computers. 7 years later, after finishing his degree in Computer and Electrical Engineering at Purdue University, he found himself writing software for industrial control systems. His first foray into testing software resulted in an innovative control system for testing lubricants in automotive engines. The Internet grabbed his attention and he became one of the first Sun Certified Java Developers. His focus then locked on performance testing of websites. As Chief Engineer for Web Performance since 2001, Chris now spends his time turning real-world testing challenges into new features for the Load Tester product.