One of the opening sessions of the Velocity 2009 conference was a joint session with engineers from Microsoft Live Search (Bing) and Google Search…yes, they were together on the same stage :>
Both organizations wanted to measure the effects of performance on the business – i.e. on revenue. Revenue for search sites is based on the number of searches and on the number of clicks on results. Both of the search giants have experimentation platforms which allow them to send a small portion of real users to a different version of the search platform, where they can modify the user experience in a controlled way and measure the differences in user metrics (number of clicks, abandonment rate, user satisfaction, etc). Here are some of the results from their testing:
Microsoft performed experiments using chunked transfer encoding, which allows the beginning of a page to be delivered quickly (e.g. page headers), while the server was still working on the search results. They found that implementing this resulted in:
0.7% may not sound like a lot, but the Microsoft engineer noted that many features they have added to the site resulted in lesser improvements in user satisfaction.
The conclusions that Google and Microsoft drew from these experiments is that (1) speed does matter and (2) users do notice even small changes in site performance.
Chris