Business Impacts of Performance
One of the opening sessions of the Velocity 2009 conference was a joint session with engineers from Microsoft Live Search (Bing) and Google Search…yes, they were together on the same stage :>
Both organizations wanted to measure the effects of performance on the business – i.e. on revenue. Revenue for search sites is based on the number of searches and on the number of clicks on results. Both of the search giants have experimentation platforms which allow them to send a small portion of real users to a different version of the search platform, where they can modify the user experience in a controlled way and measure the differences in user metrics (number of clicks, abandonment rate, user satisfaction, etc). Here are some of the results from their testing:
- When the search results pages are 2 seconds slower, the revenue per user dropped by >4%!
- Pages that were 100-400ms slower resulted in statistically relevant increases in site abandonment
- As users continued to receive slower search results over a 6 week period, the satisfaction rate continued to decline
Microsoft performed experiments using chunked transfer encoding, which allows the beginning of a page to be delivered quickly (e.g. page headers), while the server was still working on the search results. They found that implementing this resulted in:
- 9% increase in time-to-click – indicating that users were quicker to respond to the results
- 2.2% increase in use of query refinement features
- 0.7% increase in total clicks
- 0.7% increase in use satisfaction
0.7% may not sound like a lot, but the Microsoft engineer noted that many features they have added to the site resulted in lesser improvements in user satisfaction.
The conclusions that Google and Microsoft drew from these experiments is that (1) speed does matter and (2) users do notice even small changes in site performance.
Chris