Load Testing Services Case Study: Mammoth Mountain Ski Area
The Client – Mammoth Mountain is a year-round recreation area located in Mammoth Lakes, California. They offer six months or more of snow activities including downhill skiing, snowboarding, cross country skiing, snowcat tours, snowmobiling, and dogsledding. They have seven terrain parks, and received #1 terrain park awards in 2008 from Ski and Onboard magazines. In March 2009, OnTheSnow.com listed them as the best terrain park in the nation. In the summer there is mountain biking, rock climbing, hiking, zip lines, geocaching, golfing, and scenic gondola rides.
The Mammoth Value Pass (MVP) is the lowest-priced unrestricted season pass that the resort offers. For several years the passes were only open to renewing members. This year, the 10th anniversary of the pass, new purchasers will also be able to acquire passes. The pass offers several benefits aside from just skiing – passholders get discounted lodging, discounted passes at other areas, and the ability to put cash on their card for use at any Mammoth Mountain facility.
The Client’s Need – The MVP10 pass will be sold in limited numbers (38,000) and will only be sold during the month of April, which means that Mammoth Mountain’s web site will need to handle a high volume of traffic during this time period. In addition to those wishing to purchase a pass, all of the people who would just like to get information about the site will be visiting as well. Delays in page loading or errors in processing pass purchases, even with the elevated load on the site, would be unacceptable. As Brahm Goodis, Web Developer for Mammoth Mountain, put it, “it was crucial to make sure our site could handle the potential load.” This is where Web Performance, Inc. comes in.
Web Performance’s Service – Web Performance, Inc.’s (WPI) load testing services are invaluable to a client such as Mammoth Mountain. Prior to the mad rush of pass purchasers and renewers, WPI and Mammoth Mountain were able to work together to make sure the web site could handle the high volumes of traffic that were anticipated when the passes went on sale. Two of the biggest benefits that Mammoth Mountain received from hiring WPI to assist them in their load testing are expertise in using the load testing tool and large-scale Internet-based load testing infrastructure.
For example, load testing of the Mammoth Mountain site required a particularly sophisticated set of field splitters in order to manipulate the JSON data they were passing back and forth between the browser and the server. WPI sent Mammoth Mountain the test cases and field splitters that they used, saving Mammoth Mountain a lot of time and enabling them to start actual testing sooner.
Most businesses, other than those in the load testing world, do not have, nor does it make sense for them to have, the capacity to do large-scale load testing at a moment’s notice. According to Matt Drew, Performance Engineer with WPI, they can “put 1 Gbps of traffic onto any site in a few minutes, and ten times that (if necessary) on very short notice.” WPI has load engines scattered around the U.S. with various hosting sites, so they can hit the test site from multiple locations on high-bandwidth connections, closely simulating actual loading conditions.
The Details – WPI developed two test cases to test the Mammoth Mountain site: one for renewals, and one for new purchases. Both test cases stepped through the renewal or purchase process and ended with submitting test credit card information and “completing” the sale. Once the test cases were complete, WPI configured the load test to run many different test cases simultaneously, drawing on their dataset of user accounts and previous passholders. Each load test case was customized by grabbing a set of data from the dataset to be a different user either buying or renewing a different ski pass. WPI used the beta version of their Web Performance Load Tester™ 3.6 software to do the load testing.
A few errors turned up in the first phase of the load testing, particularly near the end of the testing when the load was the highest. One error code that was received was the “500 Server Error” code, which is a general response that indicates that the server was unable, for whatever reason, to generate the page. The original version of the site also did not meet the client’s criteria for timely performance of 10 seconds per page load.
Fortunately, Mammoth Mountain’s team was prepared and was already working on an updated version of the site that enabled them to make a number of performance enhancements. They put that version of the site into production, and WPI tested again against the new version of the site. The updated version performed much better, easily meeting Mammoth Mountain’s performance goals.
The Successful Implementation – WPI didn’t fix Mammoth Mountain’s problems – what they did was much more valuable. WPI provided load testing services that helped Mammoth Mountain pinpoint where their problems were, and provided them with a much greater assurance than they would have otherwise had that their website could actually handle the anticipated load without letting them down. In fact, Mr. Goodis said, “WPI exceeded our expectations and we look forward to working with them again.” That type of peace of mind can make all the difference in a company’s ability to provide a service.