Case study: Lenox Finds a Baseline in the Cloud
For Lenox, makers of fine china and crystal, e-commerce has been an important aspect of the business for over ten years. Over time, as online traffic increased, the company has invested in their e-commerce environment to keep up. They are currently running Cold Fusion, with IIS on several Windows web servers, a Cisco content switch, and an Oracle back-end.
Paul Lynch, who is responsible for testing, had recently upgraded the Oracle servers, but was looking for an additional performance boost. He wanted to know the limits of the site’s capacity, and identify performance bottlenecks. Testing the site started off hypothetical. Lynch wanted to establish a baseline of capacity and determine what would happen as traffic spiked. The site had never undergone intensive performance testing, so Lynch was eager to gain insight into where improvements could be made.
The major restriction at Lenox was time – all testing had to be done quickly so as not to take up resources during the busy holiday season. When Lynch started this process in October he had just a few weeks to complete the testing. He was looking for a testing tool that would allow him to get up and running quickly, but offer the flexibility he needed to run complex tests. “We found Load Tester to be a very sophisticated tool, and we were able to set up sophisticated but flexible testing scenarios. The learning curve for a product with this level of flexibily is a bit of a challenge, especially when you’re working on a tight deadline.” Load Tester was Lynch’s fastest answer to the question “How many users can your web site handle?”.
Lynch had a few issues while getting set up for testing, mainly related to migrating Load Tester to another Windows server, and to Amazon cloud setup. He worked with the Web Performance support team to resolve these issues, then was up and running. “Our site got confused by the cloud traffic and interepreted it as a denial of service attack. We figured this out when some of the test cases got stuck in the firewall. Once we added rules to the firewall to allow the cloud traffic, the problem was solved.” One feature Lynch especially liked is the plug-in to connect to Windows system information, which made it much easier to correlate what was happening on the servers with the actual load.
Lynch initially didn’t seek to test via Amazon cloud, but got the idea while evaluting Load Tester 4. In the end, testing with the cloud was cheaper since he didn’t have to aquire the hardware to set up a test lab, and the results more accurately reflected user activity. “We didn’t have a lot of spare hardware, and we also wanted the testing to be representative of real users. Our users are mostly people shopping online from home, so using our fast internal network just wouldn’t be as realistic. The cloud made sense on both points.”
From using Load Tester, Lynch learned that his performance bottlenecks are really in the application layer. When he ran tests of straight HTML vs. ColdFusion, he found huge differences – 1800 vs. 40 – in how many users the site could handle. He was pleased, however, that his Oracle database was having no trouble keeping up. He has been trying to work with Adobe to improve Cold Fusion performance, but so far has had limited success. An easier route to performance gains will be to upgrade the webserver equipment.
This round of testing has convinced Lynch that Windows is more challenging and costly for Lenox than a Unix platform would be, and he is now making plans to migrate for better performance in the future. “It’s clear to me now that ultimately we have to migrate our Windows/IIS platform to Unix/Apache and replace Cold Fusion. I’m starting to make the business case for that transition now.” With a Unix-based platform in place, Lynch expects to be able to handle an increased load as e-commerce grows.
Load Tester turned out to be a good introduction to cloud computing for Lenox. Before this testing project, Lynch had been thinking about how cloud computing might benefit Lenox. Web load testing was the first step, but he has other ideas about how to harness the flexibility and power of the cloud. Lenox sees most of its e-commerce traffic in the Christmas buying season, and Lynch is considering whether it would be sensible to use pre-defined server images in the cloud to augment peak time capacity. He is also considering it for backups.
Because of the complexity of their test cases and the sophistication of Load Tester, in conjunction with their limited timeframe, Lynch believes they would get more out of the product if they hired Web Performance for professional services, rather that purchasing the software. “In retrospect, I think we could have used help in determining the most accurate test cases. We came up with several that were the best representation we could find, based on server metrics. But the load wasn’t quite what we expected. Our practical experience told us that the load should look a little different, so I’m working on making the business case for having the engineers at Web Performance come in and build our test cases for us next time.”
When Lynch needed to test the Lenox website user capacity on short notice, Load Tester was the right tool for the job. It offered the features and flexibility to build complex, realistic test cases and identify performance bottlenecks. The cloud testing engine was an easy alternative to collecting hardware and building a test lab, and better reflected real users’ network conditions. Load Tester helped Lenox find a baseline in the cloud, and chart a course for performance improvements.