Site outage report for 27-28 October, 2011
As many of our customers have noticed, the Web Performance website was unavailable for most of the day on Thursday, October 27th. Much of the site was functional mid-day on Friday, but the support system remained inaccessible throughout the day.
Over the weekend all functionality has been restored and we believe that no data has been lost. However, if you have not received a response to any inquiries via e-mail, our contact form, or our support system, please contact us again so that we may both answer your inquiry and investigate the data loss.
Earlier in the week we experienced a hard drive failure on our main server. Because it was part of a redundant drive array, the site maintained full functionality while we planned the drive replacement. In order to minimize the downtime, we performed backups of only the critical parts of the system prior to the drive replacement. The drive was replaced late Wednesday night and the site was back up and running within a few hours. Unfortunately, during the array rebuild process, the primary drive also failed, resulting in a complete system failure early Thursday morning. Since we had not performed a complete backup, we had to rebuild the sever configuration from an older backup before we could restore the system data. Due to this and a series of mis-communications about the restoration procedure, it took more than 24 hours to restore basic website and mail services.
We recognize that our customers rely heavily on our website and support system during the load testing process and that our outage may have caused delays in your efforts. Occasional service disruption is inevitable for any system, but the duration of this outage was unacceptable and has revealed serious flaws in our backup/restoration procedures. The responsibility for these failures rests on me – I and my team will be analyzing the failure in more detail in the coming weeks and developing a plan to ensure this does not happen again in the future.
My humblest apologies,
Chris Merrill, Chief Engineer