Hurricane Sandy – InterServer’s Response and Recovery

When the reports that Hurricane Sandy was heading toward the New Jersey area we, like everyone, began our preparation process right away.  As the area continues to focus on recovery we wanted to share some information about how we prepared, responded and recovered from this devastating storm.   While we feel we were very successful in our response to Sandy, we’re constantly looking for ways to improve our services, and the information we’ve collected during the storm and the following recovery process are invaluable in that improvement process.

Preparation

InterServer has had a disaster recovery plan in place for years, and when it became clear that Sandy was going to be hitting the area hard we began putting our plans into action.  Our disaster recovery plan includes several levels of redundancy for both power and data transmissions as well as being our datacenters being housed in state of the art buildings which can withstand powerful storms.   In the days before Sandy made landfall we tested and confirmed the readiness of all of our emergency power systems, and scheduled additional staff to be onsite and on call to respond to any type of outage immediately.

During the Storm

When the storm made landfall it quickly became clear that this was going to be a historic event both in terms of the power of the storm and the damage it causes from the winds, flooding and fires.  During the storm and for several days after our team of electricians and technical engineers responded to numerous issues related to the storm.  Some of the key events which took place include the following:

• Access to Commercial Power – Along with millions of other people in the area, we quickly lost access to commercial power due to the storm.  Our diesel generators immediately kicked on and provided us with uninterrupted power throughout the storm.  There were several instances when commercial power was restored and lost during the storm and recovery.
• Data Circuit Interruptions – As expected, there were many different data communication interruptions throughout the storm.  Some of them were brief, others were sustained outages.  Due to our comprehensive redundancy plan using multiple telephone companies routed through a variety of geographic paths, we were able to reroute data through alternative circuits minimizing any impact on our customers.
• General Access Issues – Due to road closures and other problems many of our staff had trouble traveling to and from our datacenters.  Fortunately we had planned ahead with additional staffing prior to the storm and confirmed all of our employees knew alternate routes to get to the datacenter if they were able.  We also had staff on call to report to work, or work remotely, if other individuals were unable to make it in.
• Notifications – We felt it was important to keep our customers apprised of our status during the storm.  As information became available to us we posted it via our twitter feed and on our website so clients knew the status of their data in real time.

Recovery

When the storm was over we, like millions of others, began the process of recovery.  Recovering from a disaster like this is extremely complex for large datacenters due to the fact that the power, data and other services are provided by a wide range of different vendors.  Our disaster recovery team worked closely with all of our service providers to ensure that no additional problems were caused by the recovery efforts.   We made sure to plan all recovery efforts to maintain a stable transition back to normal operations.

As part of our recovery we will go through all the data collected during this storm so we can learn what worked well and identify any improvement opportunities for future outages.  We’d like to extend our thanks to all of the InterServer employees, as well as the employees of all of our vendors who worked tirelessly through this event to ensure it went as smoothly as possible.