Disaster Recovery

Search Google for a list of WTC tenants on September 11, 2001.  There were hundreds of companies in those towers, many of which did not survive the disaster.  How many of them had disaster recovery plans?  Do you have a disaster recovery plan? 

Each of us has a responsibility to prepare for disaster.  Since there is no such thing as life insurance for businesses, we have to make a plan for ourselves.  Luckily there are many things we can do to insure that our businesses survive catastrophes both minor and major.  The first step is to identify the events that we will most likely have to overcome. 

As I see it there are three general types of disasters.

            Acts of God such as Fire, Flood and Tornado. 

            Malicious acts like Arson, Sabotage, like computer viruses and theft .  Digital vandalism is a growing problem that we must address when thinking about disaster recovery.  Instead of beating your mailbox flat with a ball bat, lots of kids today write a worm.  Same stuff different time. 

      And most commonly……Accidents           

            These include hardware failures like crashed hard drives and computer failures, accidentally deleting files from hard drives, or losing an important file.  Nobody loves these, and any of us can have them happen at any time.   

The results from these failures are the same.  They mean business interruption and loss of revenue.  Disasters also subject us to other risks if our offices or homes are destroyed.  We can be put at huge risk from litigation.  After a fire or a tornado the documents we need to defend ourselves in court may be lost.  For example we may not have the contracts and information we need to defend ourselves in court cases.   

Missing or destroyed insurance policies may be overlooked or uncollectible in the wake of a disaster.  Quick think of all the policies you have and the insurance company who rights them.  Even small businesses can have dozens of policies, unless you have copies of them, you can overlook them very easily. 

Collecting debts can be daunting after an information loss.  In the absence of accounting records you won’t know who owes you, how much they owe you and you may not be able to produce invoices or purchase orders to justify your charges.  It can destroy your cash flows.  I have enough trouble getting some customers to pay and we haven’t even had a disaster in my offices. 

Borrowing money in the wake of a disaster can be very difficult.  Without vital information such as balance sheets, A/R Reports, titles, deeds, contracts, and insurance policies banks can be very reluctant to lend.   

The presence of all these documents and a well thought out disaster recovery plan can be very reassuring to potential investors and lenders.  The fact that you have planned for disaster recovery shows to the lenders an increased likelihood of your company surviving disaster.  It will let them know you are not just scrambling. 

Smaller disasters such as hard drive failures or computer virus attacks can cause huge losses.  While no physical damage is done, employee downtime can cost 1,000’s of dollars per hour in lost productivity as everyone sits on their hands waiting for IT to repair their computers.   

Clearly, disasters and data loses can have significant effects, even the minor events can cause problems.  In the recent ice storm, a customer of mine was unable to safely reach their offices.  They had a conference call with key employees and decided to not open their offices.  It wasn’t safe to drive.  Their next logical step was to call all of their customers.  Coincidentally, no one had a customer contact list offsite.  They couldn't call their customers from home.  They couldn’t even email them.  So they had to drive to the office, or at least as far as they could, and walk the rest of the way to the office, just so they could call their customers. 

Not having a customer contact list available almost cost them a collision insurance deductible! 

Business interruptions can affect our revenues, expose us to risk and destroy our customer service.  In my 10+ years of helping customers deal with these issues, and in many cases recover from fires, floods, water main breaks, explosions, leaky roofs, broken fire sprinklers, dumb employees and other events, I have identified steps you can take now that will virtually insure your ability to emerge from disasters intact. 

82%
Companies agreeing that improving recovery objectives without increasing bandwidth is important

45%
Companies capable of recovering from a primary data center site failure within five hours

8%
Companies that can measure time to recovery and data loss in minutes

63%
Companies that say bandwidth prevents them from extending replication or remote backup beyond mission-critical apps

Source: "The Impact of the WAN on Disaster Recovery Capabilities," Forrester Consulting for F5 Networks, January 2007

I want to give you some ideas about how we can plan for these disasters and increase our likelihood of surviving them.  The first thing on the list will be the thing you’ll be most glad of in a disaster; the second will be the second, and so on.  The last idea on our last will definitely not be the last thing you’ll be thinking of though! 

Your first consideration should be to identify all of your vital information.  Make a list of information that is crucial to the ongoing survival and operation of your business.  These documents can include customer and vendor contracts, leases, customer contacts, employee contact info including home and cell phone numbers, accounts receivable journals, and warrantee information if you are a manufacturer.  Of course insurance policies are pivotal in a disaster.   

You must also install redundant hard drives on all of your computers.  These RAID drives protect you against equipment failures.  With redundant drives you will have no more data loses due to crashed hard drives.  According to a Carnegie Mellon University computer science department study, as many as 1/3 of all hard drives fail in their first five years of use.  That makes hard drive failure your largest and most likely disaster recovery concern.  It also makes disks the most unreliable information archiving system in common use.  

Additionally, in order to protect your digital data from both hard drive and natural disasters, it is crucial that you backup your computers daily.  You must back them up to removable media such as tape, optical media like DVD’s or portable hard drives.   

In addition to backing up your data, compile a library or all your software.  Often the software your company uses costs more than all of the computer hardware in your office.  It can also be time consuming to replace.   

After you have compiled a list all of your vital documents, backups and software, make copies of them and protect them.  Protect your paper in lockable water resistant document cases.  More importantly, protect your backups and software in shock resistant cases, such as a turtle case.  They are lockable, and hold your media securely in foam padding.  Protecting storage media from shocks is crucial to insuring that the backups will be reliable when you need them.  

Once you have all of these copies protected, get them out of your office.  It does no good to have all of these things stored in your offices if your building is blown away or burns to the ground.  GET THEM OUR OF THERE.  Preferably get them to a full-service records management facility like Data Storage.  We have state of the art security and are staffed around the clock to help you in case of a disaster.  But the important thing is to get them out of your offices.  Do not send them home with an employee.  No one wants to be named in the next 10,000 social security numbers stolen on portable hard drive.  Local news loves these stories and is hoping you’ll be the next one.  Take your disaster recovery kit to a safety deposit box or records management facility. 

Now that you have your vital information offsite, you must construct a disaster recovery plan.  A good plan should include a calling tree that dictates who calls whom and in which order in the event of a disaster.  Make a list of parties to notify.  Your insurance agents, employees, customers, BMS Catastrophe.   

Your plan might even include a list of 3 to 5 potential temporary office solutions.  This could include mobile mini offices, RV’s, temporary office leases, etc.  You will need a base of operations while struggling through your disaster.  Brainstorm for other contingencies that could arise that are particular to your business.  Here is a disaster recovery check list that I have created that will help you get an idea about what a good recovery kit might contain.  Your plan may not include everything here, but it may be helpful. 

Once you are satisfied with your plan, make copies for every employee.  You should hold training sessions like this one and take suggestions for improving your plan.  Then run a drill.  Make sure your computer backups are recoverable.  Run through your calling tree.  That will confirm that your contact info is accurate.  It will also let you know how long it takes to complete notifying everyone.  Your customers will be glad to hear you are running a test too. 

Lastly, call your insurance companies.  Have them verify your coverage in a written summary.  Chances are you will turn up an oversight.  Very few of us know every detail of our coverages.  Is your software covered?  Do you have valuable document coverage?  Do you have business interruption insurance?   

I estimate that about four hours of planning and $600.00 will allow you to create your own business life insurance plan that will help your business survive what life throws at us.  Remember that replacement equipment and inventory can be bought, but you cannot buy your one of a kind business information!  More than likely your disaster recovery plan will save your digital pictures from a crashed hard drive.  Or maybe just keep all of your sales quotes intact.  But, when you leave here today you’ll be ready for what the weather and Murphy throw at us.