What’s Old is New: Pre-Staging Can Make All the Difference

When you’re in the middle of a business interruption, I think it’s safe to say time is your biggest enemy. Every minute vital systems are down it costs you money and lost productivity, and potentially puts your reputation at risk in the marketplace.  Also, depending on the type of incident, minutes or seconds can make a life-or-death difference. Therefore, in the interest of saving time responding to an incident, I will always recommend something that’s been a good practice for decades: Pre-Staging. This is the sixth blog in my “What’s Old Is New” blog series.

Pre-Staging

So, what do I mean by Pre-Staging? Pre-Staging is the process of getting critical resources and vital information together ahead of time and having it ready to go in case there is an incident. Taking the proactive step of having those resources and information available exactly when you need them can save a tremendous amount of time during a recovery, and as a result help you recover more effectively.

For example: when I worked for a financial services company, our Business Continuity Management software package had a multi-server set up, with one server in the group dedicated to processing print jobs. If you needed a physical copy of the plan, you’d schedule a job and the print server would take over. You would get either our PDF file or printed copy when it was done processing. In our particular implementation, the print server had developed a reputation for being a little slow and cantankerous, so those of us on the technology recovery side learned not to trust it over time. We eventually started creating monthly copies of all the pertinent site recovery plans, saving them to encrypted flash drives. Those of us on the recovery teams carried those flash drives around, and therefore always had up-to-date copies of all the recovery documentation.

Documentation

One chilly December morning, we lost power to a building where 450 people worked. Somebody on another team decided it would be a good idea to print every single plan related to every single team that worked in that building. This effectively locked everyone out of being able to print any plans as the server rendered the 500-odd pages of planning material into a single print job. Fortunately, those of us that needed to engineer the recovery effort had copies of plans on our flash drives and were able to execute the recovery. If I remember correctly, everyone was relocated and back to work at a different facility about 90 minutes before that print job even finished. Pre-Staging critical recovery information saved our company several hours of downtime for those 450 people.

Contact Information

Documentation isn’t the only thing that is a good idea to have pre-staged to speed recovery — contact information is also critical.  Most companies see the benefit of having contact information on hand in case there is some sort of incident, but it’s a good idea to take that information a step or two further.  If there’s a specific vendor that you know you need to execute a recovery of some system, having their Contact information handy will save you the time of having to look for it during an incident. Frankly, it’s a good idea to have that sort of contact information available for all of your critical system and technology vendors, because you never know what sort of issue might come up during system recovery.

Messaging Templates

If you have a mass communication system available, I would suggest taking Pre-Staging one step further, and Pre-Staging messaging templates. Having messaging ready for several potential business interruptions (weather, traffic, logistical, problems, etc.) will let you get information out to your employees and business partners faster during an incident. I have also worked at multiple employers that had emergency communications to employees pre-written just in case there was a need for them to respond to an evacuation request, bomb threat, or active shooter incident. Being able to send out well-crafted communications quickly when every second counts can potentially save lives.

Technology

It’s also a good idea to have technology pre-staged for recovery wherever possible. Several of the backup tools available in the marketplace these days will allow you to back up systems and applications in a format that is much more readily usable in a production environment. Rather than backing up individual files or pieces of databases, many tools now allow you to back up entire virtual machines and allow near instantaneous recovery of them, when properly configured. This sort of Pre-Staging greatly reduces the recovery times of applications, making it easier not only to recover during an incident, but also to test the quality of your backup and recovery processes.

“Doing Your Homework”

There’s also some Pre-Staging the businesses could do that I would almost classify as “doing your homework.” For example, one financial services company I worked for had a recovery plan for one office as renting out a ballroom at a nearby hotel to use as a recovery workspace for employees. They had actually “done their homework” on this plan however, and had a contract in place to give them priority over other people trying to rent that space if they needed it. Similarly, when I worked for an auto manufacturer, there were hazardous materials disposal companies that were contracted to come in on short notice if there were some sort of spill or fire at a facility; those contracts were in place, making response times much faster and recovery go more smoothly.

Cross-Training Employees

Another type of preparation businesses can do that could be classified as Pre-Staging is cross training employees. Making sure the knowledge needed to perform critical tasks is not exclusive to one group in the business allows a company to be more flexible in their recovery efforts.  (This can also help with employee satisfaction, teamwork, and retention, but that’s definitely a topic for another time.)

Pre-Staging Cost Consideration

Admittedly, Pre-Staging resources can have some costs. I’m not suggesting that companies need to pre-stage everything in their environments (although that would make recovery quite a lot easier), but I am suggesting that they at least entertain the possibility of doing it. For most of the things I’m talking about it’s not difficult to compute the cost and time and resources – simply calculate what it would take to pre-stage something and compare that with the potential cost of having to do that during an incident. If it’s something that costs very little to do ahead of time, or it’s something that can save the company a tremendous amount of time or money during an incident, then I would definitely consider doing it. Many types of Pre-Staging cost almost nothing to implement (like pre-writing emergency notification emails) and I would recommend them to almost everybody.

There is also sometimes a recurring cost to having things pre-staged, and that’s a detail that shouldn’t be ignored. In that incident with the cantankerous print server I described earlier, the reason why our encrypted flash drives full of recovery documentation were so valuable is because we spent the time every month making sure that they were kept up-to-date. Spending those couple of hours a month and keeping up-to-date documentation readily available absolutely paid dividends when the time came that we actually needed the documentation. If we hadn’t been willing to commit to that monthly effort and had outdated plans on those flash drives, that would have probably hurt us more in a recovery than it helped us. Likewise, Pre-Staging technology resources to have them ready for recovery requires diligence as systems grow and change over time. I would recommend that Pre-Staging configurations be updated at least annually when you go through and update the data in your business impact analysis, if not more often. Not sure which parts of your resiliency program would best benefit from actively Pre-Staging? Not sure where to begin? Sayers is here to help. Our team has decades of experience helping companies like yours be more responsive in times of crisis.