Thoughts on “The Velveteen Rabbit: Can AI Create Viable Resiliency Plans?”

AI is definitely THE hot topic this year when I go to conferences, and it’s the latest buzz word in every product feature demonstration that I encounter lately. (If attendance at my conference presentations is any indicator, AI is about 5 times more interesting than anything else I’ve presented about.)  And, as one of my coworkers joked about the new versions of products coming out these days: “Our product is basically the same as last year’s, but now WE USE AI!“

Writer’s Note: I’ve done some more extensive research on this topic, and created a full-blown presentation to deliver at conferences about it. This post will cover some of the same points that I cover in that 25 minute presentation, but certainly not all of them. I’m also covering some topics “around the periphery” of what I cover in that presentation, dealing with AI systems in general and what you might encounter yourself.

The buzz is well-founded, though, and AI applications have the potential to change the way just about everyone works. Large language model AI can be a revolutionary time saver for anybody tasked with analyzing or creating large volumes of text. AI has also changed how we create and consume art, and the way we digest data. It does a wonderful job of gathering and summarizing vast volumes of data for concise analysis. That powerful analysis capability is why we see it in so many products coming out these days in the IT world: IT departments, Information Security departments, Compliance departments, and just about every other department are presented with tremendous amounts of data to analyze and act upon, and AI assistance has the potential to make that analysis faster, easier, and more accurate.

“Artificial intelligence and generative AI may be the most important technology of any lifetime.” 

Marc Benioff, chair, CEO, and co-founder, Salesforce

There’s also generative AI which manages to construct grammatically-correct sentences in a way that seems like magic.  You ask it to write something for you, and based on the context of your request and the training of the AI engine, it instantly composes text to meet your request. There are limitations to what it can do, but it’s still a really marvelous technology. In the right situations, Generative AI has the potential to be a tremendous labor saving device.

Generative AI does have some shortcomings, however.  In my presentation, I generally characterize Generative AI as a precocious 5 year old with a huge vocabulary. It’s eager to please, it uses a few too many big words, it doesn’t *quite* understand how everything works, it gets bored and loses focus if you ask it too many questions in a row, and it just randomly makes stuff up sometimes, like saying the Mona Lisa was painted in the 1800’s. Because it uses so many big words and speaks so confidently, it’s easy to get fooled into thinking your AI’s smarter than it really is.  AI-generated text also has a style to it that careful readers (and software tools) can detect, so be careful trying to pass off AI-generated paragraphs as your own work.  AI hallucinations are a real thing too. It’s estimated that 2-3% of “Facts” that you get out ChatGPT and the like are completely made up, and that problem isn’t going away in the foreseeable future.  It’s best to make your requests carefully so you can avoid hallucinations, independently check all facts, and maybe even ask your AI to generate your text with a particular audience in mind so it writes more conversationally.

We also need to talk a little bit about prompting, or the process of making requests of your AI engine. To make things easy on myself, I asked ChatGPT for an example of how to write a good prompt, and I thought the response was very good (here’s a screenshot):

We can see from this that questions need to be very specific if you want to get a specific type of output from the AI engine.

(As a sidenote, the prompting process is just as important when you’re trying to create some sort of visual art using an AI engine. I spent many hours trying to figure out the best way to take the Bing AI art generator and “paint it into a corner with specific requests” so I can get consistent results on the art I create for presentations. Even with carefully constructed prompting, I would say only about one out of every 30 images I create is useful.) 

With all of this in mind, I figured I would have a few potential problems trying to get a generative AI to create a viable response plan:

• Does ChatGPT even know what’s supposed to go into a Response Plan?
• Response plans aren’t any good without supporting data about the business. Does ChatGPT know this, and how does ChatGPT handle adding that data in?
• Can ChatGPT stay focused long enough to write us a plan?
• Is the plan any good?

Does ChatGPT have an idea of what goes into a response plan? To that question, I will respond to definite “yes.“ Before creating a plan, I first asked ChatGPT to come up with a list of topics of what it would include in a response plan, and there was a high correlation between what it listed and what you would find in best practices.  I then used ChatGPT’s own outline to create the plan.

ChatGPT also seemed to have a pretty good idea about the need for supporting data about the business and how to include it.  For the first few sections of ChatGPT’s plan outline, I asked it what information it would need to create that section of a plan, and then I provided it with the data it was requesting (using a fictitious business I often use for demonstration purposes).  ChatGPT took that information and incorporated it into the sections talking about the roles of people on response teams, and it also created an impressively comprehensive list of the kinds of information to be included in the document’s appendix.

On this Third Point was where I started to encounter problems. ChatGPT started giving out less detailed information after the 4th or 5th section of the document (and there were 12 separate sections of the document), meaning that the responses it gave were more open-ended and generalized. On the places where it had bullet points, it stopped having as many bullet points to back up each subject area. I shared the plan with several experienced business resiliency professionals, and one of them commented that the later sections of the plan were generic enough they could have gone into any number of different types of plans (business continuity, disaster recovery, incident response, or even communications plans).

Which brings me to the 4th point — “Is the plan any good?” Well, as I’ve joked to a few people: If you can hear the footsteps of your auditor coming down the hall and you need to hand them a plan when they get to your desk, you could do worse than trying to generate a plan out of ChatGPT.  The plan covered several important areas of business resiliency (at least on a superficial level), and it certainly generated a lot of content. The plan I generated was about 30 pages long once I pasted it all into Microsoft Word. It also, as I mentioned before, had an impressive set of things to include in its appendices, regarding information about various parts of the business. On the other hand, the plan was verbose and redundant; I think there were only about 12 pages of ‘useful’ content in that 30-page plan. Plus, there were all the problems I mentioned above – it was open-ended and generalized.

“There is no substitute for hard work.”

Thomas Alva Edison

Overall, this was an interesting exercise, but I don’t think ChatGPT is going to be putting me out of a job anytime soon.  A plan generated this way is not going to be nearly as good as one that’s been manually created by a business resiliency professional. I wouldn’t stake my business on it, and it’s probably not even a good enough plan to make your insurance company happy. On the other hand, I think this is a great starting point for someone with zero experience in creating a resiliency plan, or someone that wants to put together a quick rough draft and kickstart the process of creating a proper plan.So, having ChatGPT create your Business Resiliency documentation might not be the best idea, and creating those documents from scratch isn’t easy either. Sayers is here to help. Sometimes you just need some expertise from outside your organization to help bring it all together, and our Business Resiliency team has decades of experience in all aspects of building and maturing programs just like yours.  We’ve built our business around meeting the needs of our customers, and we’re here to help you make the most of your team’s time and effort.

Interested in more Sayers blogs? Subscribe below!