With Halloween over, an even scarier event is on the horizon for IT staff: the online doorbusters that come with the winter holiday season. The term “doorbuster” applies (sometimes literally) to brick-and-mortar stores during the holiday season. It also translates well to the information technology systems that get stressed heavily on big online sales days such as Black Friday and Black Monday.

Preparing for this year: Battle School
Time is short, and most likely your software and IT systems are as well prepared as they’re going to be for the coming storm. So what can you do in the remaining time to better prepare for this year’s doorbusters?

In IT terms, doorbusting could be any manner of system failures that may occur; for example, as system transaction levels reach new peaks. Many failures are recoverable, but only if your team is ready and reacts well in high-stress situations. (Because unrecoverable failures are by definition beyond repair, I won’t address them in this article.)

(Related: Getting all hands on deck with agile)

A good place to focus effort with just weeks to go is training and human preparedness. The Battle School style of training is a good way to increase your team’s preparedness quickly.

What is the Battle School style of training?

It is scenario-based. Architects and other IT staff dream up a variety of potential failure scenarios for your environment. For each scenario, turn it into a discussion exercise by documenting:

  • What indications operators would see when it breaks
  • What things an operator look should for to gather data about what’s really broken
  • How the operator should gather more information: Should he or she consult another person, or is there a quick way to get answers from a computer? What logs can be checked, and where are they?
  • What corrective actions should be contemplated or executed? Who should do them, and how?

It is pull training. Pull training seeks to pull information from the brains of sharp folks rather than getting them to produce it. Typically software and IT organizations have staff at mixed levels of confidence and skill in your systems. Pull training helps spread the wealth of knowledge from the hot shots to others.

The scenarios you plan put your hot shots in a situation where they respond to “What if?” scenarios, which turns out to be a great way to extract knowledge from technical workers—especially hard-won critical-thinking skills and technical details that might matter during a system outage or other calamity. It’s much easier than getting them to write everything they know on a wiki or series of e-mails.

It’s a controlled simulation experience. When you run a Battle School training event, the participants don’t know much about the scenarios at the start. The event consists of the set of scenarios you have dreamed up and a few hours of training content. Start each exercise by discussing only what the operator indications would be for the given problem. That is, describe what participants would see, not what actually happened (per the scenario). This style helps the team gain the critical skills and learn the technical details to help determine what broke in your environment, and then learn how to fix it. Limit your scenarios to those that are recoverable.

It has collateral benefit. The work to develop and talk through scenarios leads to training events that prepare your team to handle such challenges as system failures or outages. There’s also a big collateral benefit; while coming up with the scenarios and preparing the discussion exercises, your team will uncover weaknesses in your systems. In some cases, hot shots might opt to find ways to prevent problems, unless prohibited by moratoriums on system changes as the online doorbusters approach.

Hands on?
A Battle School training event can be put together in short order with discussion-only exercises. There’s a lot of value in that step alone. Putting together a more realistic, hands-on simulation is another matter entirely. That takes more extensive preparation but may well be worth it. Consider whether a hands-on event makes sense in your environment.

Modern cloud infrastructures are changing and improving constantly. But end-user usage patterns are finding new ways to stress the systems.

The holiday season often stresses IT systems up to and beyond their capacity. Realistic stress-testing is both expensive and elusive, especially for modern cloud-based hardware topologies with shared-everything infrastructure. But it is essential for businesses that rely on the holiday season and its higher spending levels to help their cash flow.

Cloud infrastructure has come a long way in capacity, performance and reliability. But generally cloud solutions tend to be more cost-optimized than performance-optimized. Figure out what aspects of your technology stack, at least those that you have control over, are going to need operational care when sales orders or other transactions come piling in. Many organizations these days have systems that are “half” in the cloud. If that’s you, consider whether that transitional state may actually translate into service recovery options if cloud infrastructures have challenges this year.

Preparing for next year: Emergency bug-fixing strategy
What about longer-term plans for next year and beyond?

First, if you’re not already on the agile and DevOps bandwagon, get on it. Agile development promotes releasing code quickly and incrementally, aiming for a larger number of smaller, lower-impact and lower-complexity releases. Many organizations impose moratoriums on system and software changes as they near critical times, such as Black Friday, when loads are extreme. This tactic helps reduce the guesswork for IT operational staff in figuring things out when they break.

But such moratoriums typically have exceptions for a true emergency. You might encounter a bug that only uncovers itself in the systems load of the holiday season. Your superhero coders have a better chance at delivering that emergency fix if they have agile systems in place. The trick is to have a clear path for the emergency fix, clear rules for when it’s acceptable to change anything, and clear knowledge of who can authorize such changes (and people who know when to break the rules).