When a winter storm knocked network equipment offline, the operations team had to sort through a surge of trouble tickets. The AI tool built to summarize those tickets stopped producing summaries just as engineers needed them most.
Mundakkapatta Dileep Kainary, an application architect at IBM who has spent more than two decades building enterprise systems, described the failure and the redesign in a DZone article on an LLM pipeline that survives traffic spikes. His takeaway: Counting requests is not enough if each request can suddenly become much more demanding.
The stakes can be high. IT service management vendor Virima cites an estimate of about $2 million an hour in median downtime costs for high-impact outages at large enterprises. For teams using AI to help respond to incidents, the question is whether the tool can keep working when the systems it monitors are under the most strain.
Demand and failure can arrive together
In Mundakkapatta’s account, the winter storm in early 2026 took network equipment offline and triggered a flood of alarms and trouble tickets. There were more tickets than usual, and each one could contain much longer logs. The number of requests rose, but the amount of text the model had to process rose faster.
The system hit its capacity limit. Automatic retries added to the load, and summaries stopped reaching the operations team.
“The system was busiest during the single event it existed to handle,” Mundakkapatta said. “Demand and failure turned out to be the same moment. Size an AI system for the average day and you have sized it for the calm right before the emergency.”
Building for the spike
A dashboard that counts requests can miss the difference between a short ticket and one with pages of logs. For a language model, both the amount of text it reads and the amount it generates can affect capacity and cost. Retrying a failed request can consume more of both.
Mundakkapatta’s team redesigned the system to sort incoming tickets by severity before sending them to a larger model. High-priority incidents got access to more expensive capacity. Routine tickets could go to a lower-cost model or get a basic summary without one. The system also checked the estimated cost of each job and staggered retries so they would not pile up at once.
“The trick was to stop treating every ticket as equal the moment capacity runs short,” Mundakkapatta said. “A high-severity incident earns the expensive model. A routine one gets a cheap summary, or a plain extract with no model at all. You decide in advance who gets the scarce resource, so the system never has to make that call in a panic.”
What a pilot can miss
Industry reporting on abandoned AI initiatives has highlighted the difficulty of taking projects from a pilot into production. In Mundakkapatta’s case, cleaning up and reconciling incoming data took more work than modeling. An inaccurate summary could also be harder to spot than an outright failure, particularly if it sounded convincing.
“A demo runs once, under conditions you control,” Mundakkapatta said. “Production runs every day, on data you do not. Most of the AI that dies between those two points dies on the data and the load, not the algorithm. That gap is where the real engineering lives.”
From summarizing incidents to taking action
Grand View Research projects the market for AI platforms used in IT operations could reach about $36 billion by 2030. For engineering teams, moving from summaries to automated actions changes the consequences of a mistake. An engineer can review a proposed response. A system that acts on its own needs limits on what it can do and a fallback if the model becomes unavailable.
Mundakkapatta describes that progression in stages: first helping engineers sort incidents, then drafting responses and, eventually, taking limited actions within rules set by people. In his view, how the system handles an outage matters more to that decision than how well it performs in a demo.
“What decides whether a system can act on its own is how it behaves on its worst day,” Mundakkapatta said. “Cleverness on a good one barely enters into it. Get the failure modes right and most of the automation question answers itself. That is the problem worth the next stretch of my career.”
©2026 Cox Media Group
