Microsoft 365 outage: a dormant bug hit 80% of gateways
Key takeaways
- A bug shipped in July sat undetected for more than three months because Microsoft's test hardware did not match the newer servers where it surfaced
- Automated failover made the outage worse by moving traffic onto healthy servers that then hit the same defect
- Microsoft could not simply roll back the code, so recovery meant patching groups of servers offline and returning capacity gradually
- IT teams should plan for a provider outage lasting twelve hours or more, not just short interruptions
The Microsoft 365 outage on 12 November was not a fresh failure. It was a software defect that had been running in production since July, and it took roughly 80% of Microsoft's gateway fleet down once the right conditions arrived. Microsoft's preliminary post-incident report says Outlook, Exchange Online, Teams, SharePoint and other services were disrupted for more than twelve hours.
The detail matters because it shows how a change that passed testing and ran quietly for more than three months can still break servers right across its network in under two hours.
The headline numbers
| Figure | What it refers to |
|---|---|
| July | When the defective update began rolling out to Microsoft 365 networking gateway servers |
| More than 3 months | How long the bug sat dormant before the outage |
| About 15 hours | How long before the outage Teams traffic was moved onto the Exchange Online gateways |
| Around 14% | Share of server capacity out of service because of an unrelated issue |
| Under 2 hours | Time for the failure to spread across the network |
| 80% | Share of the gateway fleet affected at the worst point |
| More than 12 hours | Duration of the disruption, per Petri |
The Independent describes the disruption as lasting around a full day, so the exact duration depends on how recovery is measured for each service. Microsoft's final report should settle that.
What the data actually shows
Three ingredients had to coincide. The first was the latent bug itself: servers became overwhelmed and unresponsive under high load, and the problem was more likely to appear on newer, higher-core-count hardware. The second was extra load, because Teams traffic had been redirected to the same gateways that serve Exchange Online. The third was reduced headroom, with around 14% of capacity offline for an unrelated issue that Microsoft was returning to service through routine processes.
None of the three would have caused an outage alone. The Independent reports that each issue on its own would normally have been fine. That is the pattern worth noticing: the failure lived in the interaction between a code defect, a traffic change and a capacity dip, each of which was managed by a different process.
Failover then did the real damage. Automated redirection exists to keep services available by moving traffic from unhealthy servers to healthy ones. Here, the healthy servers carried the same defect, so every redirect pushed more load onto machines that were likely to fail next. Fewer servers took more traffic, and the cascade fed itself. Microsoft's own mechanism accelerated the spread.
Recovery had its own trap. Servers brought back online were immediately hit by accumulated traffic and went down again. Microsoft acknowledged in its preliminary report that its recovery systems were not designed for a failure of this scale. Because the flawed code was months old, a rollback was not possible either. Engineers built a targeted fix, took groups of servers offline, applied it, restarted them and returned capacity to production gradually.
Not every service suffered equally. The Independent reports that some apps could switch away from the broken network and were restored sooner, while Outlook could not. Outlook could not switch away in this way.
What the data does not tell us
This is a preliminary report, and Microsoft says a final post-incident report will add technical detail and describe steps to strengthen testing and prevent similar cascading failures. Several gaps remain.
It does not quantify business impact. The Independent notes that hospitals, police forces and schools were affected, but gives no figures.
Microsoft's explanation of validation environments relying on older hardware accounts for the missed bug. It does not explain how change management allowed those conditions to overlap.
How this compares
The source gives no direct benchmarks against other providers' incidents, so a precise league table is not possible. What can be said is that the failure mode is a familiar one in distributed systems: a defect that only appears at scale, on specific hardware, under specific load, is the kind that test environments built from older machines will miss. Microsoft's own admission that its validation relied largely on older hardware is the key detail.
The pattern there, with restoration attempts undone by returning load, matches the recovery problem Microsoft now describes in its own report.
So what?
The practical lessons land on the customer side, in the UK and the US alike. Microsoft 365 underpins email and collaboration for organisations on both sides of the Atlantic, and the Independent's account of hospitals, police forces and schools being hit shows the exposure is not limited to office work.
For IT teams, a few concrete actions follow:
- Plan for a working day, not an hour. A twelve-hour-plus outage with a slow, staged recovery is a realistic scenario. Check that the disaster recovery and business continuity plan names an alternative channel for staff and for incident communications that does not depend on Microsoft 365.
- Separate your tooling from the affected platform. If the incident response contact list, runbooks and escalation chat live inside Teams or Exchange Online, they are unreachable at the moment they are needed.
- Understand what is and is not covered. Provider-side outages are an availability problem, which is different from data loss. The guide to Microsoft 365 and Google Workspace backup coverage explains where the boundary sits.
- Watch the final report. Microsoft has promised more technical detail and remedial steps on testing. Those details will show whether the hardware-diversity gap in validation has been closed.
A bug that testing missed and that ran for over three months caused an outage only when load rose and capacity fell at the same time. Automated failover, the mechanism meant to protect availability, spread the fault to roughly 80% of the gateway fleet in under two hours. Recovery was slow because rollback was impossible and restored servers were swamped again.
The single most important takeaway: a cloud provider's resilience features can amplify a failure rather than contain it, so customers need a continuity plan that assumes Microsoft 365 can be unavailable for a full working day.
Frequently asked questions
What caused the Microsoft 365 outage on 12 November?
A software defect from July in Microsoft 365 networking gateway servers, triggered when Teams traffic moved to Exchange Online gateways while capacity was already reduced. Automated failover then spread the fault.
Why could Microsoft not simply roll back the fix?
The flawed code had been deployed months earlier, so a rollback was not possible. Engineers built a targeted fix and applied it to groups of servers taken offline.