
When failover spreads the outage
On 12 November 2026, Microsoft 365 users around the world lost access to their Exchange Online mailboxes for close to a day. Microsoft's preliminary report, as summarised by Cybersecurity News, does not describe an attack or a spike in traffic. It describes a latent defect, a traffic move, and a gateway fleet that was already short of capacity. Failover then moved the load onto the servers that were still healthy, and those servers stopped responding too. I think that last step is the one most startup platforms should study, because the same mechanism exists in a three-node cluster.
What the preliminary report says
Microsoft hasn't published its final review yet, and Cybersecurity News reports that repair details are due on 20 November. So I'm working from second-hand summaries of the preliminary report, and the details can still change. According to those summaries, a fix for the Microsoft 365 networking gateway servers, rolled out from July, carried a latent lock-contention defect in an internal Windows process. Under certain traffic conditions, the defect exhausted CPU and memory. Microsoft's tests ran on weaker hardware, and the defect was worse on newer machines with more cores.
Two conditions set it off, according to WindowsForum's summary of the report. About 15 hours before the outage, Teams traffic moved onto the gateway that Exchange Online also uses. Nearly 14% of the gateway capacity was already out of service because of an unrelated issue. At the worst point, more than 80% of the gateway servers stopped responding, and the containment boundaries also failed because of a separate failure.
Why failover made it worse
Failover assumes that the healthy servers have room for the load of the failed ones. Take a simple case of 10 servers that each run at 60% of their limit. If 2 of them drop out, the other 8 each have to run at 75%. When the defect grows with load, as this one did under certain traffic conditions, the extra load makes the next server fail sooner. Each failure then puts more load on fewer servers, and the fleet falls over in steps.
Health checks can speed this up. A server that slows down under load fails its health check, the load balancer removes it, and its traffic moves to the rest. Status updates quoted by Control Plane described Exchange Online servers that failed automated health checks and dropped out, which overloaded the servers that remained. The same article reports that Microsoft switched to an alternative health-check method during the recovery.
The variables I check on a client platform
No startup runs at Microsoft's scale, but the same variables decide the outcome on a small platform. These are the ones I check first:
- Headroom after the loss of the largest single failure domain, such as one availability zone, measured at peak traffic and not at the daily average.
- Capacity that is already out of service. A fleet with 14% down for maintenance or repair has less headroom than the average on the dashboard suggests.
- Shared paths. Two products behind one gateway, one ingress controller or one NAT gateway share one failure domain.
- Traffic moves. A move of traffic onto a shared component is a change, and it needs the same review and rollback plan as a deploy.
- Load shedding. A service that rejects some requests early can stay up for the rest. A service that accepts everything can fail for everyone.
- Health checks under load. A check that fails on slow responses can remove servers at the worst moment.
- Test hardware. A load test on smaller instances can miss a defect that only shows up with more cores, as Microsoft's tests did.
What I would do this week
The first step I recommend is a short capacity table. It lists each tier, its failure domains, and its peak load as a percentage of what the remaining nodes can carry after the largest single loss. If any row goes above about 80% after that loss, I treat it as a finding. The second step is to read the health-check and autoscaling settings together, because they decide how fast traffic moves when a node slows down. The third step is to treat planned traffic moves, such as a new product behind a shared ingress, as changes with a rollback plan.
Microsoft runs at a scale few teams will ever see, but the rule that decided this outage holds at any size. Failover protects a platform only when the remaining capacity can carry the load. A defect that grows with load turns failover into a way to spread the failure. Microsoft's final review should add detail, and I'll read it when it lands. Until then, I'm curious how other teams measure this: how much capacity can your platform lose at peak before failover starts to make things worse?