Software · Outage

A library update broke Salesforce deployments — so Salesforce rolled 179 instances back

On November 11, a supporting runtime-library update inside a fresh Salesforce release made Apex code fail to compile or deploy across 179 instances. Salesforce halted the rollout and spent two hours putting the previous version back — instance by instance.

THE BUYER'S PULSESoftware8 min read
A software engineer focused at dual monitors of code — the desk from which a broken deployment gets rolled back
A release is a promise the vendor makes to every customer at once. When this one broke, the fix was not a patch but a retreat, executed instance by instance. A software engineer at her screens. Photo: ThisIsEngineering / Pexels.

Photo: ThisIsEngineering / Pexels (Pexels License).

The most honest move in enterprise software is the rollback: the public admission that the new version is worse than the old one, and that the fastest way forward is back. On the evening of November 11, Salesforce performed that move at fleet scale. A release that had just landed on a set of instances broke the most basic act those instances exist for — compiling and deploying Apex code — and over the following two hours Salesforce stopped the rollout, reversed 179 instances to the previous version, and documented the whole retreat on its status page, update by update, until the last instance was back.

179 instancesThe final scope of the November 11 disruption, per Salesforce's own status record
18:23 UTCWhen the disruption began on instances that had just received a recent release
2 h 16 minFrom the first status entry to the resolved notice at 20:39 UTC, after the rollback completed
1 libraryAn update to a supporting runtime library — the change Salesforce identified as the probable trigger

The setting matters. Salesforce is not an application a company installs; it is a platform other companies build on, in Apex, the platform's own programming language. Its customers' developers deploy code to Salesforce instances the way they would to their own servers — except the servers, the runtime and the release calendar all belong to the vendor. When a vendor release breaks deployments, the customers cannot fix it, pin a version, or decline the upgrade. They can only watch the status page. On November 11, the status page was worth watching.

What broke, exactly

The incident record opens at 18:23 UTC with a “functional disruption” on instances that had recently received a new release. The symptom, in Salesforce's account, was precise: compiling or deploying Apex code failed intermittently. Intermittent is the operative word — the failure that passes once and fails the next time, that survives a retry and then doesn't, that convinces three different teams their own code is at fault before anyone suspects the platform. And the blast radius reached past developers: in some cases, Salesforce notes, login attempts and other processes that depend on the same underlying code failed as well. The disruption's final scope was 179 instances — the record names ARE2S, BRA2S, BRA6S and 176 more — each one a home for many customers' organisations.

The cause, identified as the evening progressed, was almost disappointingly small: not the release's headline features, but an update to a supporting runtime library inside it — the kind of component that ships as a line item in a changelog nobody reads. Big platforms rarely fall to big components. They fall to shared, invisible ones, the library every feature quietly stands on.

Why a broken Apex is not a broken feature

It is worth pausing on what Apex is, because it decides who feels an incident like this. Apex is not a plugin API bolted onto the side of Salesforce; it is the language in which customers write the logic of their own businesses — the validation that guards their data, the automation that routes their cases, the integrations that keep Salesforce in step with everything else they run. When compiling and deploying that code fails, what freezes is not a Salesforce feature. It is the customers' own change pipeline: their fixes, their releases, their Friday deployments, all queued behind a compiler they do not control, failing intermittently enough that each team first suspects its own code. An intermittent platform failure is a tax on every engineering organisation it touches, paid in debugging hours spent proving the fault is not yours. The record's note that logins and dependent processes failed in some cases widens the point further: the same runtime that executes customer code also carries parts of the platform's own behaviour, so a library inside it fails in both directions at once — the vendor's product and the customers' products, sharing one throat.

The decision: stop the rollout, reverse the fleet

By the 18:55 UTC update, Salesforce had a probable trigger and made the two decisions that define a well-run incident. First, it stopped the deployment of the release — no new instances would receive the broken code while the diagnosis continued. Second, it chose reversal over repair: rather than patch forward on 179 live instances, it would roll the affected fleet back to the previous version, prioritising the instances where impact was confirmed. In parallel, engineers began preparing the longer-term answer — a corrective release that removes the library change entirely, to be deployed only once it is ready. Patch forward for the future; roll back for the present. The order of those two operations is the whole craft.

The rollback then proceeded the way fleet operations actually proceed: unevenly. Salesforce's updates through the evening report instances recovering in waves, prioritised by confirmed impact, and — the detail that separates a real record from a press release — a few instances that needed a second attempt. By 19:45 UTC, recovery was confirmed on at least one prioritised instance and Salesforce was warning that some login flows could still fail while the work continued. By 20:13, the rollback was nearly complete. At 20:39 UTC, two hours and sixteen minutes after the first entry, the incident closed: every affected instance back on the previous version, Apex compiling and deploying again, and a promise on the record of a full investigation and preventive actions to follow.

What a rollback costs

A rollback at this scale is not a keystroke. An instance is a shared home: many customers' organisations live on each one, and the version has to move for all of them together, in an order chosen by confirmed impact — which means, by construction, that some customers' evenings were repaired before others'. Anything the new release touched while it was live has to remain valid under the old one; a reversal is a data problem wearing a deployment's clothes. And the story does not end at 20:39. The corrective release — the same release, minus the library change — still has to be deployed across the fleet a second time, so every affected customer will live this release twice: once as an incident, once as a redo. That is the honest price of the undo, and it is still cheaper than the alternative on the table that evening, which was to debug a runtime library forward, in production, on 179 instances at once.

The same day, elsewhere in the federation

The Apex disruption was not Salesforce's only incident that day — it was the largest of five. The company's status page for November 11 also records a MuleSoft Runtime Fabric incident at 02:40 UTC, in which applications temporarily could not send logs and metrics to Anypoint Monitoring while a change was itself being rolled back, and a Heroku Private Spaces incident in Tokyo at 18:16 UTC — seven minutes before the Apex entry opened — in which an upstream cloud provider prevented new dynos from starting while running ones carried on unaffected. Across the five incidents, independent trackers summarising the page count 105 components in scope. That is the structural fact about Salesforce in 2026: it is not one product but a federation — CRM, integration, a developer platform, acquired empires each with their own infrastructure — and its status page is the map of the federation's borders, drawn in incidents.

There is one defensive instrument the model does leave in customers' hands, and November 11 shows it working in negative space. A customer cannot refuse a Salesforce release, but it can refuse to deploy its own changes into the hours when the vendor's fleet is mid-release. The teams that met the broken compiler that evening were the ones whose own deployments landed inside the window; the organisations whose change freezes happened to overlap the rollout watched the same incident as spectators. Timing your changes against the vendor's calendar is a poor substitute for controlling the calendar — but on the evidence of this record, it is the difference between an incident you read about and an incident you debug.

For customers, the practical lesson is about a calendar they do not control. In the SaaS model, the vendor's release schedule is your change window: new code arrives on the vendor's timetable, in the vendor's order, and the only lever a customer holds is the quality of the vendor's undo. November 11 is what a good undo looks like — a halted rollout, a prioritised reversal, second attempts admitted in public, and a corrective release held back until it removes the offending change rather than layering a patch on top of it. The investigation Salesforce has promised will say how a runtime-library update reached production with this effect, and what now stands in its way. Until then, the record of the evening stands as the documentation: 179 instances went forward onto a broken release, and 179 instances came back.

𝐆𝐄𝐓 𝐅𝐑𝐄𝐄 𝐅𝐔𝐋𝐋 𝐌𝐀𝐆𝐀𝐙𝐈𝐍𝐄 𝐀𝐓 thebuyerspulse.com

Sources

  • Salesforce Status (Nov 11, 2026), the incident record “Service Disruption – Core Service – (179 Instances)” (status.salesforce.com): the 18:23 UTC start, the intermittent Apex compile and deployment failures on instances that had just received a release, the supporting runtime library identified as the probable trigger, the halted rollout, the prioritised rollback with second attempts where needed, and the resolution at 20:39 UTC with a full investigation promised.
  • UptimeRobot (Nov 11, 2026), daily summary built from Salesforce's own status page: the rollback ran across five updates, and Salesforce recorded five separate incidents that day — including MuleSoft Runtime Fabric at 02:40 UTC and Heroku Private Spaces in Tokyo at 18:16 UTC — with 105 components in scope across the five.
  • Photo: ThisIsEngineering / Pexels — a software engineer coding at dual monitors in a modern office (Pexels License).