Escalation and Recovery

Module 6 of 9 in the chapter Reliability, Recovery, and Observability, chapter 14 of the Integration Architect course. Each module stands on its own. Previous: The Return Receipt · Next: What Salesforce Shows You

The question this module answers: When an integration fails and the retries don’t fix it, who finds out, what do they do, and how does the integration get back to where it should be?

Who Is Notified?

When I list the questions for an arrow, there’s one at the end that people skip: who is notified? The log has the details, the retries ran out, the failure is sitting in the to-do list. And nobody knows.

Retries handle the failures that fix themselves. Everything else needs a person. This module is how that person finds out, what they do next, and how the data gets back to where it should be. Don’t wonder if errors are going to happen. Assume they will, and decide this before they do.

The Saturday Lesson

Chapter 13 told the story of a Saturday lockout (chapter 13 module 4). Two integrations shared one integration user; a new one got the password wrong and locked out both. The fix there was about identity: one integration, one identity.

But read it again for this module, because it has two more lessons. It was a Saturday, team A wasn’t working, and we had to somehow find someone to turn their integration off. And we couldn’t turn it off ourselves.

So every integration needs two things written down before go-live:

• Who to call, on both sides, at any hour the integration runs. Not a team name. A way to reach a person.
• A way to stop it that you control. Most integration platforms can pause a flow. On the Salesforce side, with one identity per integration, you can freeze that one integration user; Salesforce’s guide says freezing is how “to prevent users from logging in to your organization.” Pair it with revoking the integration’s tokens (chapter 13 module 7). And module 4’s circuit breaker can be opened by hand.

With one identity per integration, stopping A doesn’t touch B. That’s the same design paying off twice.

Alert on Patterns, Not Single Errors

Now, who gets told, and when? The tempting answer is “email the admin on every error.” I’ve seen where that goes: a hundred emails a day, a mail rule that files them away, and the one that mattered sitting unread.

Salesforce’s Reliability guide gives the rule: “Alert when error rates exceed thresholds or sustained degradation occurs, not on isolated failures. Single errors are normal in distributed systems. Patterns of errors indicate reliability problems that require attention.” And its Operational Excellence guide names both ways it goes wrong: “Poor alerting either misses incidents (too few alerts, too-high thresholds) or creates alert fatigue (too many alerts, too-low thresholds) where operators learn to ignore notifications.”

So an alert has a severity, and the severity decides who hears it and how fast. Salesforce’s guide: “Implement alert severity levels that match operational escalation procedures.” A critical alert reaches the person on call, at any hour; a warning “generate[s] tickets for business-hours investigation.” And the severity comes from the business, not the error code: “Prioritize incidents by user impact rather than technical severity.”

Here’s what that looks like for Steve Trucking’s status arrow to Redrock, as a labelled example:

Click to enlarge

One arrow’s escalation ladder.

Look at step 3. Redrock’s team is told about their own webhook being down. That happens more than you’d think. Salesforce’s guide says it: “Integration monitoring often reveals external system problems before their operators detect them, enabling proactive escalation.” Your monitoring is often the first to know the partner is down. A good escalation plan includes calling them.

And the timing should follow the business. A status that’s an hour behind at noon, with trucks on the road, is critical. The same hour at 3 a.m., with no deliveries until morning, can wait for a ticket.

The Run That Never Happened

Some failures send no error at all.

In chapter 4, Steve Trucking’s quarterly audit tool signed in with a refresh token. It ran in January and worked. By April, the token had sat unused past Salesforce’s 30-day limit, and the run failed at sign-in with invalid_grant (chapter 4 module 6). At least that sends an error. Worse is the job that never starts: a schedule that was deleted, a server that was retired, a flow someone deactivated. Nothing fails, so nothing alerts.

So for every scheduled integration, alert on silence. Each run records that it ran, and how many records it moved. Something checks, after the run should have finished: did it check in? If not, that’s an alert, the same as an error. Salesforce’s Operational Excellence guide puts it in one line: “Proactive logging and alerting transforms silent failures into diagnosable incidents.”

It works on a quarterly job, and it works on a nightly file, a daily batch, and a webhook that usually sends a hundred statuses an hour and has sent none since noon. “None” is a number too. Watch for it.

And for the quarterly tool itself, chapter 13’s better answer still holds: use a flow with no refresh token to go quiet (chapter 13 module 7), and treat invalid_grant as a signal to authorize again, not something to retry.

An Owner and a Runbook

Every integration needs an owner: one person or team who answers for it, and whose name is on the alert. Not “IT.” A name.

And every integration needs a runbook: one page that tells whoever is holding the alert what to do. Salesforce’s Operational Excellence guide says it about deployments, and it’s just as true of integrations: a runbook takes what one person knows and turns it into “routine procedures anyone can execute.” The person on call at 2 a.m. may never have seen this integration before.

What goes on the page:

• What it does, in a sentence: which systems, which direction, how often, what it carries.
• Who owns it, on both sides, and how to reach them at any hour it runs.
• How to tell it’s broken: which alerts, which dashboard, what normal looks like (how many per hour, how long each takes).
• How to stop it: the pause in the integration layer, the integration user to freeze, the tokens to revoke, the breaker to open.
• How to bring it back: which recovery below, and how to check it worked.
• Who to tell in the business, and what to say.

It lives in the inventory, beside the arrow’s failure plan from module 1 (chapter 5 module 6). And it’s tested: once, before go-live, someone who didn’t write it follows it.

Recovery in Three Sizes

When the cause is fixed, the data still has to catch up. How depends on how big the gap is, the same question chapter 1 asked: a minute, an hour, a day, a weekend (chapter 1 module 6).

Replay one. A single failure, or a handful: a bad record, fixed, and sent again. This is what module 2’s ledger and to-do list are for. The to-do list says what failed; the ledger holds the exact payload; the retry sends it again. Salesforce’s playbook suggests a screen for it: a Lightning web component that shows the open failures, fetches each payload on demand, “with a Retry Now button.”

Catch up. An outage of minutes or hours: the partner was down, the breaker opened, the statuses waited. When the partner is back, the queue drains, oldest first, with the time of each change so the newest wins (module 4). For an event subscriber that was down, it reconnects and asks for everything after the last replay ID it saw, as long as that’s within the event bus’s 72 hours (chapter 11 module 4). Then module 5’s sweep picks up anything without a receipt.

Resync. A gap too big, or too uncertain, to replay message by message: past the 72 hours, or nobody’s sure what got through. Then you stop replaying and compare. Take what changed since the last time you know was good, and send it again as a set, the way chapter 10 keeps a copy in sync (chapter 10 module 8). Because every arrow is safe to send twice, sending a little too much is fine. Sending too little isn’t.

Click to enlarge

The size of the gap picks the recovery.

Write down, for each arrow, which of the three you’ll use and where the “last known-good time” comes from. In the middle of an outage is the wrong time to work that out.

Test knowledge. Know escalation for failed integrations (MNT-2): alert on patterns and thresholds, not single errors; severity by user impact; an owner and a runbook per integration. Know the recovery options: replay a failed payload from your own log, catch up from a stored replay ID within the 72-hour retention, or resync changes since the last known-good point when the gap is too large.

After It’s Over

One last step, and it’s the one most teams skip once the integration is green again. Ask what happened, and what would have caught it sooner.

Salesforce’s guide calls it a blameless review: “Blameless postmortems focus on system improvements rather than individual fault.” That word blameless matters. In the Saturday story, the team that typed the wrong password wasn’t the problem. A shared identity was. If the review had stopped at “they made a mistake,” the next team would have shared a user too.

Then change something: a threshold, a runbook line, an identity, a test. The integration should be harder to break the same way twice.

Beyond the exam: what AI changes. AI tools are good at the first half hour of an incident: reading the log, grouping the errors, spotting that every failure is the same partner and started at 11:42. Let them. But they can’t call Redrock’s on-call person, and they don’t know which hour of the day the business can wait. The runbook and the owner are still yours. And when you paste errors into an AI tool, paste the masked log from module 3, not the payloads.

What an Architect Decides Here

• An owner for every integration, and who to call, on both sides, at any hour it runs.
• How each integration is stopped by your own team: a pause, a frozen user, revoked tokens, an open breaker.
• The alerts: thresholds on patterns, severities by who is affected, and who each one reaches.
• The silence checks: every scheduled integration reports that it ran.
• The runbook, one page per integration, tested by someone who didn’t write it.
• The recovery for each arrow: replay one, catch up or resync, and where the last known-good time comes from.
• The review afterwards, blameless, ending in a change.

Explore

• Salesforce Architects, Well-Architected: Operational Excellence (opens in a new tab): alert severities, alert fatigue, runbooks and blameless reviews (checked 2026-11-07).
• Salesforce Architects, Well-Architected: Reliability (opens in a new tab): alert on patterns, not isolated failures (checked 2026-11-07).
• Salesforce Help, Freeze or Unfreeze User Accounts (opens in a new tab) (checked 2026-11-07).
• One Integration, One Identity: chapter 13, the Saturday lockout.
• Keeping a Copy in Sync: chapter 10, the resync.