Reliability, Recovery, and Observability

Chapter 14 of Integration Architect, Part 2: Designing Integrations That Work (Design). Previous: Trust, Identity, and Integration Security · Next: Integration Orchestration.

Errors will happen. Not because anyone built it badly, but because an integration is two systems and a network, and any of the three can stop at any moment. Salesforce’s own Reliability guide says it in five words: “Failures happen in distributed systems.”

So this chapter isn’t about preventing failure. It’s about designing for it: what each arrow does when it fails, how you find out, how it gets back to where it should be, and how you’d know it’s healthy the rest of the time. Part 2 has given you the mechanisms, the doors in, the callouts out, bulk, events and identities. Now we ask what happens when each of them stops.

Steve Trucking runs through the chapter: the status that couldn’t reach Redrock one afternoon, Redrock’s manifests that timed out on their way in, the ledger that would pass a million records in about 38 days, and a morning report the person on call reads in a minute.

The ideas in this chapter

Every arrow, out and in. For every outbound arrow: where do I log, what do I tell the users, how do I retry, and is there ever a retry? For every inbound arrow: what if they get an error, how will they retry, and how will they recover? Handled top to bottom, in every layer.
Capture everything, and keep it past the failure. One log entry per call, with one ID carried through every layer. Write it outside the failing transaction, or it rolls back with the failure it records. At volume, a big object ledger holds every payload, and a custom object holds the failures people work on.
The log is data. Kept in tiers, purged at the rate it arrives, with a do-not-delete flag. Masked when it’s written, field by field, with a keyed hash where you need to match without exposing.
One layer retries. Retries in every layer multiply. Timeouts are sized to who is waiting, a circuit breaker stops calling a system that’s down, retries that run out go somewhere a person sees them, and the receiver can take a repeat and old news.
Published isn’t delivered. The return receipt: a record saved with the change, a receipt from the other side, and a receiver that’s safe to send twice. The industry calls the first part the transactional outbox.
Who is notified. Alert on patterns, at a severity that matches who’s affected; watch for the run that never happened; an owner and a runbook for every integration; recovery in three sizes: replay one, catch up, or resync.
Your log, and Salesforce’s. Your log says what should have happened; Salesforce’s says what did. One identity per integration is what makes “normal” measurable.
Reports for each reader, with targets set per business capability, run on counts kept after the payloads go.
Three releases a year. Pinned API versions, Release Updates, a preview sandbox weeks ahead, and three kinds of test.

The path through this chapter

Nine modules, each standing on its own.

Every Arrow, Out and In. Errors will happen; log, tell, retry; how will they retry; zoom out, then zoom in; who owns the error, pattern by pattern; top to bottom; the failure plan, one row per arrow.
Capture Everything: the Integration Log. Standard door or your own; what one entry holds; one ID through every layer; the log that rolls back; the ledger and the to-do list.
The Log Is Data: Retention and Masking. Two phases, then three tiers; purging at volume; field by field; a hash to match without exposing; the law, around the world.
Retries Without Duplicates. One layer retries; a timeout sized to who is waiting; the circuit breaker; where a retry goes when it stops; repeats and old news; a retry plan for one arrow.
The Return Receipt. Registered mail; why a platform event alone isn’t enough; the outbox; three kinds of sent; receipts already built in; does every arrow need one?
Escalation and Recovery. The Saturday lesson; alert on patterns; the run that never happened; an owner and a runbook; recovery in three sizes; after it’s over.
What Salesforce Shows You. Question by question; the composite call that succeeded; how fast, really; the normal-looking call that isn’t normal; across every system.
Reporting. Who reads which report; the measures that matter; targets per business capability; where the numbers come from; the morning report.
Resilience to Releases. Your version at the bottom of the list; Release Updates; the preview sandbox; three kinds of test; a release checklist; a partner that fails on purpose.

Chapter 15 takes the arrows that go beyond one call: orchestration, where several systems take part in one piece of work, and what happens when one step fails partway through.

When you’ve finished, check yourself with the chapter study guide: a checklist for the exam, the habits beyond it, the words to know, and a few questions.

What an architect decides here

For every arrow: the failure plan, where it’s logged, who’s told, whether there’s a retry and which layer owns it, written before it’s built.
The log: what each entry holds, the ID every layer writes down, how it survives the failure, where the payloads live at volume.
The log’s lifecycle: retention per tier, the purge, the flag, and what’s masked, hashed or never written.
The retry plan: timeouts, backoff, the circuit breaker, and where retries go when they run out.
Which arrows need proof of delivery, and how the receipt comes back.
Escalation and recovery: an owner, a runbook, alerts by severity, silence checks, and replay, catch-up or resync.
What’s watched and reported, for which reader, with which targets.
What happens at every release, Salesforce’s and every partner’s.

Beyond the exam: an AI assistant will add retries to every function, draw twenty charts, and read a million log lines in a minute. What it won’t decide is which layer owns the retry, which five numbers matter, or whether a lost status costs money. Those decisions are this chapter. And whatever you hand it, hand it the masked log.

Explore

Salesforce Architects, Well-Architected: Reliability (opens in a new tab): failures, circuit breakers, timeouts, service level objectives, in Salesforce’s words.
Salesforce Architects, Well-Architected: Operational Excellence (opens in a new tab): monitoring, alerting, runbooks and incident reviews.
Salesforce Developers Blog, The Big Objects Playbook: Payload Capture and Replay (opens in a new tab): the ledger and the to-do list (June 2026).

Next: Integration Orchestration