Retries Without Duplicates

Module 4 of 9 in the chapter Reliability, Recovery, and Observability, chapter 14 of the Integration Architect course. Each module stands on its own. Previous: The Log Is Data: Retention and Masking · Next: The Return Receipt

The question this module answers: When a call fails, who tries again, how long do they wait, when do they stop, and how does the other side stay safe from the repeats?

The Bare-Bones Retry

When I ask a class “how would you retry?”, I always get good answers. Put a status on an object, have a scheduled job every ten minutes pick up anything marked for retry, and send it again. Or have the Apex schedule itself to run again in ten minutes.

The one I call the bare-bones retry goes like this. Say you’re sending closed opportunities to an ERP. Put a send status on the opportunity, and a retry count, because you need to handle more than one retry. An hourly batch looks for a send status of error with the retry count under the maximum, and sends them again. And if it finally fails, send an alert to the admin. That still works, and you can hang a flow off the status to send the alert.

The Data Architect course takes the rest of the theory apart (Safe to Retry): which failures to retry at all, why a retry waits longer each time and adds a little randomness, why a self-scheduling job doesn’t hold up at volume, and why a retry must stop. I won’t repeat it. This module is the integration architect’s half: an arrow passes through several layers, and every one of them can retry.

One Layer Retries, Not Every Layer

Here’s the trap. Each layer was built by a careful team, and each one added retries.

Picture Steve Trucking’s status arrow. Salesforce publishes a status event. The integration layer receives it and calls Redrock’s webhook. Now suppose Salesforce’s subscriber retries three times, the integration layer’s flow retries three times, and the HTTP connector inside it retries three times. Redrock goes down. How many calls hit Redrock for one status? Not three. Three times three times three: twenty-seven. For every status, from every delivery, while Redrock is trying to come back up.

Amazon’s engineers wrote this down from their own outages: “If each layer retries independently, the load on the database will increase 243x… the retries at each layer multiply — first three tries, then nine tries, and so on.” Their advice: “our best practice is to retry at a single point in the stack.”

Click to enlarge

Retries multiply.

So for every arrow, the design names one owner of the retry. Usually it’s the layer closest to the failing call, which for Steve Trucking’s webhook is the integration layer. The other layers try once and report what happened.

That’s not the same as error handling living in one place. Module 1 said handle errors top to bottom, and that still holds: every layer logs and every layer reports. Only one of them tries again.

A Timeout Sized to Who Is Waiting

Before a retry, there’s a wait. How long does the caller give the other side before it decides the call failed?

Chapter 9 gave you the mechanics: an Apex callout waits 10 seconds by default, and you can set up to 120 (chapter 9 module 2). Salesforce’s Reliability guide gives the design rule: “Configure timeouts appropriate to operation type (5–10s user-facing, 30–60s async).” And it names the trade-off plainly: “Shorter timeouts fail faster, enabling fallback strategies to engage sooner. Longer timeouts increase success rate for slow-but-functional external systems.”

Go back to the account balance from module 1, where someone is sitting at the screen. They can’t wait through a 30-second timeout and then three retries. That’s two minutes of a spinning wheel. So a call where a person waits gets one try, with a short timeout, and if it fails, the person is told and the work is handed to the background, where the retries can take their time. The background arrow is where you can wait longer and try again.

Keep an eye on the totals, too. A transaction’s callouts share a budget, 120 seconds in all, and slow transactions count against an org-wide limit (chapter 9 module 4). A long timeout that’s hit often isn’t patience. It’s a limit you’re spending.

Stop Calling a System That’s Down

Retries are for a call that failed once. What about a system that’s been down for twenty minutes? Every new status still tries, waits for its timeout, fails, and retries. You’re spending callouts and time on a call you already know will fail.

That’s what a circuit breaker is for, named after the switch in your home’s electrical panel. Salesforce’s Reliability guide describes it: “The circuit breaker pattern prevents cascading failures when integrations become unavailable. Instead of accumulating timeouts that consume transaction time and governor limits, detect failure patterns and stop calling failing systems.” In short, “fast failure rather than slow failure.”

It has three states:

• Closed: normal. Calls go through.
• Open: too many failures in a row. Calls “fail immediately without attempting external calls.” The work is kept, not sent.
• Half-open: after a wait, a few test calls “probe external systems to detect recovery.” If they work, the breaker closes and the kept work goes out. If they fail, it opens again.

Click to enlarge

Fast failure, not slow failure.

Where does the breaker live? Most integration platforms have one built in, and if the integration layer makes the call, that’s the place. When Salesforce makes the call, the guide is specific: “Implement circuit breakers using Platform Cache to store circuit state accessible across all transactions. Use Platform Events to broadcast state changes across the org.” Every callout checks the breaker first.

And an open breaker is news. It means a partner is down, right now. Somebody should hear about it, which is module 6.

Where a Retry Goes When It Stops

Every retry plan ends. After the last try, the work has to go somewhere a person will see it. Never nowhere.

In messaging systems, that place has a name. The Enterprise Integration Patterns book calls it a dead letter channel: “When a messaging system determines that it cannot or should not deliver a message, it may elect to move the message to a Dead Letter Channel.” Many integration platforms and cloud queues have one built in.

In Salesforce, it’s the to-do list from module 2: the custom object that holds only the failures. Salesforce’s playbook shows the whole cycle: a daily batch replays the open failures, and “After three failed attempts it’s marked PermanentFailure and skipped, so a persistently broken payload can’t loop forever.” Then the important part: “fire a proactive notification when a record reaches PermanentFailure, so your team doesn’t discover dropped payloads by accident.” That’s the bare-bones retry’s alert to the admin, built at volume.

The Other End: Repeats and Old News

Everything so far is the sender. Now the receiver, because retries make two kinds of trouble on the other end.

The same message twice. A retry after a timeout may send something that already arrived. The receiver has to recognize it. For an update, the partner’s key does that: an upsert on Redrock’s reference updates the Work Order that’s already there, rather than making a second one. For an event, the ID that travels with it does: the EventUuid, sent as the webhook-id (module 2), lets Redrock skip a status it has already handled.

Old news after new news. This one catches people. A delivery’s status goes to “Out for delivery,” and the call to Redrock fails. The retry is queued. Ten minutes later the status changes to “Delivered,” and that call goes through. Then the retry of “Out for delivery” finally succeeds. Redrock now shows the delivery as out for delivery, after it was delivered. A retry delivered old news on top of new news, the same out-of-order problem chapter 11 found on the event bus (chapter 11 module 4).

The fix is in the payload: send the time of the change with every status, and agree with the receiver that it keeps the newest by that time, not the one that arrived last. That’s a line in the contract, written before anything is built.

For a call that creates something new and has no natural key, say a charge for a missed delivery, there’s a newer tool, the Idempotency-Key header. The client makes a unique key for the operation and sends the same key on every retry, and the server, having seen the key, doesn’t do the work twice. The IETF draft describes it in one line: “The HTTP Idempotency-Key request header field can be used to make non-idempotent HTTP methods such as POST or PATCH fault-tolerant.” It’s still a draft, so in the IETF’s own words it’s “work in progress”, but many payment and commerce APIs already use the idea. If the other system supports it, use it. If you’re building the door, consider offering it.

A Retry Plan for One Arrow

Put it together for Steve Trucking’s status arrow to Redrock. As a labelled example:

Click to enlarge

One arrow’s retry plan.

Those numbers are for this arrow, nobody’s standard. A webhook that Redrock’s customers check within minutes might give up sooner and call a person. A nightly file can wait until morning. The architect’s job is to write the plan down, per arrow, before the first failure decides it for you.

Test knowledge. Know the circuit breaker’s three states (closed, open, half-open) and that Salesforce’s Well-Architected guidance stores its state in Platform Cache. Know Salesforce’s timeout guidance: 5 to 10 seconds where a user waits, 30 to 60 seconds for asynchronous work, within the 120-second total for a transaction’s callouts. Know that retries in several layers multiply, so one layer should own the retry.

Beyond the exam: what AI changes. Ask an AI assistant to “add retry logic” and it will, in every function it touches, because every function looks like it could fail. That’s exactly how three layers end up with three retries each. Tell it which layer owns the retry, and ask it to make every other layer try once and report.

What an Architect Decides Here

• Which one layer owns the retry for each arrow, and that every other layer tries once and reports.
• Each call’s timeout, sized to who’s waiting: short and one try where a person waits; longer, with retries, in the background.
• Where the circuit breaker lives, when it opens, how it tests, and who hears that it opened.
• Where retries go when they run out, and who’s told.
• How the receiver handles repeats and old news: a key or an ID for the repeat; the time of the change for the order; the Idempotency-Key header where there’s no natural key.
• The retry plan, per arrow, written down in the inventory.

Explore

• Salesforce Architects, Well-Architected: Reliability (opens in a new tab): circuit breakers, timeouts by operation type, retry with backoff (checked 2026-11-07).
• AWS Builder Center, Timeouts, retries, and backoff with jitter (opens in a new tab): why retries in every layer multiply (checked 2026-11-07).
• Enterprise Integration Patterns, Dead Letter Channel (opens in a new tab) (checked 2026-11-07).
• IETF, The Idempotency-Key HTTP Header Field (opens in a new tab): an Internet-Draft, work in progress (checked 2026-11-07).
• Safe to Retry: the Data Architect course, which failures to retry and how.