Safe to Retry

Image generated with AI

The question this module answers: How do integrations and agents write without creating duplicates or corrupting data when they have to retry?

The Credit That Went Out Twice

A Retail Partner’s DC manager files a damage claim in the portal: the dishwasher arrived with a dented door. Customer care approves it, and Salesforce sends the finance ERP a request through the integration layer: credit the partner $180 for this claim. The ERP issues the credit. Then the connection drops, and the answer never makes it back.

What does the integration layer know? It knows it sent the request. It knows it didn’t get a successful response. It does not know the credit failed. So it does what it was built to do: it tries again. The ERP gets a second request to credit the partner $180, and it does. The partner now has $360 for one dented door, and somebody in finance spends a morning finding it.

Click to enlarge

Nothing malfunctioned. The retry worked perfectly. That’s the problem. Modules 1 and 2 were about keeping transactions from colliding; even after all four levers, some will still fail, and networks still time out. This module is about the moment after: whether the second attempt can do no harm.

Failure and Uncertainty Are Different

Chapter 9 made this point, and it’s worth building on: a request that failed and a request whose answer was lost look identical to the caller. The request may never have reached the ERP. It may have reached it and failed. It may have succeeded completely, with only the response lost. From the caller’s side, all three can look like one word: timeout.

So reliable architecture can’t assume “no response means nothing happened.” And a timeout doesn’t mean the other side stopped working. The ERP may still be processing the first request when the retry arrives, and now the caller is competing with itself. Timeout and failed aren’t synonyms. Design for the ambiguity, and retries become a data architecture problem.

The Dangerous Retry

Some actions are harmless to repeat; many aren’t. Credit the partner, retry: two credits. Ship a replacement part, retry: two parts on two trucks. Create a case for the Recipient’s complaint, retry: two cases and two agents calling the same person. Bill a delivery, retry: the partner is invoiced twice and disputes the whole invoice.

A reliable system isn’t one where operations never fail. That’s impossible. It’s one designed so that the failures we expect don’t turn into wrong business records.

Idempotency

In module 1, I gave you the word: idempotency. Multiple attempts mustn’t break something. Press the elevator button five times and you still get one elevator.

That doesn’t mean nothing happens on the second request. The receiver might recognize it already did the work and return the first answer. It might update the same record again with the same values. It might ignore the duplicate. What matters is that the retry doesn’t create another business event that shouldn’t exist. Stripe, which processes payments for a living, puts it in one line: idempotency is for “safely retrying requests without accidentally performing the same operation twice.”

The Business Operation Needs an Identity

How can the ERP tell whether this is a second credit, or a retry of the first one? It needs an identity to check.

Sometimes the record’s own identity is enough. In Checkpoint 1 we gave every delivery a key: the Retail Partner’s own order number, a unique External ID on the Work Order. A resent manifest upserts on that key, so it updates the same Work Order instead of creating a second one. Billing works the same way: each delivery is billed once, so if the ERP keys its invoice line on that same delivery key, a retried “bill this delivery” finds the line it already wrote. That’s record identity: which record is this?

Credits are different. One delivery can legitimately earn more than one credit: the dented dishwasher today, a missing installation part next week. So the delivery key can’t tell a retry from a genuine second credit. The operation needs its own identity: which business action is this? At Steve Trucking, every credit comes from a claim, and every claim is a Case with its own number. Send that claim number with the request as the idempotency key, and the ERP can ask: have I already processed this claim’s credit? If yes, don’t credit again; return the first result.

Click to enlarge

Salesforce’s Well-Architected guidance says the same: design actions “so a retried call produces the same end state as a single successful call, using an idempotency key such as a request ID or external ID with upsert logic instead of insert, so a retry updates the same record rather than creating a duplicate.” Inside Salesforce, that’s the tool you already have: a unique External ID and an upsert. Save each incoming request on a record whose External ID is the idempotency key, and a second arrival can’t create a second record; the unique field rejects it.

Here’s what both kinds of identity look like in Apex. Record identity first: a resent manifest upserts on the delivery key, so it finds the Work Orders it already wrote.

// Partner_Delivery_Key__c is a unique External ID on the Work Order (Checkpoint 1).
// Found: update it. Not found: create it. Resent: the same Work Order, not a second one.
List<Database.UpsertResult> results = Database.upsert(
    workOrders,
    WorkOrder.Fields.Partner_Delivery_Key__c,
    false                                  // allOrNone: one bad row doesn't fail the rest
);

Database.upsert takes the field “that upsert() uses to match sObjects with existing records”; it can be “a custom field marked as external ID” (Database Class (opens in a new tab), Apex Reference Guide, checked 2026-10-31). And if the key “is matched multiple times, then an error is generated,” which is chapter 10’s point about keys that aren’t really unique (Upserting Records (opens in a new tab), Apex Developer Guide, checked 2026-10-31).

Then operation identity, on the receiving side. Say the credit requests arrive in Salesforce, and each one is saved on a Credit_Request__c whose Claim_Number__c is a unique External ID. Here an insert is the right call, not an upsert: a retry mustn’t overwrite the first request, it must be recognized.

Credit_Request__c request = new Credit_Request__c(
    Claim_Number__c = claimNumber,         // the idempotency key, unique
    Partner__c      = partnerId,
    Amount__c       = amount,
    Status__c       = 'Received'
);
Database.SaveResult result = Database.insert(request, false);

if (!result.isSuccess()
        && result.getErrors()[0].getStatusCode() == StatusCode.DUPLICATE_VALUE) {
    // We already have this claim's credit: it's a retry, not a second credit.
    // Look up the first request and return its result.
}

DUPLICATE_VALUE is Salesforce’s status code for “a duplicate value for a field that must be unique” (Core Data Types Used in API Calls (opens in a new tab), SOAP API Developer Guide, checked 2026-10-31). The database enforces the key, so two copies arriving at the same moment still make one record.

Retry the Same Intent

A retry has to represent the same intended operation. Don’t generate a brand-new key every time you retry. If attempt one carries claim C-48213 and the retry carries a freshly made request number, the ERP has no way to know they’re the same credit. From its side, two different credit requests arrived.

So the key is created once, when the business decides to act, and stored with the work: the claim number is already on the Case. Every retry, by the integration layer, a batch job or a person pressing Resend, sends that same key. And decide how long the receiver remembers keys. Stripe, for example, can prune a key once it’s 24 hours old. A retry that comes after the receiver has forgotten the key is a new request again.

Where Should I Require an Idempotency Key?

I wouldn’t add idempotency machinery to every field update in the system just because we can. Ask what happens if the operation runs twice. Setting a delivery’s status to Delivered twice is harmless: it’s still Delivered. A second $180 credit isn’t. A second replacement part isn’t. A second invoice isn’t.

So require an operation key where repeating the action creates another business event with a real consequence: money, goods, a message to a customer. The higher the consequence, the stronger the guarantee I want.

Updates Can Be Idempotent Too

Idempotency isn’t only about inserts. The API method doesn’t decide it; the business meaning does. “Set the status to Delivered” can run twice safely. “Add one to the Manifest’s delivered count” can’t: run it twice and the count is wrong.

That matters for module 2’s design. The platform event trigger that keeps the Manifest’s counts current will sometimes see an event more than once: Salesforce’s own retry mechanism for these triggers sends a batch again, and a trigger that resumes after a failure can pick up events it had started. Well-Architected’s advice for replays is that they work when “the subscriber is idempotent.” So the subscriber shouldn’t add one per event. It should recount the Manifest’s Work Orders and write the total. Recount twice, and you get the right number twice.

Proof of Delivery Still Applies

In chapter 9 we established proof of delivery: the client owns the conversation, and whoever knows what should arrive proves that it did. If I send data to another system, I don’t get to say, “I called the API, my job is done.” I need to know it got there, and retry is part of meeting that obligation.

Now add the companion rule: if the client owns the retry, the architecture must make the retry safe. Otherwise we’ve built a contradiction. We tell the sender, “keep trying until you’re sure it arrived,” while the receiver is designed so that every try might create another copy. That isn’t reliability. Completeness and idempotency belong together.

Here’s what that looks like on the same lost answer. Before it sends, the integration layer writes the request to its send log with the claim number. The ERP issues the credit and records the claim number. The answer is lost, exactly as before. The retry carries the same claim number; the ERP finds it already done and returns the first answer, with its credit reference. Now the send log can mark the claim delivered, with the ERP’s own number as the proof. One credit, and proof that it arrived.

Click to enlarge

Inside One Salesforce Transaction: All, Part, or Back to Here

Before we retry anything across the network, know what Salesforce does with a failure inside one transaction. By default, it’s all or nothing: “If the request doesn’t complete successfully, all database changes are rolled back” (Transaction Control (opens in a new tab), Apex Developer Guide, checked 2026-10-31). That’s a gift. Half a credit never gets saved.

You have two ways to choose something else.

Record by record. The Database methods take allOrNone. Set it to false, and the records that can save, save; each one that can’t comes back with its own error. That’s how you retry only what failed, and only the failures worth retrying:

List<Database.SaveResult> results = Database.update(workOrders, false);

List<WorkOrder> retryLater = new List<WorkOrder>();
for (Integer i = 0; i < results.size(); i++) {
    if (results[i].isSuccess()) {
        continue;
    }
    Database.Error err = results[i].getErrors()[0];
    if (err.getStatusCode() == StatusCode.UNABLE_TO_LOCK_ROW) {
        retryLater.add(workOrders[i]);     // transient: try again later, with backoff
    } else {
        // permanent until fixed (validation, bad data): park it, don't retry
    }
}

Back to a savepoint. When several steps must stand or fall together, but the transaction should carry on, set a savepoint and roll back to it:

Savepoint beforeCredit = Database.setSavepoint();
try {
    insert request;                        // the credit request, keyed by claim number
    claim.Status = 'Credit Requested';
    update claim;                          // the claim Case moves on with it
} catch (DmlException e) {
    Database.rollback(beforeCredit);       // undo both, never just one
    // ...then record the failure, outside the work we just rolled back
}

Across the API. An integration can ask for the same all-or-nothing. A partner’s new delivery arrives as a Work Order and its line items. Sent as separate calls, a failure halfway leaves a Work Order with half its lines. Sent as one Composite request with allOrNone, it all saves or none of it does, and both upsert on their keys, so a resend is safe too:

POST /services/data/v68.0/composite
{
  "allOrNone": true,
  "compositeRequest": [
    {
      "method": "PATCH",
      "url": "/services/data/v68.0/sobjects/WorkOrder/Partner_Delivery_Key__c/REDROCK-8472914-1",
      "referenceId": "workOrder",
      "body": { "Status": "New", "Subject": "Dishwasher delivery and install" }
    },
    {
      "method": "PATCH",
      "url": "/services/data/v68.0/sobjects/WorkOrderLineItem/Partner_Line_Key__c/REDROCK-8472914-1-01",
      "referenceId": "haulAway",
      "body": {
        "WorkOrder": { "Partner_Delivery_Key__c": "REDROCK-8472914-1" },
        "Description": "Haul away the old dishwasher"
      }
    }
  ]
}

“If the value is true, the entire composite request is rolled back”; if false, “the remaining subrequests that don’t depend on the failed subrequest are executed” (Composite Request Body (opens in a new tab), REST API Developer Guide, checked 2026-10-31). One request carries “up to 25 subrequests” (Send Multiple Requests Using Composite (opens in a new tab), checked 2026-10-31). The line item finds its Work Order by the partner’s key, not by a Salesforce ID, the way Salesforce’s upsert guide shows for “a relationship” referenced “using an external ID” (Insert or Update (Upsert) a Record Using an External ID (opens in a new tab), checked 2026-10-31).

Three rules from the same guide. Savepoints aren’t free: “Each savepoint you set counts against the governor limit for DML statements.” A record inserted before the rollback keeps its ID in your variable, so to insert it again, “create an sObject variable without setting its ID.” And a rollback only undoes Salesforce’s changes. That’s why the call to the ERP doesn’t happen in the same transaction: with saves still pending, a callout fails with “You have uncommitted work pending. Please commit or rollback before calling out.” So the credit request is saved and committed first, and a separate transaction sends it: the send log from proof of delivery, written down before anything leaves.

Test knowledge. A Salesforce transaction is all or nothing by default: any failure rolls back every change. Database methods with allOrNone set to false allow partial success, returning a result per record so you can retry only the failures. Database.setSavepoint() and Database.rollback() undo part of a transaction; each counts against the DML statement limit. A rollback can’t undo a callout, and a callout can’t be made with uncommitted work pending. Across the REST API, a Composite request with allOrNone: true rolls back every subrequest if one fails.

Which Failures to Retry, and How

Not every failure deserves another attempt. Sort them into three kinds before deciding anything.

• Transient: try again later. A record locked by another transaction, a timeout before the request was processed, a server that’s briefly overloaded. Waiting can change the result.
• Permanent until someone fixes it: don’t retry. The claim number is missing, the amount fails validation, the request is malformed. Retrying the same payload twenty times gives you twenty failures. That’s not resilience; it’s noise. Park it, fix it, then replay it.
• Unknown: check first. The request timed out and you can’t tell whether it ran. Ask the receiver by the key whether the operation already happened, or retry with the same key and let idempotency answer. Never retry an unknown outcome with a new identity.

Well-Architected spells out the common cases: retry network timeouts with a short backoff; after a rate-limit response (HTTP 429), retry “after the Retry-After header value”; retry server errors (5xx) “with exponential backoff”; for other client errors (4xx), “Don’t retry; fix the request”; and for governor limit errors, “Don’t retry in the same transaction; re-queue as an async operation.” Its agent guidance names our case directly: lock contention is a transient failure “worth one bounded retry,” while a validation error “fails identically on repetition.”

Click to enlarge

Test knowledge. Know how to make loads and integrations safe to repeat: upsert on a unique External ID so a resent record updates instead of duplicating; give an operation that can legitimately repeat (a credit, a payment) its own unique key, kept the same on every retry. Know which failures to retry: transient ones (record locks, timeouts, server errors) with backoff; rate limits after the wait the server gives; never validation errors or malformed requests until they’re fixed.

Retry Doesn’t Mean Retry Immediately

Back to 4 p.m. A hundred status updates collide on one Manifest and fail with UNABLE_TO_LOCK_ROW. If all hundred retry immediately, we’ve sent the same traffic straight back into the same intersection. They collide again, retry again, and now we have a retry storm. Marc Brooker of Amazon put it plainly: “when failures are caused by overload, retries that increase load can make matters significantly worse.”

So a retry needs a schedule. Backoff waits before trying again, and waits longer each time. Well-Architected’s example: about 1 second, then 2, then 4, then 8, with the longest wait capped at 30 to 60 seconds. And add jitter, a little randomness, so a hundred workers that failed together don’t all wake up together; it “adds some amount of randomness to the backoff to spread the retries around in time.”

On Salesforce’s own event bus, that schedule is built in. Module 2’s subscriber can ask for its batch again by throwing EventBus.RetryableException, and “events are resent after a small delay. The delay increases in subsequent retries.” Before the retry, “the DML operations performed in the trigger before the retry are rolled back,” so a half-done recount never sticks (Retry Event Triggers with EventBus.RetryableException (opens in a new tab), Platform Events Developer Guide, checked 2026-10-31). Retry the lock, not everything:

trigger WorkOrderStatusSubscriber on Work_Order_Status__e (after insert) {
    Set<Id> manifestIds = new Set<Id>();
    for (Work_Order_Status__e e : Trigger.new) {
        manifestIds.add((Id) e.Manifest_Id__c);
    }
    try {
        ManifestCounts.recount(manifestIds);
    } catch (Exception e) {
        Boolean lockWait = e instanceof QueryException      // FOR UPDATE waited too long
            || (e instanceof DmlException
                && ((DmlException) e).getDmlType(0) == StatusCode.UNABLE_TO_LOCK_ROW);
        if (lockWait && EventBus.TriggerContext.currentContext().retries < 4) {
            throw new EventBus.RetryableException('Manifest locked: try this batch again later');
        }
        // Not a lock, or retried enough: hand it to the recovery path. Don't drop it.
    }
}

The recount is what makes this safe: the resent batch recounts the same Manifests and writes the same right totals. A trigger can run “up to 10 times when it’s retried (the initial run plus 9 retries),” then “moves to the error state and stops processing new events,” and Salesforce recommends fewer than nine retries. That’s the next section’s point, in Salesforce’s own numbers.

I don’t want a data architect memorizing “retry three times at these exact intervals.” That’s implementation policy. I want the architect asking: what failure are we recovering from, and what makes us believe waiting will change the outcome?

Exactly Once Is a Dangerous Phrase

People say, “We need this to happen exactly once.” What they usually mean is, “We can’t tolerate the business action happening twice.” That’s a valid requirement. But across a network, with lost responses and retries, literal exactly-once execution is a promise nobody can keep.

The question I can design for is: how do I make repeated delivery produce the correct business result? Effectively once, not exactly once. That leads straight back to the tools in this module: a stable operation identity, idempotency, checking state before acting, deduplication, safe retries and, at the end, reconciliation that proves the books match. Those are things we can build and test.

Eventually, Automatic Retry Must Stop

A retry policy needs an exit. Otherwise a bad request lives forever, retrying in the background. After the allowed attempts, the work moves to a different path: an error queue or an exception record, an alert to someone who owns it, a fix to the data, then a replay. Chapter 9 showed two ways to keep that list: a send status and a retry count on the record, or on an integration log.

What it must never do is disappear. Proof of delivery still applies: a failed retry isn’t permission to forget the work. It means the work moved to the recovery path.

If the work rides on platform events, know how long you can go back for it. High-volume platform event messages “are stored for 72 hours (3 days)”; legacy standard-volume events, which Salesforce retires in Summer ‘27, only 24 hours. A subscriber that’s down longer than that can’t replay what it missed, so the recovery path can’t depend on the event bus alone.

Test knowledge. Know that a subscriber can replay high-volume platform events from the last 72 hours, and that new custom platform events are high volume by default (since API version 45.0). Beyond that window, recovery needs its own record of what was sent.

Agents Have the Same Unknown Outcome Problem

Agents make all of this more common. An agent decides a claim qualifies for a credit, calls an action, and the action times out. A naive agent reasons, “It didn’t work, I’ll try again.” But it doesn’t know that. It only knows it didn’t get confirmation. If the action was “search the knowledge base,” trying again is harmless. If it was “credit the partner,” we need much stronger guarantees.

The agent shouldn’t be improvising transaction recovery. I’d rather give it an established, tested action, Issue Partner Credit, that owns its own validation, authorization, idempotency key, retry behavior, logging and the exact records it may change. The agent decides when the business action is appropriate, within its guardrails. The established process carries it out safely. For any agent action with a real consequence, the action’s contract should answer: what identifies this operation? Can it run twice safely? What happens after a timeout? Which errors are retried, how long do we wait, when do we stop, and who’s told when we do? Those are architecture decisions, not prompt-writing decisions.

Beyond the exam: what AI changes. Agents retry quickly and on their own, so unknown outcomes and duplicate actions become more likely. Give agents established actions whose identity, idempotency, retry limits and write scope are fixed in the design, rather than expecting the model to reason its way out of an unknown outcome.

Trust but Verify

How do we know the retry design works? Test it, and not only the happy path. The request succeeds and the response disappears. The request fails before it’s processed. The record is locked. The same key arrives twice; the same key arrives ten times. Two workers send the same operation at the same moment. The agent retries after a timeout. The ERP is down for an hour. The recovery job restarts halfway through.

Then prove the business result, not “the API returned 200 eventually”: one credit per claim, one invoice line per delivery, the Manifest’s count matching its Work Orders. Trust, but verify.

What an Architect Decides Here

• Which operations are harmful to repeat, and which identity (the record’s key, or an operation key) makes each one safe.
• Where the operation key is created, where it’s stored, and how long the receiver remembers it.
• Which failures are retried, with what backoff and jitter, and which go straight to correction.
• What happens after the last automatic attempt: who’s told, where the work waits, and how it’s replayed.
• What contract an agent’s actions must meet before an agent may call them.

Reliability Means Surviving the Retry

Back to the dented dishwasher. The credit request timed out, and the integration layer retried, exactly as it should. With the claim number as the key, the ERP saw C-48213, recognized it, and returned the first answer. One credit, $180, and nobody in finance spent the morning hunting for it.

That’s the chapter in one line. Module 1: see what a save locks. Module 2: design the collisions out. Module 3: make sure that when something fails anyway, and it will, the second attempt does no harm. Build for today, plan for tomorrow, and plan for failure.

Next: Data Security for Humans, Applications, and Agents

Explore

• On this site: HTTP Status Codes, every status code with the two questions this module asks: whose is it to fix, and should we try again?
• Salesforce, REST API Developer Guide: Composite Request Body (opens in a new tab) (allOrNone) and Insert or Update (Upsert) a Record Using an External ID (opens in a new tab), checked 2026-11-01.
• Salesforce, REST API Developer Guide: Send Multiple Requests Using Composite (opens in a new tab), and SOAP API Developer Guide: Core Data Types Used in API Calls (opens in a new tab) (the DUPLICATE_VALUE status code), checked 2026-11-01.
• Salesforce, Apex Developer Guide: Transaction Control (opens in a new tab) and Upserting Records (opens in a new tab); Apex Reference Guide: Database Class (opens in a new tab); Platform Events Developer Guide: Retry Event Triggers with EventBus.RetryableException (opens in a new tab); checked 2026-11-01.
• Salesforce Well-Architected: Reliability (opens in a new tab) (retry rules, exponential backoff), Resource and Cost Optimization for the Agentic Enterprise (opens in a new tab) (idempotency keys, transient versus logical failures) and Reliability for the Agentic Enterprise (opens in a new tab) (idempotent subscribers), checked 2026-11-01.
• Salesforce, Platform Events Developer Guide: Enterprise Messaging Platform Events (opens in a new tab) (retention) and Subscribe to Platform Event Notifications with Apex Triggers (opens in a new tab), checked 2026-11-01.
• Beyond Salesforce: Stripe, Idempotent requests (opens in a new tab), and Marc Brooker, Amazon Builders’ Library, Timeouts, retries, and backoff with jitter (opens in a new tab) (2019), checked 2026-11-01.
• This course, chapter 9: Proof of Delivery.