Proof of Delivery

🌐 Also in 2 languages

Español · Español (México)

Image generated with AI

Module 3 of 4 in the chapter Integration and Data Movement, chapter 9 of the Data Architect course. Each module stands on its own. Previous: Keys and Write Ownership · Next: The Integration Data Contract

The question this module answers: Who proves the data arrived, and who recovers?

The ERP sends 20,000 customer updates. Salesforce processes them. Done? How many arrived: 20,000 or 19,997? Did three fail validation? Did the connection die at record 12,417? Did the ERP retry, and did the retry create duplicates? Did Salesforce process them while the response never got back? Did the integration quietly stop three days ago? Sending data isn’t proving it arrived.

Steve Trucking’s crews know this better than anyone. They don’t drop a dishwasher on the porch and drive off. They photograph it, get the Recipient’s signature, and close the stop: proof of delivery. Integrations need the same thing. This module is about who gets the signature.

This Time Salesforce Sends

In module 1, Redrock pushed manifests into Salesforce and the integration layer was left wondering whether they’d saved. Now look at the same landscape from the other side. Every time a delivery’s status changes, Salesforce publishes it, and the integration layer delivers it to Redrock’s webhook, because Redrock shows those statuses to its own customers. Tonight, Redrock’s endpoint was down for an hour.

Click to enlarge

This time Salesforce is the sender, so Salesforce has to know what arrived.

Which statuses reached Redrock, and which didn’t? Redrock can’t tell you: it never heard about the ones it missed. This time the answer is on our side.

Who Proves It Arrived?

Back to module 1: the client starts the conversation, and only the client sees the whole round trip. So the client always owns the conversation: the timeouts, the retries, knowing whether each call got an answer. In class I put it simply: whoever is connecting, whoever is the client, takes responsibility for the connection errors.

But completeness is a different question: did everything that should have moved actually move? Only the side that knows what should have moved can answer it. The receiver can’t know “you meant to send me 3,300 statuses today” unless someone tells it. So the rule is:

The client owns the conversation. Whoever knows what should arrive proves that it did.

Usually those are the same side. When Redrock sends a manifest, Redrock is the client and Redrock knows what it sent. Sometimes they aren’t, and those are the arrows to watch.

Click to enlarge

Who connects isn’t always who proves it arrived.

Look at the third row. For status out to Redrock, the subscriber in the integration layer is the client: it connects to Salesforce’s event bus. But Salesforce is the one that knows which statuses should reach Redrock, so Salesforce has to prove they did. Later in this module, that’s what the return receipt is for. And the fourth row: a file has two hops and two clients, so the proof travels inside the file, as a row count the loader checks.

And the server? The server keeps its own promises and reports what it received. Salesforce can process a request all or nothing, refuse a duplicate key, keep events for 72 hours, and return a result for every record. It can check completeness only when the sender tells it what to expect: a count, a sequence number, a file total. Then it acknowledges what it found, and the side that knows the intent compares.

That’s why, in class, I make every architect answer the same questions for every arrow. On every outbound arrow: where do I log, what do I tell the users, and how do I retry? And the reciprocal, on every inbound arrow: if they get an error, how will they retry, and how will they recover? Errors will happen. Don’t wonder whether; plan for them.

“No Errors” Isn’t Completeness

The log says zero errors. Excellent. How many did the receiver expect? If the job accidentally sent 17,000 and the target processed all 17,000 perfectly, that’s zero errors and 3,000 missing records. Technical success and business completeness are different things. So I want reconciliation: intended 20,000, sent 20,000, accepted 19,997, rejected 3. Now there’s something to reason about.

Counts are the start, not always enough. Every night Steve Trucking sends the finance ERP yesterday’s delivery counts for invoicing: say 3,212 deliveries worth $412,560 in fees. If the ERP received 3,212 deliveries worth $398,140, the counts match and something is still wrong. Depending on the integration, reconcile with record counts, financial or control totals, minimum and maximum timestamps, sequence numbers, batch IDs, checksums, business totals. Not every API call needs an accounting system around it; match the verification to the consequence. The invoices deserve more than a marketing preference does.

Know Where You Are, and Restart Gracefully

The job dies at 12,417. Restart from record 1? From 12,418? From the last good batch? How would you know? A robust integration has some notion of progress: a checkpoint, a watermark, a cursor, a timestamp, a batch ID, a replay position, something that says I know how far I got. Without it, recovery becomes “run it all again and hope,” and hope isn’t a recovery strategy.

I care whether an integration can start cleanly, stop cleanly and restart cleanly. If recovery means “delete everything, reset three tables, call four people, hand-edit a timestamp, run a mystery script someone wrote five years ago, and hope it works,” it isn’t designed. Recovery needs stable keys, known progress, safe retries, clear failure states, a way to set bad records aside, and enough observability to know what happened. The happy path is easy. Architecture shows up in the restart.

Log Every Call

So where do you log? On projects where we owned the interface into Salesforce, we logged every incoming call to a custom object, an integration log: when it arrived, how long it took, the payload, and how it ended. We wrote it asynchronously so it didn’t slow the transaction down, and it let us track everything and replay it. We even passed a unique identifier on each request, so during load testing we could match our log to the middleware team’s performance logs. Outbound calls can write to the same log, and an admin gets a list view of what failed.

Salesforce’s Well-Architected framework gives the same advice in its own words: “Log all integration calls with request ID, endpoint, response code, and duration” (Reliability (opens in a new tab), checked 2026-10-27).

But a log is data too, and it grows. Plan its cleanup from day one. I use a two-phase purge: a weekly batch, on the quietest day, deletes the successes after a few days and the errors after about a month, with the timings in custom settings and a do-not-delete flag for anything under investigation. Be ready to explain why you’re cleaning up after yourself and not using up the client’s data storage.

Retries Are Normal; Make Them Safe

Networks fail, servers time out, tokens expire, services go down, limits get hit. A retry isn’t proof the architecture failed. A system that can’t safely retry is the problem. In module 2, keys made the resend of a timed-out manifest safe: an upsert matched what had already saved. That’s idempotency: an operation that can be repeated without unintended extra effects. In distributed systems ambiguity is normal: did the request fail, or the response? They look identical to the client. So every important integration should ask: what happens if this message is processed twice? Eventually one will be.

And how do you retry later, after the user has moved on? I’ve seen two designs work. Put a send status and a retry count on the record you’re sending, and have a scheduled batch pick up the failures under the maximum retry count and send them again, then alert an admin when it gives up. Or keep the status and the count on the integration log record instead, with a link to what it carries. Scheduling a one-off retry for each failure looks simpler, but at volume it’s fragile: ten thousand failures become ten thousand scheduled jobs.

Which errors deserve a retry, and how long to wait between them, is chapter 11’s job (Safe to Retry). Well-Architected’s short version: retry timeouts and server errors with backoff, wait as told after a rate limit, and never retry a request the server rejected as invalid; fix it instead.

The Return Receipt

Now back to status out to Redrock. Platform events are a good way to push it: Salesforce publishes and moves on, and the subscriber in the integration layer delivers. And the event bus gives you a safety net. Salesforce keeps platform events for 72 hours, and each delivered event carries a Replay ID, so a subscriber that was offline asks for everything after the last one it saw (Event Message Durability (opens in a new tab), Pub/Sub API, checked 2026-10-27).

But read the fine print. Salesforce publishes events asynchronously, and in rare cases an event isn’t persisted; in Salesforce’s words, such events “aren’t delivered to subscribers, and they aren’t recoverable.” Publishing is also at-least-once, so the same event can occasionally be published twice (Considerations for Defining and Publishing Platform Events (opens in a new tab), checked 2026-10-27). Rare isn’t never. So how does Salesforce get its proof of delivery?

The most reliable design I’ve built is the return receipt. Think of a registered letter: you keep a copy of what you sent, and the post office brings back a card signed by the person who received it.

Click to enlarge

Send: keep your own record, then publish.

When the status changes, Salesforce writes a local notification record, then publishes the platform event. The subscriber picks it up, from its replay position if it was offline, and calls Redrock’s webhook.

Click to enlarge

Sign: the subscriber returns the receipt.

When Redrock accepts it, the subscriber calls back into Salesforce through the standard API and marks the notification delivered. Notice the patterns mix: events going out, a plain API call coming back.

Click to enlarge

Sweep: what has no receipt gets sent again.

A small batch job sweeps the notifications: anything without a receipt after, say, three hours is published again. Because that can send a status twice, the subscriber keeps track of what it has already delivered and ignores repeats. Tonight’s hour of downtime is now an answer, not a mystery: 26,400 statuses published today, 26,397 receipts, three sent again. The receipts are the completeness check. And the notifications are a log like any other: purge them in two phases.

Test knowledge. Platform events and Change Data Capture events stay on the event bus for 72 hours, and each delivered event has a Replay ID a subscriber can resume from. Subscribers should expect an occasional duplicate and handle it. For delivery you must be able to prove, add your own tracking: a record of what was sent and an acknowledgment back.

Partial Failure, and the Soft Landing

Total failure is often easier: nothing happened, fix it, try again. Partial failure is where architecture gets interesting. 19,997 succeed, 3 fail. Roll back the 19,997? Retry the three? Park them for correction? It depends on the business transaction, but “some failed” has to be a state the architecture understands, or integrations fall into one of two bad habits: ignore the failures, or rerun everything.

For many data integrations I want valid and invalid records separated: let the good records land, capture the bad ones with the reason, fix or route them, retry safely. I think of it as a soft landing. One bad postal code shouldn’t stop 19,999 unrelated customers updating. But transaction boundaries matter: if those 20,000 records are one atomic business transaction, partial success may be unacceptable. Understand the business meaning, not just what the API allows.

Send the Increment, Not the Universe

Five million customers; ten thousand changed today. Why send five million tonight? Send the increment: less traffic, processing, locking, automation, error surface and reconciliation, faster recovery, and fewer unchanged values overwritten. But incremental isn’t “LastModifiedDate and hope.” What does “since the last run” mean: when it started, or when it finished? It ran from 1:00 to 1:20, a record changed at 1:19, the job failed at 1:18; two systems’ clocks disagree; records arrive late. Incremental needs a deliberate watermark and recovery model. The principle doesn’t change: know what’s been safely processed before you advance the checkpoint. (Chapter 8’s SystemModstamp point applies.)

Four Shapes of Integration

Here’s where we step into integration-architect territory, briefly, because the pattern moves the recovery point (Integration Patterns (opens in a new tab), Salesforce Architects, checked 2026-10-27).

Click to enlarge

Four shapes, four recovery points.

• Request and reply: the caller sends and waits. Intuitive, but a timeout still leaves the caller unsure whether the server committed.
• Fire and forget: the caller sends without waiting for the downstream outcome. Better decoupling, but now how does anyone know processing eventually succeeded? That’s what the return receipt answers.
• Batch: groups of records on a schedule or as a job, with natural places for counts, control totals, checkpoints and per-record results, and the partial-failure problem in plain sight.
• Events: a publisher announces that something happened; subscribers react. The publisher may not know every consumer. It owns publishing; each subscriber owns consuming, its replay position and its duplicates.

That’s why I think of platform events as a backbone of modern Salesforce integration: less coupling, consumers that evolve independently. But events don’t remove reliability problems; they move them. Change Data Capture applies the same model to Salesforce record changes, publishing change events for the objects you choose instead of making someone ask “which accounts changed?” (Change Data Capture Developer Guide (opens in a new tab)). Powerful for increments. Don’t hear “CDC solves integration”; hear “CDC changes how changes are detected and distributed.” You still need identity, authority, consumer recovery, error handling and observability.

Outbound Messaging: Know It, Don’t Start There

Salesforce also has outbound messaging, older and still supported. Salesforce says it will keep supporting it “within current functional capacities” but plans no further investment, and advises caution building with it (Data Integration with Salesforce (opens in a new tab)). It also changed recently: since Spring ‘26, outbound messages no longer include a session ID, so a listener that calls back into Salesforce must authenticate with OAuth (Sending Session IDs in Outbound Messages Is Being Removed (opens in a new tab)). So for new designs I’m thinking platform events, Change Data Capture, Flow or Apex callouts, APIs and the right bulk mechanisms. The point isn’t “old technology bad.” Know what exists, understand what you inherit, and design new systems on what Salesforce is investing in.

Test knowledge. Outbound messaging is supported but legacy: no further investment, and no session ID in the message since Spring ‘26. For new fire-and-forget designs from Salesforce, expect platform events.

Monitoring Is Part of the Integration

If an integration can fail silently, it isn’t finished. Is it running? When did it last succeed? How many records did it process, how many failed and why, how far did it get, what’s waiting to retry, who’s notified, how do they investigate and restart? Those aren’t operations questions for later. An integration that stopped three days ago is now a data quality problem.

Well-Architected’s health model makes that concrete. Next to service, integration and capacity health it lists data health: sync jobs “all current” in green, “behind schedule” in yellow, “failed or stale” in red. And for asynchronous integrations it watches queue depth: a growing queue means processing is falling behind (Reliability (opens in a new tab), checked 2026-10-27). For Steve Trucking, that’s two numbers on a dashboard: notifications without a receipt, and how old the oldest one is. When retries run out, the record goes to a place where a person looks at it, with an alert, not into silence.

And observability creates data too: logs, error payloads, retry queues, notifications, audit history. Keep enough to investigate, reconcile, audit and recover, then give it its own lifecycle. Don’t solve one data problem by quietly creating another.

Trust but Verify

Trust the API, the middleware, the event bus, the source, the target. Then verify. Did the intended data arrive? Did the right records change? Were failures captured? Can we account for every batch, resume after interruption, prove the result? That isn’t pessimism. That’s distributed systems.

Beyond the exam: what AI changes. “Agent, create the customer in the ERP.” The call times out; the agent retries. If the action isn’t idempotent, you may get two customers. Or a multi-step agent process creates a customer, creates an order, then fails updating Salesforce: what state are we in? Agents start and retry distributed actions on their own. The agent may be probabilistic; the integration contract can’t be. Give agent actions stable keys, idempotency, a record of what they sent, narrow write authority and observability.

What an Architect Decides Here

• On each arrow, who is the client and owns the conversation, and who knows what should arrive and owns proving that it did.
• What reconciliation (counts, totals, receipts, checkpoints, business controls) matches the cost of missing or wrong data.
• Where every call is logged, what users are told, how retries happen, and how the log is purged.
• How the integration stops, restarts and retries without duplicates, losses or advancing its checkpoint too early.
• Whether request and reply, fire and forget, batch or events best fit the reliability and freshness needs, and where each one’s recovery starts.
• What monitoring (data health, queue depth, alerts) makes a silent failure visible.

The happy-path diagram is easy: source, arrow, target. The architecture starts when I ask what happens if it stops halfway, the response disappears, three records fail, the same message arrives twice, the subscriber is offline, or nobody noticed it hasn’t run in three days. Whoever knows what should arrive proves that it did. Give that side keys, a log, checkpoints, safe retries, receipts, increments, captured failures and monitoring, and make sure it can start, stop and restart gracefully. Moving data isn’t finished until you have proof of delivery.

Explore

• Salesforce Developers, Pub/Sub API, Event Message Durability (opens in a new tab), checked 2026-11-01.
• Salesforce Developers, Platform Events Developer Guide, Considerations for Defining and Publishing Platform Events (opens in a new tab), checked 2026-11-01.
• Salesforce Architects, Well-Architected, Reliability (opens in a new tab), checked 2026-11-01.
• Salesforce Architects, Data Integration with Salesforce (opens in a new tab) and Integration Patterns (opens in a new tab), checked 2026-11-01.
• Salesforce Developers, Change Data Capture Developer Guide (opens in a new tab), checked 2026-11-01.
• Salesforce Help, Spring ‘26 release notes (release 260), Sending Session IDs in Outbound Messages Is Being Removed (opens in a new tab), checked 2026-11-01.
• This course, chapter 11: Safe to Retry.
• The Integration Architect course: Events (coming soon), Bulk (coming soon) and Failure and Monitoring (coming soon).