Every Arrow, Out and In
Module 1 of 9 in the chapter Reliability, Recovery, and Observability, chapter 14 of the Integration Architect course. Each module stands on its own. Previous chapter: Trust, Identity, and Integration Security · Next: Capture Everything: the Integration Log
The question this module answers: What does an integration architect ask about every arrow, before it’s built, so that a failure has somewhere to go?
Errors Will Happen
Section titled “Errors Will Happen”When I teach this, I tell the class there’s one set of questions I want them to ask so often it becomes a habit. If I see a single outbound arrow going out of anywhere, I want you to be able to answer: where do I log, what do I do with the users, how do I retry? And then the reciprocal. For every inbound arrow coming in: what if they get an error, how are they going to retry, how are they going to recover, and what goes on in the meantime?
These are the questions an integration architect asks every time. Errors will happen, and you need to prepare for them.
Salesforce says the same thing in its own framework. Well-Architected’s Reliability guide puts it plainly: “Reliability isn’t about preventing all failures. Failures happen in distributed systems.” And an integration is a distributed system, by definition. Two systems, a network between them, and a conversation that can stop at any point.
You’ve heard these questions before. Chapter 1 asked them about one arrow, and asked how long it might be down (Plan for Failure). Part 2 has given you the mechanisms since then: the doors in, the callouts out, bulk, events, identities. Now we put the questions to every one of them, and this chapter answers them.
Every Outbound Arrow: Log, Tell, Retry
Section titled “Every Outbound Arrow: Log, Tell, Retry”Start with an arrow leaving your system. Say a user hits a button, and Salesforce calls out to check the weather.
Where do I log it? If the weather service is down, does anyone ever know? An admin can only fix what someone recorded: when it happened, what was sent, what came back. If the failure lives only in a user’s memory, it didn’t happen, as far as the system is concerned.
What do I do with the users? The user is sitting there, waiting. What do you say to them? “The weather service is down, try again in a few minutes”? Do you give them a try-again button? Or a spinning wheel that never stops?
How do I retry? For the weather, maybe you don’t. Nobody needs this morning’s forecast this afternoon.
Now change one thing. Instead of checking the weather, the button checks an account balance, and the next step of the business process depends on it. Same arrow, same failure. But now the answers change. If the user gives up and moves on, the work still has to happen. How will you build it to happen later, without them?
That’s why the questions come first, before the design. The arrow didn’t change. What it means to the business did, and that’s what decides how much logging, telling and retrying it needs. Notice, too, the word I use: “is there ever a retry?” Sometimes the right answer is no, and that’s a decision too, as long as someone made it.
Every Inbound Arrow: How Will They Retry?
Section titled “Every Inbound Arrow: How Will They Retry?”Now turn the arrow around. Another system calls into you.
Their side has the same three questions: they log, they tell their users, they retry. But you decide a lot of what they can do. So for every inbound arrow, I ask:
- What if they get an error? Do they get one they can act on? A clear status, a message that says which record failed and why. A timeout tells them nothing (chapter 8 module 5 is how your own door answers).
- How are they going to retry? If they send the same thing again, is that safe? It’s safe only if your side can recognize the second copy: their own reference, stored as an External ID, so the resend updates instead of creating a duplicate. The Data Architect course takes that apart in Safe to Retry.
- How are they going to recover? After a long outage, how do they find out what got through and what didn’t? What can they ask you for?
- What goes on in the meantime? While their calls fail, what happens to your users and your data? Is the delivery that should be on tomorrow’s schedule just missing?
Salesforce’s integration patterns guide is clear about who owns this. For remote call-in, “All the remote call-in methods, standard or custom APIs, require the remote system to handle any subsequent errors, such as timeouts and the management of retries.” The caller owns its retries. But whether a retry is safe is in your hands, and the same guide adds, “it’s important to ensure idempotent design characteristics.”
The questions, out and in.
Zoom Out, Then Zoom In
Section titled “Zoom Out, Then Zoom In”Let’s try it on Steve Trucking. Here’s the landscape you’ve been building since chapter 1, every box in its place. One arrow has failed.
From 30,000 feet: one red arrow among many.
It’s an ordinary afternoon. Redrock’s webhook stops answering, for, let’s say, 40 minutes. Nothing on our side is broken. The crews are still delivering, Salesforce is still recording every status change, and the integration layer is still trying to send each one to Redrock, about eight for every delivery. Every one of those calls fails.
Zoom in on that one arrow, and ask the questions.
One arrow, zoomed in: what happens next.
Where do I log it? Every attempt, on the integration layer’s side: which Work Order, which status, when, and what came back. Not just “Redrock failed,” but enough to find exactly which statuses never arrived. That’s module 2.
What do I tell the users? Here’s the interesting part. Nobody is sitting at a screen waiting for this call. It’s not the weather button. So “the user” isn’t a person in front of Salesforce. It’s the integration’s owner, who needs to know the statuses are falling behind, and Redrock’s own team, whose customers are seeing yesterday’s status. Telling them is a design decision: after how long, through what, and who? That’s module 6.
How do I retry? Retry, but don’t keep calling a system that’s already down: wait a little longer between each try. And retrying is safe here, because “set the status to Delivered” can be done twice and still be right; that’s the event twice from chapter 11. If the tries run out before Redrock comes back, the statuses wait somewhere a person can see them (module 4). And for anything that was sent but never confirmed, the return receipt finds it.
Now turn the arrow around, the reciprocal. Redrock calls us with manifests every evening. If our side times out, Redrock is the one asking the questions. Will they get an answer they can act on? Will their resend be safe? (It is, because their reference is the key on our Work Orders; the Data Architect course drew exactly this timeout.) And after a long outage, how will they find out what got through?
Same landscape, same arrow, two ends. That’s the habit.
Who Owns the Error, Pattern by Pattern
Section titled “Who Owns the Error, Pattern by Pattern”For the exam, and for every design review, know where Salesforce’s patterns guide puts the error. It doesn’t leave this vague. Its words: “It’s important to include an error handling and recovery strategy as part of the overall solution.” Then, pattern by pattern:
| Pattern | Who handles the error | Who retries |
|---|---|---|
| Request and Reply (Salesforce calls out and waits) | Salesforce, the caller: “an error message displayed on the end user’s page or logged to a table requiring further action” | The caller can retry, because “Changes aren’t committed to Salesforce until the caller receives a successful response” |
| Fire and Forget (a callout that hands off) | Salesforce only for the hand-off: “the callout only handles exceptions in the initial invocation of the remote service” | Nothing by default: “A custom retry mechanism must be created if quality-of-service requirements dictate it” |
| Fire and Forget (a platform event) | The subscriber, the remote system | The subscriber, by replaying: “There is no retry on the Salesforce side. It is up to the ESB to request that the events be replayed.” |
| Batch Data Synchronization | The job, with control tables and error tables | A restart; sometimes a delayed one, because “it allows time to triage and correct data” |
| Remote Call-In (they call you) | The remote system | The remote system, “such as timeouts and the management of retries”; middleware can provide the logic |
Read the right-hand column again. In almost every row, somebody has to build the retry. The platform gives you the pieces. The design is yours.
Test knowledge. Error handling differs by pattern (BLD-4). Request and Reply: the caller handles it, and can retry because nothing commits until success. Fire and Forget: Salesforce handles only the hand-off; any retry is custom. Platform events: no retry on the Salesforce side; the subscriber replays. Remote Call-In: the remote system handles timeouts and retries, and your side must be idempotent. Batch: control and error tables, and a restart.
Top to Bottom: Where the Handling Lives
Section titled “Top to Bottom: Where the Handling Lives”There’s an answer you’ll hear on the exam, and I want you to know it and then go one step further.
The exam will usually tell you most error handling belongs in the middleware, because it’s your integration layer. That’s true. It’s the layer built for retries, queues, alerts and logging, and it sees both ends.
But I believe it needs to happen in Salesforce and at the endpoints too. Top to bottom. The middleware can’t tell a Salesforce user that their save didn’t go through. It can’t stop a flow from committing half its work. And it can’t make the other system’s resend safe. Each layer handles the failures it can see.
Salesforce’s Reliability guide says it its own way: “Implement failure detection at multiple levels to catch issues before they cascade into full outages.” Not one level. Several.
So when you draw the arrow, ask the questions at each layer it passes through: Salesforce, the integration layer, the other system. The answers won’t be the same, and they shouldn’t be.
The Failure Plan: One Row per Arrow
Section titled “The Failure Plan: One Row per Arrow”Here’s how I make this real on a project. Every arrow in the landscape gets a row, and every row gets answers before the arrow gets built. It lives in the same inventory as everything else about your integrations (chapter 5 module 6).
A first pass for four of Steve Trucking’s arrows:
| Arrow | Where it’s logged | Who’s told | Is there a retry? |
|---|---|---|---|
| Status out to Redrock (webhook, seconds) | The integration log, every attempt | The owner when statuses fall behind; Redrock’s team if it lasts | Yes, waiting longer each time; then a queue a person works |
| Manifests in from Redrock (their call) | Our log, every call in, with Redrock’s reference | Redrock, through a clear answer; the coordinators if tonight’s manifest is late | Redrock’s, made safe by their reference as our key |
| Keystone’s nightly status file | The job’s log: the file, its count, its pickup | The owner if Keystone hasn’t picked it up by morning | The next night’s file, or a resend on request |
| Latest truck position from FleetLink | Only gaps longer than a few minutes | The owner, if positions stop | No: the next reading replaces the missed one |
Look at the last row. Not every arrow needs a retry. A truck position is out of date a few minutes later anyway, so retrying an old one is wasted work. The answer to “is there ever a retry?” can be no. What matters is that someone asked.
This is a first pass. Each module in this chapter fills in a column.
The Rest of This Chapter
Section titled “The Rest of This Chapter”Those are the questions. The rest of the chapter is how you answer them.
- Module 2: capture everything, the integration log.
- Module 3: the log is data, its retention and masking.
- Module 4: retries without duplicates.
- Module 5: the return receipt.
- Module 6: escalation and recovery.
- Module 7: what Salesforce shows you.
- Module 8: reporting.
- Module 9: resilience to releases.
Beyond the exam: what AI changes. Give an AI assistant your landscape and ask it, for every arrow, “where do I log, what do I tell the users, how do I retry?” It will produce a complete-looking table in seconds, and it’s a good first pass. But it doesn’t know that Redrock’s customers phone when a status is late, or that nobody cares about this morning’s truck position this afternoon. What each arrow means to the business is still yours to decide.
What an Architect Decides Here
Section titled “What an Architect Decides Here”- For every outbound arrow: where it’s logged, what the users are told, and whether there’s ever a retry, written before it’s built.
- For every inbound arrow: the answer a caller gets when something fails, how their resend stays safe, and what they can ask for to recover.
- Which layer handles which failure: Salesforce, the integration layer and the other system, top to bottom.
- Who owns each arrow’s failure plan, and where it lives: one row per arrow, in the inventory.
Explore
Section titled “Explore”- Salesforce Architects, Well-Architected: Reliability (opens in a new tab): failures, recovery and detection at several levels, in Salesforce’s words (checked 2026-11-07).
- Salesforce Architects, Integration Patterns (opens in a new tab): each pattern’s error handling and recovery (checked 2026-11-07).
- Plan for Failure: chapter 1, a minute, an hour, a day, a weekend.
- Safe to Retry: the Data Architect course, idempotency in depth.