Siirry sisältöön
SteveTechArcSteveTechArc Courses

Plan for Failure

Tämä sisältö ei ole vielä saatavilla valitsemallasi kielellä.

Module 6 of 6 in the chapter Think Like an Integration Architect, chapter 1 of the Integration Architect course. Each module stands on its own. Previous: Owners, Boundaries and Change · Next chapter: The Client and Server

The question this module answers: When an integration stops, what happens to the data and the users, and how does it come back?

When people think about an integration failing, they usually picture a system going down. Salesforce is down, or the ERP is down. That happens, but it isn’t the failure I see most.

Most of the time, both systems are up. It’s the pathway between them that goes down. And there are so many ways that can happen. An internet provider has an outage. A DNS service, the internet’s address book, stops answering, and nobody can find anybody. A firewall rule changes, a certificate expires, a network link drops. A patch goes in on one side, to the operating system or to the software the integration runs on, and something that worked yesterday doesn’t today. These things happen all the time, somewhere, in some way.

The point isn’t the list. It’s that anything at all between the client and the server can stop the arrow. The server can be perfectly fine, and you can’t get to it. The client can be perfectly fine, and its calls never arrive.

So planning for failure isn’t only “what if the other system is down.” It’s “what if this arrow stops, for any reason.” And I want you thinking about it right up front, in the first chapter, because the answers change the design. They’re not something you add at the end.

Normal Time and Failure Time

Section titled “Normal Time and Failure Time”

For every arrow, I want two pictures.

The first is normal time: how the traffic flows on an ordinary day. Who calls whom, how much, how often. That’s the arrow we’ve been drawing all chapter.

The second is failure time: what happens on that same arrow when it stops. And there are a few questions I ask every time:

  • What happens to the data? Is it waiting somewhere, in a queue, an event bus, a log, the other system’s outbox? Or is it gone?
  • How are the users told? Does somebody know the arrow has stopped, or do they find out when a customer calls?
  • How does it recover when the connection is back? Will it correct itself? Does it need a person to step in? Will it be in a delayed state for an hour while it catches up?
  • How are the users told it’s caught up? A user who was warned it was behind needs to know when it’s current again.

These are the same questions from the first module, what do I log, what do I tell the user, how do I retry, asked about time. Because the answer depends on how long it’s down.

A Minute, an Hour, a Day, a Weekend

Section titled “A Minute, an Hour, a Day, a Weekend”

So I ask one more question for every arrow: what happens if it’s down for a minute? An hour? A day? Three or four days?

A minute. Usually a retry handles it. The call fails, the client waits and tries again, and nobody notices. That only works if the retry was designed in, and if sending the same thing twice is safe.

An hour. Now there’s a backlog. The data is waiting somewhere, and when the connection comes back, it all arrives at once. The integration is in a delayed state while it catches up, and users may be looking at data that’s an hour old. Do they know?

A day. The backlog is a full day’s volume, arriving on top of today’s. Can the other side take it? And this is where you have to know exactly how long the data waits for you.

Three or four days. This is where data loss starts. Somewhere, the data that was waiting runs out of room or out of time.

At What Point Is It Data Loss?

Section titled “At What Point Is It Data Loss?”

Here’s an example. Salesforce’s platform events are a good way to send changes out: Salesforce publishes an event, and a subscriber on the other side picks it up. And the event bus gives you a safety net. Salesforce keeps platform events for 72 hours, and a subscriber that was offline can ask for everything after the last event it saw (Event Message Durability (opens in a new tab), Pub/Sub API, checked 2026-11-01).

So a team designs around that window. Then the system on the other side has its big maintenance weekend, the one that happens maybe once a year. It goes down on Friday evening, and it’s meant to be back on Monday morning. Inside 72 hours. Fine.

But this year the upgrade runs long, and it’s back on Tuesday. The events from Friday evening are past the 72 hours, and they’re gone. Nothing failed loudly. Nobody got an error. You’re simply dropping transactions, and you may not find out until someone counts.

The answer isn’t to hope the weekend goes well. It’s to plan, before it happens, what you do when an outage runs past the window. And the plan is usually to layer the integrations. If the events can’t bring you back, a bulk load can. It’s the same idea as a migration cutover in the Data Architect course: a big load first to bring the other system up to date, then smaller incremental loads to close the gap. Once it’s caught up, you switch back from what I call data loading to data synchronization, and the events take over again.

That’s a design decision. It needs a bulk path that already exists, already tested, and an owner who knows when to use it. You don’t want to be building it on Tuesday morning.

Here’s that one arrow again, four times: stopped in every frame, and down a little longer each time.

A minute, an hour, a day, a weekend Four frames of the same arrow. Salesforce on the left publishes events, which it keeps for 72 hours, as of Winter '27; on the right, the other system's subscriber is up but can't be reached, and the arrow between them is stopped, marked with a red X. Each frame shows, on the Salesforce side, how long it's been down and where the data is waiting, and on the other side, how it comes back. Down a minute: a few events wait on the event bus; it reconnects and carries on, and nobody notices. Down an hour: a backlog waits and users see hour-old data, so tell them; it catches up all at once, then users are told it's current. Down a day: a day's volume waits, still inside the 72 hours; the subscriber replays from the last event it saw, on top of today's. Down three or four days, in coral: past 72 hours the oldest events are gone, with no error; it comes back with a bulk load, then incrementals, then back to synchronization. Underneath: plan each one before it happens: a minute, an hour, a day, and the weekend that runs past the window. SALESFORCE THE OTHER SYSTEM Salesforce publishes events, kept 72 hours (Winter '27) Subscriber up, but can't be reached Down a minute a few events wait on the event bus Carries on it reconnects; nobody notices the arrow stops reconnect Plan each one before it happens: a minute, an hour, a day, and the weekend that runs past the window. HOW TO READ IT how it comes back © 2026 Twinshark LLC · stevetecharc.com
Click to enlarge

Down a minute. It reconnects, and nobody notices.

A minute, an hour, a day, a weekend Four frames of the same arrow. Salesforce on the left publishes events, which it keeps for 72 hours, as of Winter '27; on the right, the other system's subscriber is up but can't be reached, and the arrow between them is stopped, marked with a red X. Each frame shows, on the Salesforce side, how long it's been down and where the data is waiting, and on the other side, how it comes back. Down a minute: a few events wait on the event bus; it reconnects and carries on, and nobody notices. Down an hour: a backlog waits and users see hour-old data, so tell them; it catches up all at once, then users are told it's current. Down a day: a day's volume waits, still inside the 72 hours; the subscriber replays from the last event it saw, on top of today's. Down three or four days, in coral: past 72 hours the oldest events are gone, with no error; it comes back with a bulk load, then incrementals, then back to synchronization. Underneath: plan each one before it happens: a minute, an hour, a day, and the weekend that runs past the window. SALESFORCE THE OTHER SYSTEM Salesforce publishes events, kept 72 hours (Winter '27) Subscriber up, but can't be reached Down an hour a backlog waits; users see hour-old data: tell them Catches up all at once; then tell users it's current the arrow stops the backlog arrives Plan each one before it happens: a minute, an hour, a day, and the weekend that runs past the window. HOW TO READ IT how it comes back © 2026 Twinshark LLC · stevetecharc.com
Click to enlarge

Down an hour. A backlog, a burst, and users who need telling, twice.

A minute, an hour, a day, a weekend Four frames of the same arrow. Salesforce on the left publishes events, which it keeps for 72 hours, as of Winter '27; on the right, the other system's subscriber is up but can't be reached, and the arrow between them is stopped, marked with a red X. Each frame shows, on the Salesforce side, how long it's been down and where the data is waiting, and on the other side, how it comes back. Down a minute: a few events wait on the event bus; it reconnects and carries on, and nobody notices. Down an hour: a backlog waits and users see hour-old data, so tell them; it catches up all at once, then users are told it's current. Down a day: a day's volume waits, still inside the 72 hours; the subscriber replays from the last event it saw, on top of today's. Down three or four days, in coral: past 72 hours the oldest events are gone, with no error; it comes back with a bulk load, then incrementals, then back to synchronization. Underneath: plan each one before it happens: a minute, an hour, a day, and the weekend that runs past the window. SALESFORCE THE OTHER SYSTEM Salesforce publishes events, kept 72 hours (Winter '27) Subscriber up, but can't be reached Down an hour a backlog waits; users see hour-old data: tell them Catches up all at once; then tell users it's current Down a day a day's volume waits, still inside the 72 hours Replays from the last event it saw, on top of today's the arrow stops replay Plan each one before it happens: a minute, an hour, a day, and the weekend that runs past the window. HOW TO READ IT how it comes back © 2026 Twinshark LLC · stevetecharc.com
Click to enlarge

Down a day. Still inside the window, so it replays.

A minute, an hour, a day, a weekend Four frames of the same arrow. Salesforce on the left publishes events, which it keeps for 72 hours, as of Winter '27; on the right, the other system's subscriber is up but can't be reached, and the arrow between them is stopped, marked with a red X. Each frame shows, on the Salesforce side, how long it's been down and where the data is waiting, and on the other side, how it comes back. Down a minute: a few events wait on the event bus; it reconnects and carries on, and nobody notices. Down an hour: a backlog waits and users see hour-old data, so tell them; it catches up all at once, then users are told it's current. Down a day: a day's volume waits, still inside the 72 hours; the subscriber replays from the last event it saw, on top of today's. Down three or four days, in coral: past 72 hours the oldest events are gone, with no error; it comes back with a bulk load, then incrementals, then back to synchronization. Underneath: plan each one before it happens: a minute, an hour, a day, and the weekend that runs past the window. SALESFORCE THE OTHER SYSTEM Salesforce publishes events, kept 72 hours (Winter '27) Subscriber up, but can't be reached Down an hour a backlog waits; users see hour-old data: tell them Catches up all at once; then tell users it's current Down a day a day's volume waits, still inside the 72 hours Replays from the last event it saw, on top of today's Down 3 or 4 days past 72 hours the oldest events are gone, with no error A bulk load then incrementals, then back to synchronization the arrow stops load it again Plan each one before it happens: a minute, an hour, a day, and the weekend that runs past the window. HOW TO READ IT how it comes back © 2026 Twinshark LLC · stevetecharc.com
Click to enlarge

Down three or four days. Past the window, the events can’t bring it back; a bulk load can.

Build for Today, Plan for Tomorrow, Plan for Failure

Section titled “Build for Today, Plan for Tomorrow, Plan for Failure”

So this chapter has planted four seeds, and each one grows through the rest of the course.

  • The arrow: decide what it means, and which side starts it. That’s module 3, and chapters 2 and 5.
  • The implied requirements: the fire exits nobody asks for, and the three-year horizon. That’s module 4, and chapter 6.
  • Change: every arrow crosses a boundary between owners, and one side will change. That’s module 5, and it comes back in almost every chapter.
  • Failure: every arrow will stop one day. Know what happens to the data, the users and the recovery, for a minute, an hour, a day and a weekend. That’s this module, and chapters 10, 11 and 13.

I sum up the mindset the same way in every course I teach: build for today, plan for tomorrow, plan for failure. Build what the business needs now. Plan for the volumes and the changes of the next three years. And plan for the night the arrow stops, and for the morning it comes back.

That’s how an integration architect thinks. Now we can start learning the bricks, beginning with the most basic one of all: the client and the server.

What an Architect Decides Here

Section titled “What an Architect Decides Here”
  • For every arrow, how the traffic flows in normal time and what happens to it when it stops.
  • Where the data waits while the arrow is down, and for how long.
  • How users are told the arrow has stopped, and told again when it’s caught up.
  • Whether recovery is automatic or needs a person, and who that person is.
  • What happens after a minute, an hour, a day and three or four days, including the point where data is lost.
  • Which bulk path brings a system back after an outage longer than the event window, and when you switch back to synchronization.

Beyond the exam: what AI changes. An AI agent calling another system fails the same way any client does: the pathway drops, the other side is slow, the answer never comes. The difference is that an agent may simply carry on with what it has. So the same questions apply, and one more: when the arrow is down, what does the agent tell the user?

Explore

Section titled “Explore”

Seuraa minua

© 2023–2026 Twinshark LLC. All rights reserved. · Sisällön käyttö

Privacy notice

SteveTechArc is independent and is not affiliated with, sponsored by, or endorsed by Salesforce, Inc. Salesforce and related marks are trademarks of Salesforce, Inc.

Educational content, shared as is. Not professional, legal or financial advice. Salesforce changes several times a year, so check current Salesforce documentation before you rely on any platform detail.

Oikeudelliset ilmoitukset ovat englanniksi, ja englanninkielinen versio on ratkaiseva.