Zum Inhalt springen

Vor dem Pfeil

Auch in 19 Sprachen

English · Español · 日本語 · Português (Brasil) · Français · Deutsch · Italiano · 한국어 · 简体中文 · Português (Portugal) · 繁體中文 · Bahasa Indonesia · Türkçe · Polski · Tiếng Việt · ไทย · Українська · Español (México) · Français (Canada)

An old hand pump over a well, water pouring from the spout into a bucket.
Mit KI erstelltes Bild

Modul 1 von 4 im Kapitel Integration und Datenbewegung, Kapitel 9 des Kurses Data Architect. Jedes Modul steht für sich. Zurück: das letzte Kapitel, Architektur für Abfragen und Abruf · Weiter: Schlüssel und Schreibhoheit

The question this module answers: What must a data architect know about an integration before anyone draws the arrow? ⟲

Here’s Steve Trucking at the end of Part 1. At Checkpoint 1 I drew what I would architect: the partners’ manifests coming in, status going back out, the latest truck position from FleetLink, status events to Snowflake, delivery counts to the finance ERP. And in the middle of all those arrows, I drew a box and wrote on it: Integration, designed in Part 2. ⟲

This is Part 2. ⟲

Every arrow is a decision Checkpoint 1's landscape for Steve Trucking, in the same places as chapter 6: people on the left, one Salesforce org with Field Service in the middle, the Retail Partner portal and the tracking page below it, the identity provider above, and the integration box, designed in Part 2, to its right. Arrows connect Salesforce, directly or through the integration layer, with Snowflake, FleetLink, the finance ERP, photo storage, the SMS and email provider and the three Retail Partners. One path is picked out in amber: Redrock's manifests come in through the integration layer and are upserted into Salesforce by the partner's key. Underneath: every arrow on the landscape is a decision about data; this module zooms in on one. {"author": "Steve Simpson", "blueprint": "1.0.1", "builder": "standard 1.54.0", "built": "2026-10-30", "id": "DGM-DA-DI-10", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "system-landscape 0.13.0", "theme": "stevetecharc 1.4.0"} ABOVE: IDENTITY, APPEXCHANGE, WEB PEOPLE RIGHT: INTERNAL SYSTEMS BELOW: CUSTOMER CHANNELS BOTTOM RIGHT: PARTNERS AND PROVIDERS Dispatch coordinators 45, inside 9 partner DCs Delivery crews ~700 on 330 trucks Care and office staff 100: service, ops, finance Retail Partner users ~300 at 3 partners Recipients ~1 million a year Corporate IdP SSO for office staff only Retail Partner portal Tracking page INTEGRATION Snowflake status history, analytics FleetLink raw GPS stays here Finance ERP invoices partners, pays crews Photo storage 4–5 million a year, off core RouteDesk archive retired; 8 years of history Phone care agents Email email-to-case SMS and email provider day-before message, with a link Redrock REST API, webhooks Keystone CSV over SFTP Brightwave EDI through a third-party network Salesforce Field Service, one US org a link, no login office staff only calls cases in manifests in status out manifests in status out manifests in status out upserts, partner key status events status events latest position only delivery counts delivery counts photos, from the crews' app day-before, with a link Salesforce a community people identity, apps, web an internal system a customer channel the integration layer starts at the system that opens the connection; the label says what moves a shared certificate: trust, no direct call the arrow we zoom in on Checkpoint 1's landscape. Every arrow on it is a decision about data; this module zooms in on one. © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

Checkpoint 1’s landscape. Every arrow on it is a decision about data; this module zooms in on one. ⟲

I love system landscapes. Even two boxes protect you: put one up on a first call and you’d be surprised how many memories are suddenly jogged. “Wait, where’s my mobile app? Where’s the push of my orders?” But a landscape has a weakness, and it’s the arrows. Look at this one. Every arrow is the same thin line. One of them might carry one record every few minutes, another 500 a minute, another a nightly file of 200,000. One is a request where Salesforce waits for an answer; another is an event Salesforce publishes and forgets. Those aren’t the same architecture. The picture doesn’t show the difference. So before I draw an arrow, I want to know what it means. ⟲

That’s what before the arrow means: the data questions you answer before anyone chooses REST, events, files or middleware. ⟲

„Darum kümmert sich die Integrationsarchitektur“

Abschnitt betitelt „„Darum kümmert sich die Integrationsarchitektur“ “

Maybe. On a large program, someone else may design the APIs, events, transformations and middleware. That’s fine. This chapter is a sampler plate; the Integration Architect course is the full meal. But I absolutely want to know what the integration architect is planning, because the integration is about to touch my data. An API pushing hundreds of records a minute is a different data problem from an hourly bulk load. Those writes can fire automation, recalculate sharing, lock parent records, run duplicate checks and validation, update roll-ups, grow storage and trigger other integrations. The data architect doesn’t have to build the integration, but decides what it’s allowed to do to the data. Understand the traffic that’s coming. ⟲

So let’s pick one arrow and look closely: Redrock’s manifests, coming in through the integration layer and landing in Salesforce. In chapter 4 we zoomed in on an arrow too, and saw the plumbing: where the server listens, how the client authenticates, the protocol and the payload. That’s the integration architect’s view. Here’s the data architect’s view of the same kind of arrow: seven questions, and what each answer does to the data. ⟲

One arrow, seven questions One arrow zoomed in: Redrock, the client, with its order system and REST API, sends manifests in to Salesforce, the server, with an integration layer in front. Below, the seven questions, each with Redrock's answer and what it means for the data. Who starts it: Redrock calls in; Salesforce listens and can't fetch a manifest nobody sent. Which way does the data move: manifests move from Redrock to Salesforce; status goes back on its own arrow. How current: sent a day or two ahead and revised on the day of delivery, so minutes, not seconds; a coordinator must see a change before the truck leaves. How much, and in what shape: about 1,100 deliveries a day, a set per distribution center each evening, then single revisions; about 350,000 Work Orders a year, growing 30% a year. What's the key: Redrock's order and delivery reference, held in an External ID, so a resent manifest updates and never duplicates. Who may write what: Redrock owns what was ordered and where it goes; Salesforce owns the appointment, the crew and the route. Who knows it worked: Redrock sees the round trip, so it resends what failed; Salesforce must say what it rejected, and why. Underneath: seven questions before anyone picks REST, events or a file. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.43.0", "built": "2026-10-27", "id": "DGM-DA-DI-13", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "system-landscape/connection 0.12.0", "theme": "stevetecharc 1.4.0"} manifests in CLIENT Redrock SERVER Salesforce Order system, REST API Integration layer in front FOR REDROCK'S MANIFESTS WHAT IT MEANS FOR THE DATA Who starts it? Redrock calls in. Redrock is the client. Salesforce listens. It can't fetch a manifest nobody sent. Which way does the data move? Manifests move from Redrock to Salesforce. Status goes back on its own arrow, with its own answers. How current? Sent a day or two ahead; revised on the day of delivery. Minutes, not seconds: a coordinator must see a change before the truck leaves. How much, and in what shape? About 1,100 deliveries a day: a set per DC each evening, then single revisions. About 350,000 Work Orders a year, growing 30% a year. What's the key? Redrock's order and delivery reference. An External ID, so a resent manifest updates and never duplicates. Who may write what? Redrock owns what was ordered and where it goes. Salesforce owns the appointment, the crew and the route. Who knows it worked? Redrock sees the round trip, so it resends what failed. Salesforce must say what it rejected, and why. Seven questions before anyone picks REST, events or a file. © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

One arrow, seven questions. None of the answers names a technology. ⟲

Read down the left column. Redrock starts it. The manifests move from Redrock to Salesforce. They’re sent a day or two ahead, but revised on the day. About 1,100 deliveries a day arrive as a set per distribution center each evening, then as single revisions. Redrock’s order and delivery reference identifies each one. Redrock owns what was ordered and where it goes. And Redrock, as the client, is the one that sees whether the call worked. ⟲

Now the right column, which is where the data architect lives. A resent manifest must update, not duplicate. Redrock may change the items and the address, but not the crew. A coordinator must see a revision before the truck leaves the dock. About 350,000 Work Orders a year come down this one arrow, growing 30% a year. Notice what isn’t in either column: REST, Bulk API, platform events, MuleSoft. Choose the mechanism before you understand the movement, and you’re designing backwards. ⟲

The seven questions, for every important arrow: ⟲

  1. Who starts it? ⟲
  2. Which way does the data move? ⟲
  3. Wie aktuell müssen sie sein?
  4. Wie viele Daten, und in welcher Form?
  5. What’s the key? ⟲
  6. Wer darf welche Felder schreiben?
  7. Wer weiß, ob es funktioniert hat, und wer stellt wieder her, wenn nicht?

This module works through the first four, plus what an arrow costs. Module 2 takes the key and the write authority; module 3 takes the last question. ⟲

What does an arrow on a landscape mean? Ask a room and you get two answers. One: who’s the client and who’s the server, who opens the connection. Two: which way the data flows. Both are fair, and they can point in completely opposite directions. A user connects to Salesforce and a sea of data flows back the other way. So you have to decide what your arrows mean. I draw mine from the system that opens the connection, and I put what moves, and which way, in the label. That’s the rule on every landscape in this course. ⟲

Take FleetLink. At Checkpoint 1 we decided Salesforce keeps only each truck’s latest position. That data moves one way: FleetLink to Salesforce. But who starts it? Two designs: ⟲

  • Salesforce asks. Every few minutes a scheduled job in Salesforce calls FleetLink: “Where are my trucks?” Salesforce is the client. The data still flows FleetLink → Salesforce. ⟲
  • FleetLink tells. Whenever a truck moves far enough, FleetLink calls into Salesforce with the new position. FleetLink is the client. The data still flows FleetLink → Salesforce. ⟲

Same data, same direction, different initiator. And it matters, because the initiator carries the load and the responsibility. In the first design, Salesforce spends its own limits making calls, holds FleetLink’s credentials, and has to handle FleetLink being down. In the second, FleetLink spends Salesforce’s API calls, and FleetLink has to notice when Salesforce rejects something. In class I call this the direction of action: who’s starting the action, which can be different from the data flow. ⟲

Same data, different initiator Two designs for FleetLink's latest truck position, Salesforce on the left and FleetLink on the right. First, Salesforce asks: a scheduled job in Salesforce, running every few minutes and tagged "starts it", calls FleetLink with "where are my trucks?", and the positions come back to Salesforce in a navy arrow. Then FleetLink tells: the scheduled job is gone, FleetLink is tagged "starts it", and it sends Salesforce each new position in a navy arrow. The legend says the data always moves from FleetLink to Salesforce. Underneath: same data, same direction, different initiator; whoever starts it carries the load, the credentials and the errors. SALESFORCE FLEETLINK Scheduled job runs every few minutes starts it Salesforce latest position on each truck FleetLink 200,000 GPS readings a day where are my trucks? positions come back Same data, same direction, different initiator. Whoever starts it carries the load, the credentials and the errors. HOW TO READ IT the data: always FleetLink to Salesforce © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

Salesforce asks. Salesforce starts it; the positions flow FleetLink → Salesforce. ⟲

Same data, different initiator Two designs for FleetLink's latest truck position, Salesforce on the left and FleetLink on the right. First, Salesforce asks: a scheduled job in Salesforce, running every few minutes and tagged "starts it", calls FleetLink with "where are my trucks?", and the positions come back to Salesforce in a navy arrow. Then FleetLink tells: the scheduled job is gone, FleetLink is tagged "starts it", and it sends Salesforce each new position in a navy arrow. The legend says the data always moves from FleetLink to Salesforce. Underneath: same data, same direction, different initiator; whoever starts it carries the load, the credentials and the errors. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.43.0", "built": "2026-10-27", "id": "DGM-DA-DI-11", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "cells 0.2.0", "theme": "stevetecharc 1.4.0"} SALESFORCE FLEETLINK Salesforce latest position on each truck FleetLink 200,000 GPS readings a day starts it here's a new position Same data, same direction, different initiator. Whoever starts it carries the load, the credentials and the errors. HOW TO READ IT the data: always FleetLink to Salesforce © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

FleetLink tells. FleetLink starts it; the positions still flow FleetLink → Salesforce. ⟲

At its simplest, it’s still client and server. The client starts the conversation and the server listens. The server knows whether it received, rejected, processed or failed the request. What it can’t see is whether its response made it back. The client can. Only the client sees the whole round trip. That has a big consequence for recovery, which module 3 makes concrete. ⟲

Test knowledge. Separate who initiates an integration from which way the data flows. “Salesforce calls the ERP to get the credit status” is Salesforce-initiated, but the data flows from the ERP into Salesforce. Expect scenarios that describe one and ask about the other. ⟲

The first word I challenge is “real time.” Why? What happens if it’s five seconds old? Thirty seconds, five minutes, an hour, tomorrow morning? There are real real-time requirements, and plenty labeled real time because nobody asked the next question. ⟲

I teach three timings: ⟲

  • Real time is synchronous: the caller waits for the answer. A user can’t take the next step until it comes back. It’s the hardest to build and has the most error handling. ⟲
  • Near real time is anywhere from under a second to a minute, two, maybe five. There’s a big difference between sub-second and one to five minutes, and near real time is a lot cheaper and easier to maintain than real time. ⟲
  • Batch is hourly, daily or weekly. Old school, and often exactly right. ⟲

Now put Steve Trucking’s arrows on that ladder. ⟲

The arrowHow current it needs to beTiming
Status out to Redrock, by webhookAs it happens: Redrock shows it to its own customersNear real time, seconds
Latest truck position, FleetLink → SalesforceA few minutes old is fine for “your truck is about 20 minutes away”Near real time, minutes
Redrock’s manifest revisionsBefore the truck leaves the dockNear real time, minutes
Status events, Salesforce → Snowflake, through the integration layerToday’s dashboards; nobody runs the company from the last secondNear real time or hourly batch
Status file to KeystoneKeystone asked for it each nightBatch, daily
Delivery counts to the finance ERPPartners are invoiced monthlyBatch, daily is plenty

Look at what’s missing: nothing on this landscape needs Salesforce to wait for another system’s answer. That’s not unusual. Real time costs more calls, more events, more processing, more contention and more operational complexity, so I want a reason for every one. If a nightly update meets the need, I’m not embarrassed to use one. Freshness is a requirement, not a default. ⟲

Salesforce’s Well-Architected framework makes the same point from the other side: the mechanism you pick sets how much you can lose and how stale a copy can get. Its Reliability (öffnet in einem neuen Tab) guidance lists replication choices by that measure. Change Data Capture provides “near real-time replication with minimal development effort.” Platform events carry your own payloads, but need publish logic you build. A scheduled Bulk API 2.0 extraction is the simplest, “with an RPO equal to extraction frequency”: a nightly extract can be a day stale, by design. ⟲

Test knowledge. Match the timing to the mechanism. Waiting for an answer (request and reply) needs a synchronous call. Fire and forget from Salesforce: platform events (outbound messages are the older option). Keeping another system in step with Salesforce changes: Change Data Capture. Large scheduled loads: the Bulk API. Watch for scenarios that say “immediately” when the business only needs minutes. ⟲

There’s a way to get near real time that looks simple: ask. A process outside Salesforce calls in on a schedule. “Has anything changed? Has anything changed? Has anything changed?” That’s polling. Once a minute doesn’t sound horrendous, until you do the math: 60 times 24 is 1,440 calls a day, for one question, from one system. Every call uses an API request from your allocation and puts its own load on Salesforce, whether or not anything changed. ⟲

Say the finance ERP wants Steve Trucking’s delivery counts, and someone builds a job that asks Salesforce every five minutes. That’s 288 calls a day and over 105,000 a year, for numbers the business invoices once a month. Almost every one of those calls comes back with nothing that matters. Well-Architected’s Resource and Cost Optimization (öffnet in einem neuen Tab) pillar has a name for that: an integration that polls every five minutes and finds nothing “is pure waste.” A nightly batch serves finance better, at a fraction of the calls. ⟲

Now take Snowflake, which wants every status change, and soon. First question: who asks? Not Snowflake. A database stores; a process calls. Tables don’t start conversations. Something has to wake up, make the call, handle the error, retry and write the result. Salesforce can be that process for its own outbound calls, but Salesforce doesn’t poll itself on another system’s behalf. So when neither end runs the process you need, you add a third piece of software: middleware, the integration layer. ⟲

Look closely and even the “built-in” options are that third piece. Snowflake’s own Salesforce connector runs on Snowflake Openflow, which Snowflake describes as “an integration service” built on Apache NiFi; it polls Salesforce’s Bulk API 2.0 on a schedule, from every minute to once a day (Openflow (öffnet in einem neuen Tab) and its Salesforce Bulk API connector (öffnet in einem neuen Tab), checked 2026-10-27). Or you write a stored procedure that calls out through Snowflake’s external network access (öffnet in einem neuen Tab) and schedule it as a task. Either way it’s a process, with its own credentials, schedule and failures, and someone has to own it. (When that connector’s page says it “supports Change Data Capture,” it means its own change detection by timestamp, not Salesforce’s change events.) Can it be done? Yes. Would I, and should I? That’s the architecture decision, not a feature of the database. ⟲

So the integration layer can poll: every minute, “anything changed?”, 1,440 calls a day, most finding nothing. Or Salesforce can tell it. I could build the telling myself: a record-triggered flow on the Work Order, query the fields, put them on a platform event, and the integration layer subscribes and writes them to Snowflake. It works. Then someone adds a field to the Work Order. Now I have to add it to the flow’s query, add it to the platform event, and change the subscriber and Snowflake’s table too. That’s code to maintain, forever, keeping up with the object’s definition. ⟲

Or I turn on Change Data Capture for the Work Order. No flow and no event to maintain: Salesforce publishes a change event for every create, update, delete and undelete, with the changed fields, and a new field is picked up automatically. Work Order and Service Appointment both support it, checked against the Object Reference (öffnet in einem neuen Tab), Winter ‘27. ⟲

But count the zeros first. About 3,300 deliveries a day, with about 8 status changes each, is roughly 26,400 status changes a day, before any other field on the Work Order changes. CDC sends every change, not just status. And event delivery has a daily allocation: without an add-on, 50,000 delivered events in a rolling 24 hours for Performance and Unlimited Editions and 25,000 for Enterprise Edition, shared with high-volume platform events and counted per subscriber, so two subscribers use it twice (Change Data Capture Allocations (öffnet in einem neuen Tab), checked 2026-10-27). When I first taught this, the warning was “you’re going to get a tsunami of data.” Today you can filter the stream (öffnet in einem neuen Tab) in a custom channel, so the subscriber receives only the status changes it asked for. And the subscriber is middleware too: receiving change events means holding a subscription open to Salesforce’s Pub/Sub API, a job for an integration layer such as MuleSoft or a Kafka connector, not for the warehouse. Event-driven is better than polling for this arrow, but it still has a number of zeros. ⟲

Asking every minute, or told on change Three areas in a row: Salesforce's Work Orders on the left, an integration layer in the middle, Snowflake on the right, keeping status history and storing only what it's sent. First, the integration layer polls: a polling job, tagged "starts it", asks the Work Orders "anything changed?" every minute, 1,440 calls a day, and writes what it finds to Snowflake, mostly nothing. Then it listens: Change Data Capture is on for Work Orders, filtered to status; the integration layer's subscriber, tagged "starts it", subscribes once and listens, receives only status changes in a navy arrow, and writes them to Snowflake. The legend says delivered events count against a daily allocation without an add-on: 50,000 for Performance and Unlimited Editions or 25,000 for Enterprise Edition, per subscriber, as of Winter '27. Underneath: a database stores; a process calls. Asking on a clock spends calls on nothing; listening for change spends events, so count them too. SALESFORCE INTEGRATION LAYER SNOWFLAKE Work Orders ~26,400 status changes a day Polling job every minute: 1,440 calls a day starts it Snowflake status history; stores what it's sent anything changed? writes what it finds: mostly nothing A database stores; a process calls. Asking on a clock spends calls on nothing; listening for change spends events, so count them too. HOW TO READ IT delivered events count against a daily allocation: 50,000 (Performance, Unlimited) or 25,000 (Enterprise), per subscriber, without an add-on (Winter '27) © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

The integration layer asks on a clock: 1,440 calls a day, most of them finding nothing. ⟲

Asking every minute, or told on change Three areas in a row: Salesforce's Work Orders on the left, an integration layer in the middle, Snowflake on the right, keeping status history and storing only what it's sent. First, the integration layer polls: a polling job, tagged "starts it", asks the Work Orders "anything changed?" every minute, 1,440 calls a day, and writes what it finds to Snowflake, mostly nothing. Then it listens: Change Data Capture is on for Work Orders, filtered to status; the integration layer's subscriber, tagged "starts it", subscribes once and listens, receives only status changes in a navy arrow, and writes them to Snowflake. The legend says delivered events count against a daily allocation without an add-on: 50,000 for Performance and Unlimited Editions or 25,000 for Enterprise Edition, per subscriber, as of Winter '27. Underneath: a database stores; a process calls. Asking on a clock spends calls on nothing; listening for change spends events, so count them too. {"author": "Steve Simpson", "blueprint": "1.0.1", "builder": "standard 1.57.1", "built": "2026-10-30", "id": "DGM-DA-DI-14", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "cells 0.5.0", "theme": "stevetecharc 1.4.0"} SALESFORCE INTEGRATION LAYER SNOWFLAKE Work Orders Change Data Capture on, filtered to status Subscriber subscribes once, then listens starts it Snowflake status history; stores what it's sent status changes only writes them A database stores; a process calls. Asking on a clock spends calls on nothing; listening for change spends events, so count them too. HOW TO READ IT delivered events count against a daily allocation: 50,000 (Performance, Unlimited) or 25,000 (Enterprise), per subscriber, without an add-on (Winter '27) © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

The integration layer listens instead. Snowflake receives either way: a database stores; a process calls. ⟲

Salesforce’s own Best Practices for Deployments with Large Data Volumes (öffnet in einem neuen Tab) gives an older form of the same advice for keeping another system in step with Salesforce. Use the SOAP API’s getUpdated() and getDeleted() calls “to sync an external system with Salesforce at intervals greater than 5 minutes,” and “use the outbound messaging feature for more frequent syncing” (Extracting Data from the API (öffnet in einem neuen Tab), Winter ‘27, checked 2026-10-27). The idea still holds. On a schedule, ask only for what changed, and don’t forget what was deleted. When it has to be sooner than that, let Salesforce tell you. The mechanism has moved on: outbound messaging gets no further investment (module 3), so today the telling is usually Change Data Capture or platform events. ⟲

Test knowledge. Pulling changes out of Salesforce on a schedule: getUpdated() and getDeleted() return what changed and what was deleted in a time window, for intervals over 5 minutes. More often than that, let Salesforce send the change (Salesforce’s guide names outbound messaging; today, Change Data Capture or platform events). Expect a scenario where a full extract every few minutes is the wrong answer. ⟲

Die Anzahl der Nullen gilt auch für Integrationen

Abschnitt betitelt „Die Anzahl der Nullen gilt auch für Integrationen “

Then: how much? Not “is it a lot?” Give me the zeros. And not just the daily number; tell me how it arrives. An integration team wants three numbers: transactions per second, bytes per second, and the activity window, the hours the traffic actually runs. You may have 24 hours in a day, but the data doesn’t arrive at 2 a.m. Steve Trucking’s crews work about twelve hours, so that’s the window. ⟲

Steve Trucking’s trafficA dayPer second, in a 12-hour windowBytes
Truck positions from FleetLinkabout 200,000 readingsabout 4.6tiny: a position and a time
Delivery status changesabout 26,400about 0.6small
Delivery photosabout 13,200 photosabout 0.3heavy: at about 4 MB each, 53 GB a day, 1.2 MB every second

Look at the photos: under one transaction every three seconds, and still the heaviest arrow on the landscape. Positions are the opposite: the most transactions, almost no bytes. A lot of data with low transactions per second, or high transactions per second with little data: they have completely different behavioral profiles. Count both. And plan for the busiest season, not the average day. ⟲

I also think of integration traffic in three shapes: ⟲

  • Singles. One record at a time: a status change, an order. Exactly right for transactional activity. But send 100,000 records one at a time as fast as you can, and a reasonable pattern becomes a machine gun: every call has overhead, may run automation and competes for resources. ⟲
  • Sets. Bundle them, tens or hundreds per call, and let the receiver process sensible groups. Redrock’s evening manifest is a set. For many integrations, the middle ground is the best fit. ⟲
  • Bulk. Hundreds of thousands or millions of records, like a historical load. Don’t pretend two million records are two million little real-time transactions. Salesforce has the Bulk API for bulk work: in class I call it the 18-wheeler next to the bank teller. ⟲

It’s a teaching model, not a limits table, but it makes you respect the zeros. ⟲

Salesforce’s guide does draw one line in numbers: “Any data operation that includes more than 2,000 records is a good candidate for Bulk API 2.0”; fewer than that should be “bulkified” synchronous calls, in REST (Composite, for example) or SOAP. And when you update, “send only fields that have changed” (Loading Data from the API (öffnet in einem neuen Tab), checked 2026-10-27). Sending every field every time is the integration version of the machine gun. Chapter 10 has the rest of that guide’s loading advice. ⟲

Singles, sets and bulk Three panels side by side. Singles, one record a call: at Steve Trucking, a delivery status change or a truck's new position; fits transactional events as they happen; watch out for sending 100,000 of them as fast as you can, a machine gun. Sets, tens or hundreds a call: Redrock's evening manifest, about 370 deliveries a distribution center; fits most integrations, sensible groups the receiver can process; watch out for one bad record in the set and what happens to the rest. Bulk, hundreds of thousands or millions: RouteDesk's eight years, about 5 million deliveries; fits historical loads and big extracts through the Bulk API; watch out for locking, automation and sharing at volume. Underneath: a teaching model, not platform limits; it makes you respect the zeros. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.43.0", "built": "2026-10-27", "id": "DGM-DA-DI-12", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "comparison 0.4.0", "theme": "stevetecharc 1.4.0"} Singles one record a call STEVE TRUCKING A delivery status change A truck's new position FITS Transactional events, as they happen WATCH OUT 100,000 of them as fast as you can: a machine gun Sets tens or hundreds a call STEVE TRUCKING Redrock's evening manifest: about 370 deliveries a DC FITS Most integrations: sensible groups the receiver can process WATCH OUT One bad record in the set: what happens to the rest? Bulk hundreds of thousands or millions STEVE TRUCKING RouteDesk's 8 years: about 5 million deliveries FITS Historical loads and big extracts, through the Bulk API WATCH OUT Locking, automation and sharing at volume A teaching model, not platform limits: it makes you respect the zeros. © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

Singles, sets and bulk. A teaching model, not platform limits. ⟲

Test knowledge. For hundreds of thousands or millions of records, the answer is the Bulk API, not many single REST calls. Salesforce’s line: more than 2,000 records is a good candidate for Bulk API 2.0; fewer, bulkified synchronous calls. When API limits bite on a set of related records of different objects, look at the composite resources, which carry several operations in one call. When updating, send only the fields that changed. ⟲

Angenommen, ein externes System erstellt 500 Datensätze pro Minute. Das sind 30.000 pro Stunde, 720.000 pro Tag, wenn es durchläuft: Hunderte Millionen Operationen pro Jahr. Vielleicht läuft es nicht ständig; vielleicht sind die meisten Aktualisierungen. Gut, deshalb will ich die Zahlen. Denn dann beginnen die Datenfragen. Welches Objekt treffen sie? Wie ist die Eigentümerschaft verteilt? Teilen sie sich Elterndatensätze? Welche Automatisierung läuft, welche Freigabe wird neu berechnet, entstehen Verlaufsdatensätze, gibt es Rollups, wie lange ist die Aufbewahrung? Könnten wir einen Hotspot erzeugen? Brauchen wir jeden einzelnen dieser Datensätze in Salesforce?

An integration can create a large-data-volume problem on its own. “Every time FleetLink sends a GPS position, we’ll create a Salesforce record.” That’s 200,000 a day: 73 million records a year. Before I care whether that’s REST, events or middleware, I ask: why would we put 73 million GPS readings a year into the transactional data model? At Checkpoint 1 we decided not to; only the latest position matters to the org. The integration mechanism doesn’t answer that question. It’s a data architecture decision. Sometimes understanding the integration redesigns the model; sometimes it redesigns the integration; sometimes it decides not to move the data at all. ⟲

Jede Kopie hat Folgen: Speicher, Aktualität, Synchronisierung, Sicherheit, Datenschutz, Stammdatenführung, Löschung, Abgleich. Kopieren Sie Kundendaten aus dem ERP nach Salesforce, und Sie haben eine weitere Darstellung, die gesteuert werden muss. Wem gehört sie? Wie veraltet darf sie werden? Was, wenn jemand sie in Salesforce ändert, oder das ERP fünf Minuten später einen anderen Wert sendet, oder die Kundin vergessen werden will, oder ein Agent sie aktualisiert? Der Pfeil trägt nicht nur Bytes. Er schafft einen weiteren Ort, an dem Unternehmensdaten gesteuert werden müssen.

Two of the seven questions get the next module. What’s the key? If Redrock resends delivery 8472914, does Salesforce create, update, reject or link? Without a stable key, retries become duplicates and reconciliation becomes guesswork; that’s why Checkpoint 1 gave every Work Order a key that matches the partner’s order. Who may write? If the finance ERP owns billing terms and Salesforce owns the appointment and the crew, an integration doesn’t mean “the ERP may update the record.” It means “the ERP may update these fields.” That’s chapter 6’s authority map made executable. A system-of-record diagram that doesn’t change write behavior is decoration. ⟲

It will. Here’s our landscape again, the same places, with the arrow we zoomed in on. Tonight’s Redrock manifest, 370 deliveries, reaches the integration layer, and the call into Salesforce times out. ⟲

The arrow fails The same Steve Trucking landscape as before, in the same places: people on the left, Salesforce in the middle with the Retail Partner portal and the tracking page, the identity provider above, the integration layer to the right, and arrows to Snowflake, FleetLink, the finance ERP, photo storage, the SMS and email provider and the three Retail Partners. Redrock's manifests come into the integration layer, but the arrow from the integration layer into Salesforce is red with an X in a circle and the label "timed out: saved or not?". The legend says red with an X means the call failed. Underneath: tonight's Redrock manifest, 370 deliveries, times out on its way into Salesforce. Did it save? All, none or some? And what happens when Redrock sends it again? {"author": "Steve Simpson", "blueprint": "1.0.1", "builder": "standard 1.54.0", "built": "2026-10-30", "id": "DGM-DA-DI-15", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "system-landscape 0.13.0", "theme": "stevetecharc 1.4.0"} ABOVE: IDENTITY, APPEXCHANGE, WEB PEOPLE RIGHT: INTERNAL SYSTEMS BELOW: CUSTOMER CHANNELS BOTTOM RIGHT: PARTNERS AND PROVIDERS Dispatch coordinators 45, inside 9 partner DCs Delivery crews ~700 on 330 trucks Care and office staff 100: service, ops, finance Retail Partner users ~300 at 3 partners Recipients ~1 million a year Corporate IdP SSO for office staff only Retail Partner portal Tracking page INTEGRATION Snowflake status history, analytics FleetLink raw GPS stays here Finance ERP invoices partners, pays crews Photo storage 4–5 million a year, off core RouteDesk archive retired; 8 years of history Phone care agents Email email-to-case SMS and email provider day-before message, with a link Redrock REST API, webhooks Keystone CSV over SFTP Brightwave EDI through a third-party network Salesforce Field Service, one US org a link, no login office staff only calls cases in manifests in status out manifests in status out manifests in status out timed out: saved or not? status events status events latest position only delivery counts delivery counts photos, from the crews' app day-before, with a link Salesforce a community people identity, apps, web an internal system a customer channel the integration layer starts at the system that opens the connection; the label says what moves red with an X: the call failed a shared certificate: trust, no direct call Tonight's Redrock manifest, 370 deliveries, times out on its way into Salesforce. Did it save? All, none or some? And what happens when Redrock sends it again? © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

The arrow fails, on the whole landscape. ⟲

Now zoom in on that one arrow. Leave everything else behind: Salesforce on the left, the integration layer on the right, and the call between them. ⟲

One failed arrow, zoomed in The failed arrow from the landscape, zoomed in: Salesforce's Work Orders on the left, the integration layer on the right. First, the integration layer, tagged "starts it", sends tonight's Redrock manifest of 370 deliveries into Salesforce, and the call times out: the arrow is red with an X. Next, what each side knows: the Work Orders card, tagged "only Salesforce knows", asks whether all 370 saved, none, or 367 with 3 rejected; the integration layer's card says "no answer: did it work?". Last, the integration layer sends it again, and two outcomes appear under Salesforce: in green, with Redrock's reference as the key, the resend upserts and there are still 370 Work Orders; in coral, without a key, there could be up to 740 Work Orders for 370 deliveries. Underneath: the X says something broke, not what state the data is in; a resend is safe only when every record has a key. SALESFORCE INTEGRATION LAYER Work Orders Redrock's deliveries for tomorrow Redrock's manifest tonight's: 370 deliveries starts it timed out The X says something broke, not what state the data is in. A resend is safe only when every record has a key. HOW TO READ IT red with an X: the call failed; coral: what goes wrong what the key protects © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

One arrow, one timeout. ⟲

Before you read on, stop and answer it. Did the manifest save? There are three honest answers. All of it: Salesforce saved all 370 Work Orders, and only the reply was lost on the way back. None of it: the call never finished. Some of it: 367 saved and 3 were rejected by a validation rule, and nobody has read the errors yet. Salesforce knows which one happened. The integration layer, the client, only knows it didn’t get an answer. ⟲

One failed arrow, zoomed in The failed arrow from the landscape, zoomed in: Salesforce's Work Orders on the left, the integration layer on the right. First, the integration layer, tagged "starts it", sends tonight's Redrock manifest of 370 deliveries into Salesforce, and the call times out: the arrow is red with an X. Next, what each side knows: the Work Orders card, tagged "only Salesforce knows", asks whether all 370 saved, none, or 367 with 3 rejected; the integration layer's card says "no answer: did it work?". Last, the integration layer sends it again, and two outcomes appear under Salesforce: in green, with Redrock's reference as the key, the resend upserts and there are still 370 Work Orders; in coral, without a key, there could be up to 740 Work Orders for 370 deliveries. Underneath: the X says something broke, not what state the data is in; a resend is safe only when every record has a key. SALESFORCE INTEGRATION LAYER Work Orders All 370 saved? None? 367 saved, 3 rejected? only Salesforce knows Redrock's manifest no answer: did it work? starts it timed out The X says something broke, not what state the data is in. A resend is safe only when every record has a key. HOW TO READ IT red with an X: the call failed; coral: what goes wrong what the key protects © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

Salesforce knows what it saved. The integration layer only knows it heard nothing back. ⟲

Now the integration layer does what clients do: it sends the manifest again. If every Work Order is matched by Redrock’s order and delivery reference, the resend updates what’s there and creates only what’s missing. If it isn’t, and the first call had saved, Steve Trucking now has 740 Work Orders for 370 deliveries, and two crews may be sent to the same door. Same timeout, very different data. That’s why the key comes before the mechanism (module 2), and why somebody has to own knowing whether it all arrived (module 3). ⟲

One failed arrow, zoomed in The failed arrow from the landscape, zoomed in: Salesforce's Work Orders on the left, the integration layer on the right. First, the integration layer, tagged "starts it", sends tonight's Redrock manifest of 370 deliveries into Salesforce, and the call times out: the arrow is red with an X. Next, what each side knows: the Work Orders card, tagged "only Salesforce knows", asks whether all 370 saved, none, or 367 with 3 rejected; the integration layer's card says "no answer: did it work?". Last, the integration layer sends it again, and two outcomes appear under Salesforce: in green, with Redrock's reference as the key, the resend upserts and there are still 370 Work Orders; in coral, without a key, there could be up to 740 Work Orders for 370 deliveries. Underneath: the X says something broke, not what state the data is in; a resend is safe only when every record has a key. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.43.0", "built": "2026-10-27", "id": "DGM-DA-DI-16", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "cells 0.2.0", "theme": "stevetecharc 1.4.0"} SALESFORCE INTEGRATION LAYER Work Orders All 370 saved? None? 367 saved, 3 rejected? only Salesforce knows Redrock's manifest no answer: did it work? starts it With Redrock's reference as the key the resend upserts: still 370 Work Orders Without a key up to 740 Work Orders for 370 deliveries sends it again The X says something broke, not what state the data is in. A resend is safe only when every record has a key. HOW TO READ IT red with an X: the call failed; coral: what goes wrong what the key protects © 2026 Twinshark LLC · stevetecharc.com
Zum Vergrössern klicken

The resend. Safe with a key; duplicates without one. ⟲

A timeout is only one of the common failures. A malformed payload, an expired token, a validation rule rejecting a record, a server down, record 7,428 of 20,000 failing, an event delivered twice. Here integration and data architecture become inseparable. The integration architect thinks about retries, queues, replay, error handling and monitoring. The data architect thinks about duplicates, missing records, partial state, stale values, broken relationships and the wrong system writing. Two views of the same failure. ⟲

So for every arrow I ask the same things. On every outbound arrow: where do I log, what do I tell the user, and how do I retry? On every inbound arrow: how do they recover? Errors will happen. ⟲

Und „Die API hat 200 zurückgegeben“ ist keine Datenqualität. Die Schnittstelle hat funktioniert. Sind alle 10.000 geplanten Datensätze angekommen, oder 9.997? Wurden drei abgelehnt? Hat eine Wiederholung etwas dupliziert? Haben sich die richtigen Datensätze geändert? Hat eine Transformation einen Wert verfälscht? Haben wir Felder überschrieben, die uns nicht gehören? Deshalb passt meine wiederkehrende Regel hier so gut: Vertrauen, aber überprüfen. Modul 3 macht sie konkret.

Beyond the exam: what AI changes. An agent looking up a delivery in another system is an integration. An agent updating Salesforce through an API, or calling Salesforce’s hosted MCP servers, is an integration; an agent retrying after a failure is an integration retry. The caller is new; the questions aren’t. Who starts it? The agent, on behalf of a user, with that user’s permissions. How much? An agent can ask the same question many times a day, so the number of zeros grows with every conversation (chapter 16’s tireless assistant (coming soon)). Treat every agent read and write path as an arrow, with identity, authority, volume, failure handling and observability. ⟲

  • What every important arrow means: who starts it, which way the data moves, how current it must be, and its volume and shape. ⟲
  • Whether “real time” is a requirement or a habit, and which timing each arrow really needs. ⟲
  • Which process runs each arrow (Salesforce, the other system, or middleware), whether it polls or listens for change, and what its events cost in allocations. ⟲
  • Ob die Integration das Datenmodell durch Automatisierung, Sperren, Freigabe, Speicher oder Volumenwachstum unter Druck setzt.
  • Ob die Daten überhaupt nach Salesforce gelangen sollten.
  • Welchen stabilen Schlüssel und welche Schreibautorität die Integration braucht, bevor ein Mechanismus gewählt wird.
  • Wer genug vom Austausch sieht, um den Erfolg zu überprüfen und für die Wiederherstellung zuständig zu sein.

Diagrams make systems look clean. Real integrations are conversations between systems that move data under rules and fail in specific ways. You don’t have to build each one, but once the arrow starts moving data, the integration design becomes part of your data architecture. So before REST, Bulk API, events or middleware, zoom in on the arrow and know what it means. Then design it. ⟲