Moving Data at Scale

Chapter 10 of Integration Architect, Part 2: Designing Integrations That Work (Design). Previous: Salesforce as an API Consumer · Next: Events.

Volume changes everything. A few hundred records go through the teller window: the REST API, 200 at a time, an answer for every one, right away. A million records would mean 5,000 trips to the back of the line. For those, Salesforce has a loading dock at the back of the building: the Bulk API. You back the truck up, load it, ring the buzzer, and come back for the results.

This chapter is about the dock, from the program’s side: the calls it makes, the choices that make a load fail, and the checks that prove it worked. It’s not about designing a one-time migration. The Data Architect course does that in its Migration and Synchronization chapter, and this chapter links to it rather than repeating it. An integration drives the truck every night, for years, with nobody watching.

Steve Trucking runs through every module: Keystone’s nightly status file, the weekly export to Snowflake, Redrock’s evening manifests at the teller window, and the RouteDesk migration’s cutover weekend.

The ideas in this chapter

Count the records in one trip. Over about 2,000, Salesforce points you to the dock; under it, the bulkified synchronous calls. Then ask what the sender needs back: an answer now, or results later.
A job is five calls, and its state tells you the next one. Open, upload, “upload complete”, “are we there yet?”, results. Nothing runs until you ring the buzzer, and for loads, you still ask; query jobs can now tell you, in a Beta.
Bulk API 2.0 is the default; 1.0 is for what only it does. Serial mode, your own batch size, XML and attachments, manual PK chunking. And every 1.0 program needs its sign-in planned before SOAP login() retires in Summer ‘27.
Parallel is faster, until two batches want the same parent. Saving a child locks its parent, so sort by parent, count every writer in the window, and keep serial mode on 1.0 as the last lever.
A query job’s work is in the download. One call opens it; the results come set by set, or five at a time. Some SOQL won’t run, and on 2.0 the chunking is automatic.
Complete isn’t the same as all there. Three result files have to add up to what you uploaded, every key once. Sort the failures, and upsert on your own key so a resend can’t duplicate.
Size the truck at year three, not today. Match the dock’s limits against your numbers at year three and the org’s shared allowance; change the door only where the count crosses the line. And check which dock you’re at: Data 360’s bulk door has very different numbers.
A copy kept in sync starts from the copy, not the clock. The date goes in the query on SystemModstamp, the window overlaps on purpose, and the deletes need a plan of their own.

The path through this chapter

Eight modules, each standing on its own.

The Bank Teller and the 18-Wheeler. Two doors into the same bank; always a call in; the tools that drive the truck; Steve Trucking’s feeds, counted.
A Job, Call by Call. The five calls and the job’s states; “are we there yet?” in 2026; your lab bench.
Two Docks: 2.0 Now, 1.0 for What Only It Does. Salesforce has picked a side; the same job two ways; the four things only 1.0 does; the sign-in deadline.
Parallel and Locking. What parallel means on 2.0; inside the job and outside it; sort by parent; what you’ll see when it bites; the levers, in order.
Bulk Query. A million records out; three calls and a download; the SOQL it won’t run; chunking; its constraints; your lab bench.
Did It All Arrive?. Three files, one count; sorting the failures; the safe resend; keeping the evidence; proving an export arrived.
Choosing the Size of the Truck. Salesforce’s line; the dock’s own limits; Steve Trucking at year three; the tools’ settings; bulk as a pattern, not one product.
Keeping a Copy in Sync. Where the date goes; SystemModstamp; one run in four steps; deletes three ways; whether it should be a copy at all.

Chapter 11 takes the event doors: the push that replaces “are we there yet?”, and the stream that keeps a copy fresher than any schedule.

When you’ve finished, check yourself with the chapter study guide: a checklist for the exam, the habits beyond it, the words to know, and a few questions.

What an architect decides here

The door for every feed, by the records in one trip and whether the sender needs the answer now, checked again at year three.
Which dock each bulk job uses, 2.0 unless it needs what only 1.0 does, and how every 1.0 program signs in after Summer ‘27.
Who else is writing in the window, how the files are sorted, and when a hot parent needs a change in the data model.
The proof for every run: the count check, the failures sorted, the key that makes a resend safe, the evidence kept, and the receiver’s acknowledgment for an export.
The allowance: the dock’s limits and the org’s shared API calls, for every feed together, with the release you checked.
For every copy of Salesforce data: whether it should be a copy at all, the mark, the overlap, the deletes, and when to rebuild it from scratch.

Beyond the exam: an AI assistant will happily write the script that opens the job, uploads the file and downloads the results. What it won’t decide for you is whether the three files add up, who else is writing at 2 a.m., or what the copy misses when a formula changes. Those decisions are this chapter.

Explore

Salesforce Developers, Bulk API 2.0 and Bulk API Developer Guide (opens in a new tab): both docks, in Salesforce’s own reference.
Salesforce Developers, Bulk API and Bulk API 2.0 Limits and Allocations (opens in a new tab): the dock’s limits, and Salesforce’s 2,000-record line.
Salesforce Architects, Data Integration decision guide (opens in a new tab): Salesforce’s own guide to choosing among the ways to move data.

Next: Events