Did It All Arrive?

Module 6 of 8 in the chapter Moving Data at Scale, chapter 10 of the Integration Architect course. Each module stands on its own. Previous: Bulk Query · Next: Choosing the Size of the Truck

The question this module answers: The job says it’s complete. How does your program know every record arrived?

Complete Isn’t the Same as All There

When I demo a bulk load in class, the last step is my favourite: download the successes, and there’s the new record with its Salesforce ID. Then download the failures. That’s step five, and it’s two downloads, not one.

A job in the JobComplete state has told you that Salesforce finished working on it. It hasn’t told you that every row made it. Some rows can fail while the rest are saved, and the saved ones stay saved: a failed job’s “data that was successfully processed isn’t rolled back” (Create a Job (opens in a new tab), Bulk API 2.0 and Bulk API Developer Guide, Winter ‘27, checked 2026-11-01). On a nightly feed, nobody is watching the screen. If the program doesn’t check, nobody does.

Three Files, Not One

An ingest job gives you three result files, each a CSV with your original columns, plus Salesforce’s own columns at the front (same guide, checked 2026-11-01):

• Successful results (.../successfulResults/). Every row that was saved, with sf__Id, the Salesforce record ID, and sf__Created, which “indicates if the record was created.” On an upsert, that tells you which rows were new and which were updates (Get Job Successful Record Results (opens in a new tab)).
• Failed results (.../failedResults/). Every row that was processed and hit an error, with sf__Error, the “error code and message” (Get Job Failed Record Results (opens in a new tab)).
• Unprocessed records (.../unprocessedrecords/). Rows Salesforce never got to, because the job was stopped or failed partway. Salesforce makes the distinction itself: “Unprocessed rows are not the same as failed rows. Failed rows are processed but encounter an error during processing” (Get Job Unprocessed Record Results (opens in a new tab)).

Two notes appear on all three pages, and both belong in your code. First: “The order of records in the response is not guaranteed to match the ordering of records in the original job data.” Row 50 of your upload isn’t row 50 of the results. Match the rows by your own key, the one you sent in the file. Second: “Results are not recorded for batches that exceed the daily batch allocation.” A job that ran past the allowance can come back with results missing. So check the allowance before a big job, not after (module 7).

Count In, Count Out

So here’s the check I want in every bulk program. Count the rows you uploaded. Count the rows in the three result files. They have to be the same number.

Click to enlarge

An example night on RouteDesk’s Work Order wave: the three files add up to the upload.

If they add up, every row is accounted for: saved, failed with a reason, or never tried. If they don’t add up, something you can’t see happened, and the program stops and tells a person. It doesn’t carry on to the next wave, and it doesn’t start the next night’s run. In class, the question about a load scheduled every 15 minutes was this one: what if the last one isn’t finished? Check that the previous run is complete, and accounted for, before you start the next.

Then go one step further: match by key. Every partner key you uploaded should appear exactly once across the three files. A key that’s missing, or appears twice, is a problem even when the totals match.

Sort the Failures

A failed row isn’t one problem. It’s a reason, in sf__Error, and the reasons need different answers. Back when I’d load through the synchronous API, a big part of the work was going through each answer to see which records succeeded and which hit a validation error. The bulk job hands you the same answers in one file. Sort them:

• Try again later. Lock contention, UNABLE_TO_LOCK_ROW: the row was fine; the timing wasn’t. Resubmit those rows in a new job when the parents are quiet (module 4).
• Fix the data first. A validation rule (FIELD_CUSTOM_VALIDATION_EXCEPTION), a required field (REQUIRED_FIELD_MISSING), a bad value. Sending the same row again fails again. These go back to whoever owns the data, with the error.
• Fix the order. A child whose parent isn’t there yet. That’s a load-order problem: the parent’s wave comes first.
• Stop and look. An error you haven’t seen before. Don’t guess at it at 2 a.m.

Watch the rate, not just the count. Twenty failures out of a hundred is a broken feed; twenty out of two hundred thousand might be a normal night. The Data Architect course makes that case in Loading at Volume.

Make the Resend Safe: Upsert on Your Key

Here’s the night that worries me. The program uploads the file, the job runs, and then the program crashes before it reads the results. When it restarts, it doesn’t know which rows were saved. Does it send them all again?

With an insert, a resend makes duplicates: every row that was saved the first time is created a second time. With an upsert on an External ID, a resend is safe. Salesforce describes upsert’s three outcomes: no match, and “a new record is created”; one match, and “the record is updated”; several matches, and “a 300 error is reported” with nothing created or updated (Insert or Update (Upsert) a Record Using an External ID (opens in a new tab), REST API Developer Guide, API 68.0, checked 2026-11-01). On a bulk job, you name the field in externalIdFieldName when you open the job, and every row carries its value.

RouteDesk’s Work Orders upsert on Partner_Delivery_Key__c, the partner key, marked unique in Salesforce. Send a row twice, and the second one updates the first. That’s what makes a resend safe, and it’s what lets Redrock resend a manifest on cutover weekend without making a second delivery. The Data Architect course takes this idea all the way down in Safe to Retry.

Upsert has a cost: Salesforce’s own guidance ranks it slower than insert (Loading at Volume has the details). For a one-time load that knows which rows are new, insert those and upsert the rest. For a feed that runs every night, unattended, I’d pay for upsert.

Keep the Evidence

The results are kept for seven days, then deleted with the job. So the program downloads all three files every run and keeps them, with the job ID, the counts, the retries number, and how long it took. When someone asks in three weeks what happened to one delivery, you want to answer from the files, not from memory.

And the successful results carry the Salesforce IDs. Write them back to wherever the data came from, beside the key that produced each one. Then the next wave can find its parents, and a reload can send just the rows it needs to.

The Other Direction: Did the Export Arrive?

A bulk query doesn’t have failed rows, but it can still come up short. A program that stops halfway through the download, a file that’s cut off on the way, a set that’s skipped. So the export gets its own count:

1. Add up Sforce-NumberOfRecords across every set you downloaded (module 5), and compare it with the rows you wrote to the file.
2. Ask Salesforce for the number another way. Bulk queries won’t run COUNT(), but the REST API will: SELECT COUNT() FROM WorkOrder with the same WHERE clause and the same fixed time window. If the records can still change while the job runs, expect a small difference, and know why.
3. Get the receiver’s word for it. Writing the file isn’t delivery. For Keystone’s nightly status file, the contract in the Data Architect course is plain: “a file isn’t proof; Keystone’s pickup or acknowledgment is, and a missing one is chased by morning” (Checkpoint 2). That’s the return receipt from Proof of Delivery, for a file.

Who looks at these checks, how they raise an alert, and what the dashboard shows is chapter 14, Reliability, Recovery, and Observability. This module is what the program itself has to do.

Test knowledge. Know the three Bulk API 2.0 ingest results: successful (with sf__Id and sf__Created), failed (with sf__Error), and unprocessed (rows never processed because the job failed or was aborted, not the same as failed). Know that records saved before a failure aren’t rolled back, that results are kept seven days, and that failed records must be resubmitted. Know why upsert on an External ID makes a resend safe: no match creates, one match updates, several matches is an error.

What an Architect Decides Here

• The count check. Uploaded rows equal the three result files, every key exactly once, and what happens when they don’t: the program stops and a named person is told.
• How each kind of failure is handled. Which are resent and when, which go back to the data owner, which stop the run.
• The key. Which External ID each feed upserts on, so a resend can’t duplicate, and where that is a deliberate insert instead.
• The evidence. Where the result files, counts and job IDs are kept, for how long, and where the Salesforce IDs are written back.
• The export’s proof. How each outbound file is counted, and whose acknowledgment says it arrived.
• The handoff to monitoring. What the program records so chapter 14’s alerting has something to watch.

Explore

• Get Job Successful Record Results (opens in a new tab), Get Job Failed Record Results (opens in a new tab) and Get Job Unprocessed Record Results (opens in a new tab): Bulk API 2.0 and Bulk API Developer Guide, Winter ‘27 (checked 2026-11-01)
• Insert or Update (Upsert) a Record Using an External ID (opens in a new tab): REST API Developer Guide, API 68.0, how upsert matches (checked 2026-11-01)
• Safe to Retry and Proof of Delivery: the Data Architect course, making a resend safe, and proving data arrived
• Salesforce Developers, Bulk API 2.0 and Bulk API Developer Guide, Create a Job (opens in a new tab), checked 2026-11-07.