Migration Is a Quality Project First

🌐 Also in 19 languages

Español · 日本語 · Português (Brasil) · Français · Deutsch · Italiano · 한국어 · 简体中文 · Português (Portugal) · 繁體中文 · Bahasa Indonesia · Türkçe · Polski · Tiếng Việt · ไทย · Українська · Español (México) · Français (Canada) · Deutsch (Schweiz)

Image generated with AI

Module 1 of 5 in the chapter Migration and Synchronization, chapter 10 of the Data Architect course. Each module stands on its own. Previous: the previous chapter, Integration and Data Movement · Next: Loading at Volume

The question this module answers: What should happen to data before you start loading it into Salesforce?

A migration project can fool you. You have data over here, you need it over there, so somebody starts talking about Data Loader, the Bulk API, CSV files, batch sizes and load order. Those things matter. They’re not where I start. My first question is: what exactly are we about to move? If the source holds duplicate customers, invalid addresses, obsolete values, broken relationships and fifteen years of data nobody needs, Salesforce will happily become the new home for all of it. Faster loading doesn’t fix that. Migration is a quality project first.

Start With the Picture

Here’s Steve Trucking’s landscape from Checkpoint 1, the weekend RouteDesk retires. RouteDesk is the desktop dispatch tool the coordinators used inside each distribution center (DC) to turn manifests into routes. It holds eight years of delivery history, about 5 million deliveries, and the quality varies by DC. At Checkpoint 1 we decided it retires, and we left one question open: what, if anything, comes into the org? That’s a migration decision, and this is the chapter for it.

Click to enlarge

One new arrow. It runs once, and then it disappears.

A migration is an arrow on the landscape like any other. Everything chapter 9 asked of an arrow applies: what moves, how much, by which key, who proves it arrived. The difference is that this arrow has an end. When it’s done, RouteDesk is switched off and the arrow comes off the drawing. That makes it easy to rush, because “we only do it once.” But the data it carries stays in Salesforce for years.

Moving Bad Data Faster Isn’t Success

The legacy system has 500,000 customer records, and we migrate 500,000 customer records. Did we succeed? Maybe. What if 40,000 are duplicates, 75,000 haven’t been active in ten years, thousands have invalid state codes, some have no usable identifier, some children point to parents that no longer exist, and some fields mean different things depending on when the record was created? The report can still say 500,000 source, 500,000 target, 100 percent loaded, and the project can still be a failure. Completeness matters, but completeness by itself doesn’t prove quality.

So before I design the load, I want to understand the source. What data exists, and what does it mean? How old is it, how trustworthy, which fields are actually used, which values are valid, which relationships exist, which identifiers can I trust? What should survive the migration, and what shouldn’t? This is where the earlier chapters come due: chapter 4 taught us the enterprise landscape, chapter 5 that quality has to be designed, chapter 6 identity and authority. Migration applies all of it to an existing body of data. It isn’t just moving rows. It’s deciding which parts of the past deserve to become part of the future architecture.

Migrate, Archive or Discard, and Know Why

A migration inventory needs at least three answers: migrate, archive, or discard according to policy. Five-year-old transactional detail may need keeping for compliance without living in the active data model; obsolete temporary data may have neither a retention requirement nor value; history may belong in a searchable archive. Chapter 7’s lifecycle principle was “design the exit,” and migration is the perfect time to apply it. Don’t carry every record forward because it’s there.

So zoom in on RouteDesk and take it apart, one class of data at a time.

Click to enlarge

Every class of data gets a decision, a place and a reason.

The deliveries not yet made have to come in: manifests arrive one to two days ahead, so on cutover weekend RouteDesk holds about two days of work, roughly 6,400 deliveries, and the crews need them on Monday morning. The recent history comes in because of claims. A damage claim is a Case in Salesforce, and the agent working it needs the delivery behind it. So deliveries still inside the claim window come in too. Steve Trucking accepts claims for 60 days after a delivery, so that’s about 190,000 more Work Orders. Had the business said 30 days, it would be half that. The window is the business’s number, not ours.

The other 4.8 million deliveries go to Snowflake, where Checkpoint 1 already sends status history, kept by the partner key so a claim on an old delivery can still find it, and so leadership’s five-year reports still work. They don’t belong in the operational core: that would make Salesforce RouteDesk’s retirement home, and chapter 7 showed what Work Orders at volume cost. And the route plans and the coordinators’ drafts? Eight years of yesterday’s plans have no value once the deliveries are done. But I don’t decide that alone.

The opposite mistake is “we don’t need the old stuff.” Who decided? Legal, finance, operations, service, compliance? The data architect shouldn’t invent retention policy during a migration; we translate the business and regulatory requirement into architecture. Before excluding data, establish whether it must be kept, for how long, where, who needs access, and whether its relationships and source identifiers must survive. Eight years of Recipients’ names, addresses and phone numbers is exactly that kind of question: archiving them isn’t the same as keeping them forever. The goal isn’t to migrate less. It’s to know why every class of data is moving, or isn’t.

Test knowledge. Migrate what the new system needs to operate, not everything the old one held. Expect a scenario with years of legacy history: the answer is usually to migrate active and recent records and archive the rest outside Salesforce where it stays reachable, with the retention decision made by the business, not by the migration team.

Where the Preparation Happens

Before the steps, the place. All of this preparation happens outside Salesforce, in a working database of its own: Postgres, SQL Server, Oracle, MySQL, even SQLite for a small job, whatever the team knows. Not in Salesforce, and not in a spreadsheet that gets emailed around. I call it the preparation data store, or the preparation store for short. It sits in between: the source feeds it, and it feeds Salesforce.

Click to enlarge

The preparation data store sits in between. Every step in this module happens inside it.

The extract lands there raw. The standardizing, cleaning, enriching, deduping and validating all run there, as code. The quarantine is a table, the counts are queries, and the whole thing can be run again from the top. Some tools connect a database like this straight to Salesforce, copying data down and sending changes back; others export files for a data loader. There are many ways to load data. The principle doesn’t change: prepare it outside, then load the result.

And it holds more than the source. Pull down the relevant parts of the target org too. Why? Because dedupe has to compare against what’s already in Salesforce, and when the survivor of a merge is a record that’s already there, the new children need its key to link to it. You can’t decide the merge without both sides in front of you. For RouteDesk, the preparation store holds the RouteDesk extract and a copy of what Salesforce already has: the Retail Partner Accounts, the Products and Service Territories, the claim Cases with their delivery references, and the Work Orders the new integration has already created, because the partners don’t stop sending manifests for our cutover weekend.

Then you practice. Run the preparation, load a sandbox, reconcile, find the rule that was wrong, fix it, run it all again. By the time we reach production, the preparation has run many times, and it’s boring. That’s the goal.

My Order: Standardize, Clean, Enrich, Dedupe, Validate

When I’m preparing data for migration, I work in this order, and the order matters. All five steps happen in the preparation data store.

Standardize. Get equivalent values into one form. One source has CA, California and Calif.; another has US, USA and United States. A company comes in as Steve Plumbing, Steve's Plumbing and Steve's Plumbing LLC. Phone numbers arrive as 5555550142, (555) 555-0142 and +1 555 555 0142. Before we compare records or apply matching rules, we need to know which differences are formatting and which are meaningful.

Names are the classic trap. I had a system with first and last name in one field, and the Salesforce contact needs them apart. “Daniel Reyes” splits easily. Then you get “Mr. Daniel Reyes,” “Daniel Reyes PhD,” “Daniel Reyes III,” and names with two family names, such as Roberto González Suárez, where González isn’t a middle name: split that wrong and you’re in trouble. Build the patterns (two words, three words, a title, a suffix), run the splits, and have the customer validate the results.

Clean. Deal with what’s wrong, unusable or obsolete: invalid codes, impossible dates, misspellings, placeholder values, fields holding data that doesn’t match their meaning. Map picklist values deliberately: a legacy lead status of Cold might become Closed in the target. When the list is long (I’ve seen 200 values), build a transformation table and have the business fill in the target for each source value. Salesforce validates picklists strictly, so catch these before they arrive, and don’t wait for validation rules to tell you what your source contains.

RouteDesk’s “quality varies by DC” shows up right here. Each DC set up its own status codes over eight years, so the same outcome arrives in nine spellings. The transformation table maps every one of them to the Work Order’s status values in Salesforce:

DC	RouteDesk says	Work Order status
DC 1	DLV	Completed
DC 4	Delivered - OK	Completed
DC 7	DONE/INST	Completed (the install is on its line item)
DC 1	NH	Cannot Complete (nobody home)
DC 4	Refused	Cannot Complete
DC 7	CXL	Canceled

The left two columns come from profiling the extract; the right column comes from the business, not from us. Completed, Cannot Complete and Canceled are three of the standard Work Order status values (WorkOrder (opens in a new tab), Object Reference, Winter ‘27, checked 2026-10-27). Notice the third row: one RouteDesk code carried two facts, the delivery and the install, and the target model keeps them apart.

Enrich. Sometimes missing information can be reliably added: a normalized address, a postal code from city and state, a county, a geolocation, a mapping from a legacy code to the new enterprise taxonomy, an enterprise identifier. Enrichment needs provenance. If we add or change information, I want to know where it came from and why I trust it. Some of RouteDesk’s older deliveries don’t name the Retail Partner at all, because each DC served one partner and everybody knew. The DC and the order number’s format tell us which partner it was. That’s enrichment from a rule we can show, not a guess.

Dedupe. Then find the records that represent the same real-world thing. Why not start here? Because matching dirty data is harder: Steve's Plumbing and STEVES PLUMBING LLC become much easier to compare after standardization. Dedupe within the incoming data, then against what’s already in Salesforce. Chapter 6’s rule still applies: merging is destructive; linking isn’t. Don’t collapse records because a fuzzy score looks convincing.

Validate. Finally, test the transformed data against the target: required information present, reference values valid, relationships resolvable, keys unique where they should be, values fitting the model, authority rules preserved. Only then do I want to talk seriously about loading. And don’t expect the import tool to catch everything. Salesforce’s own Trailhead project on the NPSP importer makes the point: it catches invalid picklist values and missing required fields, but if a spreadsheet turned a postal code like 02134 into the number 2134, the importer loads the four digits (Perform an Import Dry Run (opens in a new tab), Trailhead, checked 2026-10-27). That’s one more reason the preparation lives in a database with typed columns, not in a spreadsheet.

Click to enlarge

Five steps, in order. A record that fails one goes somewhere visible, with its reason.

Test knowledge. Data quality is fixed before the load, not after. Know the preparation steps (standardize, clean, enrich, deduplicate, validate) and why the order matters: you can’t match records reliably until their values are in one form. Picklist values are mapped with a transformation table the business signs off, because Salesforce rejects values a restricted picklist doesn’t have.

Clean Before the Move

The temptation is “let’s get everything into Salesforce first and clean it up afterward.” I don’t like that as the default. Once bad data is in production, it participates in production: users see it, automation, reports, integrations and agents see it, duplicates start collecting related records, and cleanup gets harder. Where practical, improve the data before it enters the target. Not every historical imperfection has to be fixed; that’s a cost-benefit decision. But “we’ll clean it later” isn’t a migration strategy. It’s technical debt with a launch date.

And not every record deserves the same effort. An active customer with open orders, open cases, current contracts and billing relationships is different from one who bought something once nine years ago. Both may need to be kept. For the critical, active data I may require stricter validation, more enrichment, closer duplicate review and detailed reconciliation; for low-value history, preserving traceability may matter more than making every record perfect. Architecture is partly deciding where quality effort creates value. For RouteDesk, the 6,400 deliveries the crews will drive to on Monday get the strictest checks. The 4.8 million going to the archive need their keys and their dates right; nobody will fix eight years of typos in delivery notes.

Preserve the Source Identity, and Load by Keys

One of the easiest mistakes is losing the ability to trace a Salesforce record back to where it came from. If legacy customer CUST-847291 becomes Account 001…, I still want to know it came from CUST-847291, for reconciliation, troubleshooting, reloads, auditing, matching child records, rollback analysis and the questions that arrive after launch. The new Salesforce ID matters inside Salesforce; it doesn’t erase the source identity. This is the External ID principle from chapter 9. Migration and integration meet at identity.

At Steve Trucking they meet in one field. Every Work Order already carries the partner’s key, Partner_Delivery_Key__c, an External ID marked unique, holding values like REDROCK-8472914-1 (chapter 9, module 2). RouteDesk kept the partner’s order number on each delivery, so the migration builds the same key and upserts on it, exactly as the integration does. That matters on cutover weekend: if Redrock resends a manifest for Tuesday after we’ve loaded Tuesday from RouteDesk, the integration finds the migrated Work Order and updates it instead of creating a second one. RouteDesk’s own delivery number comes along too, in a plain field, so anyone can trace a Work Order back to the record it came from.

So don’t rebuild relationships by name (“find the Account whose Name is Acme Distribution”). If the legacy contact carries Legacy_Account_ID = CUST-847291 and the Salesforce Account carries that identifier as an External ID, the relationship is rebuilt deterministically. The Bulk API does this in the load file itself: a column header such as WorkOrder.Partner_Delivery_Key__c says “find my parent by this indexed field,” and Salesforce looks up the parent’s ID for you (Relationship Fields in a Header Row (opens in a new tab), Bulk API 2.0 and Bulk API Developer Guide, checked 2026-10-27). Keys turn “I think this child belongs here” into “I know which parent this came from,” and that matters more as the Number of Zeros grows.

Relationships also create sequence. If contacts reference accounts, the accounts must exist first; if line items reference orders and products, those come first. So migration isn’t one giant load. It’s reference data, parents, children, junctions, then dependent history, in an order the target data model dictates. That’s one more reason the data architect belongs in migration planning: the model tells us how the migration has to be assembled. Here’s RouteDesk’s open work, on Checkpoint 1’s model.

Click to enlarge

Wave 1: what the Work Orders will point to has to exist first. Most of it already does.

Click to enlarge

Wave 2: the Work Orders, upserted by the partner key.

Click to enlarge

Wave 3: the children, each finding its Work Order by the same key.

Click to enlarge

Wave 4: the claim Cases already in Salesforce get their link to the delivery. The model sets the order.

Test knowledge. Keep each legacy record’s key in an External ID field, load parents before children, and let each child find its parent by that key: an upsert on the External ID, and a relationship column in the load file (Parent.External_Id__c) instead of looking up Salesforce IDs. Reloads then update instead of duplicating, and every record traces back to its source.

Merged Records Need a Child Strategy

Here’s the problem people underestimate. Analysis finds that legacy customers A, B and C are the same customer, so the target gets one Account. Fine. But A has 12 cases, B has 4 orders and C has 3 contacts. Merge the parents conceptually without redesigning the child mappings and you’ve solved the parent and broken the relationships. Every dedupe decision needs a child strategy.

RouteDesk has its own version. When a partner revised a manifest on the day, a coordinator sometimes keyed the delivery in again instead of editing it. Now RouteDesk has two deliveries with the same partner order number: one with the old window and two stops, one with the new window and the crew’s notes. Salesforce will have one Work Order, because the partner key is unique. Which window wins, and what happens to the stops and notes on the other one?

First, the fields: when two records match, which values win? A blank against a value is easy. A conflict isn’t: Steven or Stephen, the email in Salesforce or the one in the load? Pick a survivor and decide field by field, then put sample cases in front of the business early so their fingerprints are on the choices.

Then the children. I see three options: add the children to the survivor, don’t load the merged record’s children, or load the merged record as inactive, linked to the survivor, with its children kept on it.

Click to enlarge

Three ways, none free. Notice the third one at RouteDesk: the inactive duplicate can’t reuse the partner key, because the survivor already holds it, so it needs a key of its own.

Whichever you choose, it happens in the preparation store, before anything loads. That’s where the survivor is picked, sometimes a record that’s already in Salesforce, pulled down with its key. That’s where each child gets the key of the parent it will point to. And that’s where the merged record either keeps its children, hands them over or is left out. Salesforce receives the result of the merge, not the job of doing it.

How do I choose? It depends on how critical the data is and whether we need traceability back to what was loaded. And weigh the risk against the benefit of merging at all. Joining the wrong two people’s purchase history in a retail customer list is an annoyance; showing one patient’s medications on another patient’s record is a serious breach. The higher the stakes of a wrong match, the more conservative the merge. Whatever you decide, can each child still be traced to its original source? Migration dedupe isn’t “run duplicate matching on Accounts.” The whole relationship graph matters.

I’ve been on the other side of this. A retailer I’d bought from since my college days was migrating its customer data, and when we looked, Steve Simpson showed up under about half a dozen email addresses and street addresses: my college address, the house I rented, the places after that. Should we merge them? There are other Steve Simpsons out there. The business decided that tying together my old history wasn’t worth the risk of merging me with the wrong Steve Simpson. That’s the question every merge has to answer: what’s the benefit of bringing the data together, and what’s the risk of picking the wrong people?

Test knowledge. Deduplicate before the load, within the incoming data and against what’s already in Salesforce, with matching rules and a survivor chosen field by field. Expect the follow-up question: what happens to the merged record’s child records? Re-parent them to the survivor, leave them out (and archive them), or keep them on an inactive linked record. Each has a cost.

Repeatable Transformations, Raw Source Kept

Legacy status A becomes Active, I becomes Inactive, S becomes Suspended. Where does that mapping live? Not in somebody’s memory, not in an Excel formula copied differently into three files, not as a manual step on Friday night. If we reload tomorrow, we should get the same result, so mappings, standardization and cleansing rules, transformations, dedupe decisions and key generation all have to be reproducible. The migration process is code, even when some pieces aren’t literally written in a programming language.

And don’t destroy the starting point. Keep the original extract in raw tables, exactly as it came out, and never change them. Each step writes its result to a new staged table: standardized, cleaned, deduplicated, load-ready. When something looks wrong, I want to know whether it was wrong in the source, changed by our transformation, changed by the load, or changed by Salesforce automation afterward, and with every table kept I can walk back through them and find out. Overwrite your only copy of the source while cleaning it and you lose the evidence.

I learned how much this pays off on a support migration. A company was moving its support desk from Zendesk into Salesforce, and Zendesk’s tickets were very similar to Salesforce Cases, but not exact. So I loaded every Zendesk export into a SQL database as raw tables, one per Zendesk structure: organizations, users, tickets, custom fields, comments, attachments. Then I built interim tables that got closer and closer to the Salesforce structure, with the field and picklist mappings kept as tables of their own. Then I loaded Salesforce from those tables and brought the Salesforce IDs back. In the end every row had three things: its original Zendesk key, the transformations that had been applied to it, which were never one way, and its Salesforce key. That’s what let me do partial loads and reloads whenever I needed to.

Here’s the same design for RouteDesk, one step at a time.

Click to enlarge

Extract: one raw table per RouteDesk structure, each row with its RouteDesk key. These tables never change.

Click to enlarge

Generate: new tables shaped for Salesforce, built from the raw ones with the mapping tables. The raw tables are still there.

Click to enlarge

Load: the Work Orders upserted by the partner key, then their children.

Click to enlarge

Keys back: each Salesforce ID is written to its row. Now every row knows where it came from, what was done to it and where it landed.

So no transformation is allowed to lose fidelity. If a step throws information away (truncates a note, drops a code it didn’t recognize, keeps one of two conflicting values without saying which), I can’t trace back through it, and I can’t repeat the run and get the same answer. Keep the raw tables, keep the staged tables, keep what each load sent and what came back. Then you can rerun the whole thing from the raw extract at any time, and you have the history of every load. This is migration’s version of lineage: preserve where the data came from and what you did to it.

Sometimes the Staging Is Inside Salesforce

The preparation store isn’t the only place to stage. In a few special cases, the last stop before the real objects is a plain custom object inside Salesforce: one flat row per source record, no automation, loaded fast. Then a batch process works through the rows and builds the real records. I’ve used it when Salesforce’s own logic had to create the records and was too slow to load through: a scheduling process that took a few seconds per appointment. Module 3 comes back to that case.

Salesforce uses the same pattern in its own products, and it’s worth learning from. The Nonprofit Success Pack loads a spreadsheet into its NPSP Data Import object first. A dry run checks for matches without creating anything. Then the importer creates the accounts, contacts and donations, and writes each new record’s ID back onto the row it came from (How the Import Process Works (opens in a new tab), Salesforce Help, checked 2026-10-27). That’s a reference, not a rule: staging inside Salesforce is the special case, not the way to load everything.

For RouteDesk, a one-time move with heavy cleaning and merging, the preparation store outside Salesforce does the work, and the load sends the result.

Bad Records Need Somewhere to Go

Of 500,000 records, 497,800 pass validation and 2,200 don’t. Quietly drop them? No. Force garbage into required fields? No. Stop the project until every historical record is perfect? Probably not. I want a quarantine and remediation path that records which source record failed, why, which rule rejected it, whether it must be fixed before cutover, who owns the decision and whether it can be deferred. Now exceptions are visible architecture work instead of hidden data loss.

Before you call a record bad, look at the rule that rejected it. Sometimes the record is fine and the rule is wrong. Salesforce’s Well-Architected framework puts it plainly: “data architecture decisions determine the populations that become visible—and invisible,” and a required email address “excludes populations that don’t have reliable internet access or personal email addresses” (Fairness (opens in a new tab), checked 2026-10-27). RouteDesk has thousands of deliveries where the Recipient gave only a phone number. If the new Work Order requires an email, those aren’t bad records. They’re real people the rule would make invisible. Fix the rule, not the person.

Then prove the transformation before the load. How many source records did we start with, how many did we exclude on purpose, how many duplicates were consolidated, how many are ready, how many quarantined, how many parent-child relationships resolve, and what are the control totals? If the numbers don’t reconcile before the load, loading faster won’t help. Module 4 proves the migration with counts, totals, spot checks and aggregate reports, but proof starts here.

AI Can Help Clean. It Doesn’t Get to Invent Truth.

AI is useful in migration: suggesting mappings, finding suspicious values, grouping potential duplicates, spotting anomalies, generating transformation code, explaining patterns in rejected records. But if an enrichment process decides “this company is probably in Arizona,” probably isn’t enough. And if it decides two customers look similar enough to merge, remember chapter 6: a false match can do more damage than an under-match. Use AI to speed up the investigation. Don’t let it silently manufacture authoritative enterprise data.

Beyond the exam: what AI changes. AI can dramatically accelerate profiling, mapping, anomaly detection and candidate matching during a migration. Keep deterministic rules, provenance and human review around decisions that create, merge or alter authoritative business data.

What an Architect Decides Here

• Which source data is migrated, archived or excluded, according to business value and retention requirements.
• Where preparation happens (a preparation store outside Salesforce, a staging object inside it, or both), and which parts of the target org are pulled down to merge against.
• How data is standardized, cleaned, enriched, deduplicated and validated before it enters the target.
• Which source identifiers and lineage survive, so every important target record can be traced and reconciled.
• The load order the target data model dictates, and the keys each child uses to find its parent.
• How dedupe decisions treat fields and children across the full relationship graph, weighed on criticality, traceability and the risk of a wrong merge.
• Which transformation and exception-handling rules must be repeatable before any production load begins.

Don’t Measure Migration by Rows Loaded

“We loaded 500,000 out of 500,000 records” tells me the loader worked. I want to know: did we move the right data, improve what needed improving, preserve identity and relationships, consolidate duplicates safely? Can we explain what we excluded, reproduce the transformation, trace a Salesforce record to its source, and prove the result? Do that work first. Then we’ll worry about how fast we can load it.

Explore

• Salesforce, General Guidelines for Data Loads (opens in a new tab) and Relationship Fields in a Header Row (opens in a new tab), Bulk API 2.0 and Bulk API Developer Guide, checked 2026-11-01.
• Salesforce, WorkOrder (opens in a new tab), Object Reference for the Salesforce Platform, Winter ‘27 (API version 68.0), checked 2026-11-01.
• Salesforce Help, How the Import Process Works (opens in a new tab) (the Nonprofit Success Pack’s staging object), checked 2026-11-01.
• Hands-on: the Trailhead trail Import Data With Nonprofit Success Pack (NPSP) (opens in a new tab) walks through the whole staging pattern in a practice org: a template, staging rows, a dry run, fixing the failed rows, then the import. It runs in a Trailhead Playground; Trailhead notes that as of December 2025, NPSP is no longer offered as a preconfigured package in production orgs. Checked 2026-10-27.
• Salesforce Architects, Well-Architected, Fairness (opens in a new tab), checked 2026-11-01.
• In this course: Master Data and Identity (matching, survivorship, link vs merge), Data Quality and Governance, and Keys and Write Ownership (the partner key).
• Trailhead, Perform an Import Dry Run (opens in a new tab), checked 2026-11-01.