콘텐츠로 이동
SteveTechArcSteveTechArc Courses

Keeping Another System in Step

19개 언어로도 제공

English · Español · 日本語 · Português (Brasil) · Français · Deutsch · Italiano · 简体中文 · Português (Portugal) · 繁體中文 · Bahasa Indonesia · Türkçe · Polski · Tiếng Việt · ไทย · Українська · Español (México) · Français (Canada) · Deutsch (Schweiz)

An old timber barn on the far bank of a still farm pond, its reflection exact in the water except at one corner, where a breeze has started to ripple it.
AI로 생성한 이미지

Data Architect 과정 10장인 마이그레이션과 동기화의 모듈 5/5. 각 모듈은 독립적으로 읽을 수 있습니다. 이전: 전환 주말 · 다음: 다음 장, 트랜잭션, 동시성, 신뢰성

The question this module answers: How do you get a copy of Salesforce data into another system, at volume, and keep it current? ⟲

So far we’ve talked about getting data into Salesforce. Data architects also have to get data out. Not to leave Salesforce: because the data lives in Salesforce, and another system needs a copy of it. A warehouse such as Snowflake for reporting and analytics, an archive or a backup, another Salesforce org, or a sandbox with realistic data for a rehearsal. The Number of Zeros works in reverse too: 500 records out is easy; 50 million out is architecture. And the copy is rarely a one-time thing. Usually the other system wants it today, and again tomorrow, and every day after that. ⟲

Start With the Picture

섹션 제목: “Start With the Picture”

Here’s Steve Trucking’s landscape after cutover. RouteDesk is retired. Salesforce is the system of record for every delivery. And one arrow carries the status history out of Salesforce, through the integration layer, into Snowflake, where leadership’s five-year reports and the analysts live. ⟲

The mirror that never ends Steve Trucking's landscape after cutover, in the same places: people on the left, Salesforce with Field Service in the middle, the integration layer to its right, and the systems around it, with RouteDesk retired and its history in Snowflake. Two arrows are picked out in amber, the mirror this module zooms in on: status events from Salesforce to the integration layer, and status events from the integration layer to Snowflake. Underneath: after cutover, Salesforce is the system of record; Snowflake keeps a mirror of its status history, every day, with no end date. {"author": "Steve Simpson", "blueprint": "1.0.1", "builder": "standard 1.54.0", "built": "2026-10-30", "id": "DGM-DA-MG-50", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "system-landscape 0.13.0", "theme": "stevetecharc 1.4.0"} ABOVE: IDENTITY, APPEXCHANGE, WEB PEOPLE RIGHT: INTERNAL SYSTEMS BELOW: CUSTOMER CHANNELS BOTTOM RIGHT: PARTNERS AND PROVIDERS Dispatch coordinators 45, inside 9 partner DCs Delivery crews ~700 on 330 trucks Care and office staff 100: service, ops, finance Retail Partner users ~300 at 3 partners Recipients ~1 million a year Corporate IdP SSO for office staff only Retail Partner portal Tracking page INTEGRATION Snowflake status history, analytics FleetLink raw GPS stays here Finance ERP invoices partners, pays crews Photo storage 4–5 million a year, off core RouteDesk retired; its history in Snowflake Phone care agents Email email-to-case SMS and email provider day-before message, with a link Redrock REST API, webhooks Keystone CSV over SFTP Brightwave EDI through a third-party network Salesforce Field Service, one US org a link, no login office staff only calls cases in manifests in status out manifests in status out manifests in status out upserts, partner key status events status events latest position only delivery counts delivery counts photos, from the crews' app day-before, with a link Salesforce a community people identity, apps, web an internal system a customer channel the integration layer starts at the system that opens the connection; the label says what moves a shared certificate: trust, no direct call the mirror we zoom in on After cutover. Salesforce is the system of record; Snowflake keeps a mirror of its status history, every day, with no end date. © 2026 Twinshark LLC · stevetecharc.com
클릭하여 확대

Salesforce keeps the data. Snowflake keeps a mirror of it, every day. ⟲

Salesforce isn’t going anywhere. Snowflake doesn’t replace it; Snowflake needs a copy of part of it, kept in step. That’s what getting data out is for, and it’s two problems in a row: the first copy, which can be big, and every copy after it, which has to be current. ⟲

Why You’d Pull a Lot Out

섹션 제목: “Why You’d Pull a Lot Out”

The first copy is the big one. When Snowflake starts mirroring status history, it needs everything so far, not just today’s changes. At Steve Trucking that starts right after cutover: module 3 deferred the status history of RouteDesk’s 196,400 migrated Work Orders to “one backfill after the load.” That backfill is the first big extract out of Salesforce. And as the business grows, every new mirror, every new report store, every rebuild after a problem starts with a copy of everything. ⟲

Other reasons bring you here too: an archive for records that leave the core (chapter 7), a backup you can restore from, another Salesforce org that needs a slice of the data, a sandbox with realistic data for a rehearsal. ⟲

그리고 할 수 있다고 전부 내보내지 마십시오. 8년치 이력, 큰 파일, 낡은 레코드, 임시 데이터, 파생 값, 아카이브된 트랜잭션. 각 분류를 왜 추출하는지 이해하십시오. 모든 레코드에는 시간, 저장 공간, 변환, 이동, 검증, 대사, 보안, 하위 보존 의무라는 비용이 있습니다. “org 전체를 내보내자”가 자동으로 아키텍처가 되지는 않습니다.

밖으로 나가는 길

섹션 제목: “밖으로 나가는 길 ”

Salesforce에는 출구가 여러 개 있고, 각각 다른 일에 맞습니다.

  • Data Export Service. Setup’s Data Export produces a zip of CSV files for the whole org, weekly (Enterprise, Performance and Unlimited Editions) or monthly (all editions except Database.com), on demand or on a schedule. You can run it manually once every 7 days (weekly) or 29 days (monthly). Professional and Developer Editions get the monthly export only. The files are deleted 48 hours after the email is sent, not counting weekends; formula and roll-up summary fields are always excluded; and whoever has the Weekly Data Export permission can see all exported data (Export Backup Data from Salesforce (새 탭에서 열림), Salesforce Help, checked 2026-10-27). It’s a backup-shaped snapshot, not a migration pipeline. ⟲
  • Data Loader와 보고서. Data Loader는 객체를 하나씩 CSV로 내보내고, Export All로 삭제되거나 아카이브된 레코드도 포함합니다. 보고서 내보내기는 작고 다듬어진 추출에 맞습니다. 둘 다 수천 행에는 괜찮지만 수천만 행에는 아닙니다.
  • Bulk query. For volume, the loading dock works in both directions: sometimes the truck is unloading, sometimes it’s being loaded. A bulk query runs asynchronously and returns its results in files. Bulk API 2.0 manages the batching for you, and on large tables it chunks the query itself: its query jobs “automatically determine the best way to divide your query job into smaller chunks, helping to avoid failures or timeouts” (Understanding Bulk API 2.0 Query (새 탭에서 열림), checked 2026-10-27). On Bulk API (1.0) you turn that on yourself, with PK chunking. Salesforce’s Large Data Volumes guide draws the line: “When using a query that can return more than one million results, consider using the query capability of the Bulk API 2.0” (Extracting Data from the API (새 탭에서 열림), Winter ‘27, checked 2026-10-27). ⟲
  • 복구를 위한 백업 제품, 예를 들어 Salesforce 자체의 Backup & Recover. 마이그레이션과는 다른 일입니다. 데이터를 새 집으로 옮기는 것이 아니라 특정 시점을 복원하는 것입니다.

For a mirror, I want the extract repeatable, logged and reconcilable, so that usually means a bulk query driven by a script, not clicks. Steve Trucking’s backfill is exactly that: a bulk query of the migrated Work Orders and their status history, written to Snowflake once, and reconciled. ⟲

Test knowledge. Know the ways data comes out of Salesforce and what each is for. The Data Export Service is a scheduled, whole-org CSV backup (weekly or monthly by edition; files kept 48 hours; no formula or roll-up fields). Data Loader and reports suit small, object-by-object or shaped extracts. Bulk API query handles volume asynchronously. A backup product restores a point in time; it isn’t a migration tool. ⟲

Very Large Copies: PK Chunking

섹션 제목: “Very Large Copies: PK Chunking”

Fifty million records as one giant query may never finish. PK chunking divides the table by its primary key, the record ID: Salesforce adds ID boundaries to your query and runs it as a series of smaller queries, each a separate batch with its own results. The default chunk is 100,000 records and the maximum 250,000. Salesforce recommends it for tables of more than about 10 million records, or when a bulk query keeps timing out. That’s where I use it too: it’s specialized, for very large exports. It’s a Bulk API 1.0 header, it works on a defined list of objects, and each chunk counts toward the daily batch allowance (PK Chunking (새 탭에서 열림), checked 2026-10-27). Bulk API 2.0 does the same chunking for you, but an ORDER BY or a LIMIT switches it off on either API, and the query may then time out. ⟲

Steve Trucking will get there. About a million Work Orders a year, growing 30 percent a year, adds up to about 12.8 million by the end of year 6, unless chapter 7’s archiving keeps the table smaller. Work Order is on Salesforce’s list of supported objects. ⟲

PK chunking: one big table, many small queries On the left, highlighted, one big table: Work Orders, about 12.8 million by year 6 if nothing is archived. On the right, chunks by record ID, each its own query: chunk 1, the first 250,000 record IDs; chunk 2, the next 250,000; chunks 3 to 51, range after range; and chunk 52, the last record IDs. Each chunk returns its own results. Underneath: on Bulk API 1.0 you switch PK chunking on, and each chunk is a batch you download; Bulk API 2.0 query chunks for you. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.48.0", "built": "2026-10-27", "id": "DGM-DA-MG-51", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "cells 0.4.0", "theme": "stevetecharc 1.4.0"} ONE BIG TABLE CHUNKS BY RECORD ID, EACH ITS OWN QUERY Work Orders about 12.8 million by year 6, if nothing is archived Chunk 1 the first 250,000 record IDs Chunk 2 the next 250,000 Chunks 3 to 51 range after range Chunk 52 the last record IDs its own results its own results their own results its own results Bulk API 1.0: you switch PK chunking on, and each chunk is a batch you download. Bulk API 2.0 query: it chunks for you. HOW TO READ IT the table: past 10 million rows, Salesforce recommends chunking © 2026 Twinshark LLC · stevetecharc.com
클릭하여 확대

Twelve million rows become about 52 queries of 250,000. ⟲

문법은 문서에 맡기십시오. 아키텍트가 알아봐야 할 것은 패턴입니다. 0의 개수가 바뀌면 추출 전략도 바뀌어야 할 수 있습니다.

Test knowledge. For very large extracts (Salesforce’s line is more than 10 million records), PK chunking splits a bulk query into ranges of record IDs, each run and downloaded separately. It’s a header you set on Bulk API 1.0; Bulk API 2.0 query jobs chunk automatically. ORDER BY and LIMIT disable it. ⟲

Then Keep It Current

섹션 제목: “Then Keep It Current”

The first copy is a snapshot. By the time it finishes, the source has changed: crews closed deliveries, coordinators changed windows, partners revised manifests. That’s normal. The architecture has to answer: what changed after our baseline? That’s the job of a watermark: a timestamp, a sequence, a change-tracking mechanism, a source identifier. Conceptually it says: this is what we copied, this is the point through which it was current, these are the changes since. Without that boundary, delta processing is guesswork. On the Salesforce side, the watermark is usually SystemModstamp, which is indexed (chapter 8): each run asks for what changed since the last run’s watermark, and moves the watermark forward only when the run succeeds. ⟲

Don’t forget the deletes. A record deleted in Salesforce must disappear from the mirror too, or the reports keep counting it. A normal query doesn’t return deleted records; a bulk query with the queryAll operation “returns records that have been deleted because of a merge or delete” (Create a Query Job (새 탭에서 열림), Bulk API 2.0, checked 2026-10-27). But deleted records stay in the Recycle Bin for 15 days at most, less if someone empties it (Deleting Data (새 탭에서 열림), Best Practices for Deployments with Large Data Volumes, checked 2026-10-27), so a mirror that looks for deletes less often than that will miss some. And a hard delete skips the Recycle Bin altogether, so plan how the mirror learns about those too. ⟲

How often? That’s chapter 9’s question, and Salesforce’s own guidance gives the line: the SOAP API’s getUpdated() and getDeleted() calls for intervals over 5 minutes, and for anything sooner, let Salesforce send the change (Asking Every Minute). Steve Trucking does both: the backfill is a bulk query, and from the cutover watermark onward, Change Data Capture sends each status change through the integration layer as it happens. ⟲

Keeping the mirror current Salesforce on the left, highlighted, the system of record, with the Work Orders and their status. Snowflake on the right, the mirror: status history, never written back. In between, how the copy moves, added one at a time. First, once: the first copy, a bulk query of everything so far, which sets the watermark, sends everything so far to Snowflake. Then, as it happens: every change after, by Change Data Capture through the integration layer, sends each change. Last, every few days: the deletes, a queryAll check well inside the Recycle Bin's 15 days, tells Snowflake what to remove. Underneath: one big copy, then a stream of small ones; each knows where the last one stopped. THE SYSTEM OF RECORD HOW THE COPY MOVES THE MIRROR Salesforce Work Orders and their status The first copy a bulk query of everything so far; sets the watermark Snowflake status history; never written back once everything so far One big copy, then a stream of small ones. Each knows where the last one stopped. HOW TO READ IT the system of record: fixes happen here © 2026 Twinshark LLC · stevetecharc.com
클릭하여 확대

Once: the first copy, everything so far. It sets the watermark. ⟲

Keeping the mirror current Salesforce on the left, highlighted, the system of record, with the Work Orders and their status. Snowflake on the right, the mirror: status history, never written back. In between, how the copy moves, added one at a time. First, once: the first copy, a bulk query of everything so far, which sets the watermark, sends everything so far to Snowflake. Then, as it happens: every change after, by Change Data Capture through the integration layer, sends each change. Last, every few days: the deletes, a queryAll check well inside the Recycle Bin's 15 days, tells Snowflake what to remove. Underneath: one big copy, then a stream of small ones; each knows where the last one stopped. THE SYSTEM OF RECORD HOW THE COPY MOVES THE MIRROR Salesforce Work Orders and their status The first copy a bulk query of everything so far; sets the watermark Every change after Change Data Capture, through the integration layer Snowflake status history; never written back once as it happens everything so far each change One big copy, then a stream of small ones. Each knows where the last one stopped. HOW TO READ IT the system of record: fixes happen here © 2026 Twinshark LLC · stevetecharc.com
클릭하여 확대

Then every change after the watermark, as it happens. ⟲

Keeping the mirror current Salesforce on the left, highlighted, the system of record, with the Work Orders and their status. Snowflake on the right, the mirror: status history, never written back. In between, how the copy moves, added one at a time. First, once: the first copy, a bulk query of everything so far, which sets the watermark, sends everything so far to Snowflake. Then, as it happens: every change after, by Change Data Capture through the integration layer, sends each change. Last, every few days: the deletes, a queryAll check well inside the Recycle Bin's 15 days, tells Snowflake what to remove. Underneath: one big copy, then a stream of small ones; each knows where the last one stopped. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.48.0", "built": "2026-10-27", "id": "DGM-DA-MG-54", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "cells 0.4.0", "theme": "stevetecharc 1.4.0"} THE SYSTEM OF RECORD HOW THE COPY MOVES THE MIRROR Salesforce Work Orders and their status The first copy a bulk query of everything so far; sets the watermark Every change after Change Data Capture, through the integration layer The deletes a queryAll check, well inside 15 days Snowflake status history; never written back once as it happens every few days everything so far each change what to remove One big copy, then a stream of small ones. Each knows where the last one stopped. HOW TO READ IT the system of record: fixes happen here © 2026 Twinshark LLC · stevetecharc.com
클릭하여 확대

And every few days, the deletes. One big copy, then a stream of small ones. ⟲

Test knowledge. To keep another system current, extract only what changed since a watermark (SystemModstamp is indexed), and use queryAll to catch records deleted or merged. Deleted records stay in the Recycle Bin for 15 days, so run the delete check more often than that. For changes needed within minutes, use events (Change Data Capture) instead of polling. ⟲

A Copy Once, or a Copy Forever

섹션 제목: “A Copy Once, or a Copy Forever”

제가 쓰는 구분은 이것입니다. 마이그레이션에는 정해진 시작과 끝이 있습니다. 한 상태에서 다른 상태로 옮겨 갑니다. 첫 운영 추출로 시작하고, 사전 적재를 여러 번 하고, 일주일 걸리고, 델타 적재를 여러 번 포함할 수도 있습니다. 그래도 마이그레이션입니다. 목적지, 즉 마이그레이션이 완료되는 지점이 있기 때문입니다.

동기화는 두 시스템을 일관되게 유지하는 지속적인 프로세스입니다. “마이그레이션 완료, 끄자”가 없습니다. 두 시스템이 모두 의미 있고, 변경은 계속 일어나고, 그 변경은 움직여야 합니다. 이제 다시 9장입니다. 어느 시스템이 값을 소유하는지, 어느 방향으로 움직이는지, 어떤 키가 식별하는지, 누가 보내고 누가 완전성을 보장하는지, 실패는 어떻게 재시도하고 모니터링하는지, 두 시스템이 같은 것을 바꾸면 어떻게 되는지. 요구사항이 “이 시스템들을 일관되게 유지하라”가 되는 순간, 우리는 더 이상 마이그레이션을 설계하고 있지 않습니다. 통합을 설계하고 있습니다.

Steve Trucking had one of each, on the same weekend. RouteDesk into Salesforce was a migration: its delta loads closed the gap, then it ended. Salesforce to Snowflake is synchronization: it started that weekend, and it never ends. ⟲

A copy once, or a copy forever Two equal panels. A copy once, migration, RouteDesk into Salesforce: it ends Monday morning, when RouteDesk retires; built with a runbook, delta loads and a cutover; proven by the four proofs, once, before go; when it fails, fix and rerun from the preparation data store; owned by the migration team, until it's done. A copy forever, synchronization, Salesforce to Snowflake: it never ends, for as long as the reports need it; built with a contract, a watermark and events; proven by proof of delivery, every run; when it fails, catch up from the last good watermark; owned by an integration owner, for good. Underneath: migration has a defined start and end; synchronization is an ongoing process to keep two systems consistent. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.48.0", "built": "2026-10-27", "id": "DGM-DA-MG-53", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "comparison 0.4.0", "theme": "stevetecharc 1.4.0"} A copy once migration: RouteDesk into Salesforce IT ENDS Monday morning, when RouteDesk retires BUILT WITH A runbook, delta loads and a cutover PROVEN BY The four proofs, once, before go WHEN IT FAILS Fix and rerun from the preparation data store OWNED BY The migration team, until it's done A copy forever synchronization: Salesforce to Snowflake IT ENDS Never: for as long as the reports need it BUILT WITH A contract, a watermark and events PROVEN BY Proof of delivery, every run WHEN IT FAILS Catch up from the last good watermark OWNED BY An integration owner, for good Migration has a defined start and end. Synchronization is an ongoing process to keep two systems consistent. © 2026 Twinshark LLC · stevetecharc.com
클릭하여 확대

Same weekend, two arrows. One has an end, so everything about it is different. ⟲

하지만 델타가 자동으로 동기화를 뜻하지는 않습니다. “델타 적재가 있으니 우리는 동기화하고 있어요.” 꼭 그렇지는 않습니다. 물으십시오. 이 프로세스에 정해진 끝이 있나요? “네, 전환 때까지 이 델타를 돌리고, 그다음 레거시 프로세스는 멈춥니다”라면 여전히 마이그레이션입니다. 델타는 그 최종 상태에 이르도록 돕습니다. “아니요, 두 시스템 다 활성으로 남고, 이것이 무기한 일관성을 유지합니다”라면 동기화입니다. 작업은 같아 보일 수 있습니다. 아키텍처상의 의무가 다릅니다.

Test knowledge. Migration has a defined end; synchronization is ongoing. Delta loads before a cutover belong to the migration. Keeping another system in step with Salesforce afterward is synchronization, and it needs an integration design with a clear owner for each value. ⟲

The Mirror Isn’t the Master

섹션 제목: “The Mirror Isn’t the Master”

Salesforce owns the delivery status. Snowflake holds a copy. That has to stay true. Suppose an analyst notices a wrong status in Snowflake and corrects it there. Now the two systems disagree, and the next sync either overwrites the correction or, worse, somebody builds a feed to send it back. Then both systems think they own Delivery Status. Last write wins? Salesforce wins? Human review? If that’s unanswered, we haven’t built a mirror; we’ve created two writable copies. Chapter 9’s rule: technical permission isn’t architectural authority. Fix it at the source, and let the sync carry the fix. ⟲

So write the arrow’s contract down, the way chapter 9 does: who owns each value, which way it moves, what key identifies it, how fresh it must be, what counts as delivered, how failures are retried, how it’s monitored, what the watermark is. Those aren’t extract-script questions; they’re operating-model questions. The first copy builds the starting state. Synchronization keeps it coherent. ⟲

Prove the First Copy, Then Keep Proving It

섹션 제목: “Prove the First Copy, Then Keep Proving It”

Use module 4’s four proofs (counts, totals, spot checks, aggregate reports) on the first copy. Prove the baseline before synchronization becomes the normal operating model, or the sync will faithfully maintain a bad starting point. Integration doesn’t repair a bad copy just because it runs continuously; it can keep two systems perfectly in step around the wrong data. ⟲

Then keep proving it. A mirror that stops quietly is a data quality problem with a delay: the reports look fine and slowly go stale. Chapter 9’s proof of delivery applies to every run: the side that knows what should arrive checks that it did, with counts against the watermark window, and someone is told when it doesn’t. ⟲

AI Can Help Spot the Drift. Keep the Boundary Deterministic.

섹션 제목: “AI Can Help Spot the Drift. Keep the Boundary Deterministic.”

AI can help find changed records, unexpected differences between Salesforce and the mirror, anomalies, likely conflicts and records that seem to have been missed. Useful. But “is this change in the mirror yet?” shouldn’t depend on an AI’s guess from context. Watermarks, keys, ownership rules and state decide it. ⟲

Beyond the exam: what AI changes. AI can make comparison and anomaly detection between a system and its mirror much faster. Keep the sync deterministic: explicit keys, authority and watermarks decide what moves and when. ⟲

아키텍트가 여기서 결정하는 것

섹션 제목: “아키텍트가 여기서 결정하는 것 ”
  • Why each copy of Salesforce data exists, which classes of data it really needs, and which system owns each value. ⟲
  • 일과 0의 개수에 맞는 내보내기 메커니즘: Data Export Service, Data Loader나 보고서, Bulk 쿼리, 또는 아주 큰 테이블을 위한 PK 청킹.
  • What watermark establishes the baseline, how later changes and deletes are captured, and how often (polling or events). ⟲
  • Whether a data movement is a migration with an end or a synchronization without one, and what integration contract keeps the mirror consistent. ⟲
  • How the first copy is proven, and how every run after it is checked. ⟲

Synchronization Keeps You There

섹션 제목: “Synchronization Keeps You There”

Getting data out isn’t leaving Salesforce. It’s giving another system what it needs: once at volume, then every day after, current, complete and never the master. The first copy is the big job; keeping it current is the long one. Remember the line from this chapter: migration has a defined start and end; synchronization is an ongoing process to keep two systems consistent. The first copy gets the mirror there. Synchronization keeps it there. ⟲

더 알아보기

섹션 제목: “더 알아보기 ”