Change as a Feed

Module 4 in the chapter Under the Hood: What Data 360 Sits On, chapter 2 of the Data 360 Architect course. Each module stands on its own. Previous: Iceberg: A Table Over Files · Next: The Engines, and the Second Layer

The question this module answers: How does Data 360 know what changed, and why does it matter so much?

Picture a table with a hundred million rows. A thousand changed since this morning. If every job downstream had to read all hundred million rows to find the thousand, Data 360 would be slow and very expensive. It doesn’t, and the reason is the snapshot from module 3.

From polling to listening

Salesforce describes a change in how the platform works: storage-native change events (SNCE) “fundamentally changed Data 360 into a reactive and incremental data platform.” Instead of jobs “actively polling the data lake”, asking over and over whether anything changed, the platform is now “passively monitoring for atomic commit events.”

Here’s how it works, in Salesforce’s words and then in plain ones.

1. A write finishes. “Every successful write operation (INSERT, UPDATE, DELETE) to an Iceberg table culminates in an atomic swap of the table’s current metadata file pointer in the catalog.” That’s module 3’s pointer moving.
2. Storage announces it. The cloud storage “is configured to emit a native notification event (like an S3 event) whenever a new metadata snapshot is written.” The storage itself says: this table just changed.
3. The feed says what changed. The change data feed (CDF) “leverages Iceberg snapshots” to produce the stream of changed records. Salesforce adds a detail that matters: its “optimized Iceberg writer computes and persists the changes as part of the write operation itself”, so working out what changed costs almost nothing extra.
4. Downstream acts on the change only. Jobs “subscribe and act only when data has changed, significantly boosting efficiency by avoiding costly full-table scans.”

So the event says when a table changed, and the feed says what changed.

Click to enlarge

What runs on the feed

Salesforce names the features this makes possible:

• Streaming transforms and streaming calculated insights “selectively process only the altered records” (chapters 7 and 10).
• Incremental identity resolution (chapter 9).
• Segmentation and data actions that react when data changes (chapters 11 and 12).

And it’s the same idea behind every “incremental” setting you’ll meet:

• the CRM connector’s incremental refreshes (chapter 5);
• incremental batch transforms (chapter 7);
• identity resolution processing only new or changed profiles (chapter 9);
• incremental activations (chapter 11).

Why an architect should care

Incremental is the cheap path, by design. Data 360 bills mostly by rows processed (chapter 21). A job that reads the feed processes the thousand rows that changed; a full rerun processes the hundred million. Many Salesforce cost tips come down to “stay incremental”: pause automatic runs while you make several changes, avoid full refreshes you don’t need, filter early.

Some changes break the incremental path. When the shape of the work changes, a job may have to start over and read everything. Salesforce says plainly that changing an identity resolution ruleset’s rules or mappings triggers a full rerun (chapter 9). Now you can see why: the feed tells you what changed in the data, not what a new rule would have done to the old data.

Fast isn’t instant. A feed reacts to a finished write. A stream that writes once an hour feeds an hourly change, so a “triggered” flow on that data runs hourly too (chapter 12). The feed removes polling; it doesn’t make the source faster.

Deletes are changes too. A delete is a write and a snapshot, so the feed carries it. That’s how a deletion can flow through to everything built on the table, as long as each step reads the feed. Chapter 15 shows one case where it doesn’t: a federation cache refreshed incrementally adds and updates rows but doesn’t remove deleted ones.

Why every change comes through the front door

There’s no screen in Data 360 where you open a record in the lake and change a value. Every change comes in the way the data came in: through a connection, a data stream and its data lake object. People coming from core find that strange. The foundations explain it, with one honest note first: the format itself doesn’t forbid an edit. Any write to an Iceberg table, even one row, still makes a snapshot and an event. So this is a design choice, and three reasons make it the right one.

1. The lake is built for loads, not edits. Files are written once (module 2), and every change is new files, a new snapshot and a pointer swap (module 3). Editing records one at a time would mean that whole write for each row: thousands of tiny files and snapshots for the housekeeping to merge later. The stream gathers changes into loads, which is what a lakehouse does well.

2. Everything downstream listens to the pipeline. Identity resolution, insights, segments and data actions react to what changed, through the feed. When every change arrives through the stream, they all see it in the same place and the same order. A change slipped into a data model object directly would skip the mapping from its data lake object, and the two would disagree.

3. The source stays the truth. Data 360 isn’t the system of record (chapter 1); a data lake object is the source’s data, fed in. Edit a value inside Data 360, and the next load from the source brings the source’s value back. So you change data in Data 360 by changing what you feed it, and every value keeps a clear trail back to a source and a load.

Salesforce’s own design shows this, even for deletes. To delete ingested records:

• the data stream “must have an upsert refresh mode”;
• for the cloud storage connectors, the records to delete “must be read from the same source directory that the corresponding data stream is ingesting from”: you drop a deletion file where the stream already reads;
• the Ingestion API “provides a delete operation for both streaming and bulk interaction patterns.”

Even taking a record out comes in through the same door.

What an architect decides here

• Keep every job you can on the incremental path, and know which changes force a full rerun.
• Where each correction is made: in the source, or in what you feed Data 360, never by hand in the lake.
• Batch configuration changes together, so a full rerun happens once, not five times.
• Match the schedule upstream to the freshness you need downstream; the feed can only pass on what the source has written.

Explore

All checked 2026-11-14.

• Data 360 Architecture (opens in a new tab), Salesforce Architects: storage-native change events and the change data feed.
• Iceberg table specification (opens in a new tab), the Apache Software Foundation: snapshots and the atomic swap.
• Delete Ingested Records in Data 360 (opens in a new tab), Salesforce Help: deletes through the same stream, by deletion file or the Ingestion API.