コンテンツにスキップ
SteveTechArcSteveTechArc Courses

Iceberg: A Table Over Files

このコンテンツはまだ日本語訳がありません。

Module 3 in the chapter Under the Hood: What Data 360 Sits On, chapter 2 of the Data 360 Architect course. Each module stands on its own. Previous: Parquet: Data Stored by Column, Written Once · Next: Change as a Feed

The question this module answers: What turns a folder of Parquet files into a table, and what does that explain about Data 360?

Module 2 left us with a pile of files, each written once and never edited. That raises questions. Which files make up the Flights table right now? What happens when a row changes, if files can’t be edited? What if two jobs write at the same time? A folder of files can’t answer any of that. That’s the job of Apache Iceberg.

What Iceberg is

Section titled “What Iceberg is”

The project calls it “an open table format for huge analytic datasets.” It doesn’t store your data; Parquet does. Iceberg keeps the metadata that makes those files behave like one SQL table. Salesforce’s guide puts it this way: “Tables are organized as folders, with data partitions stored as Parquet files within these folders… Iceberg manages a metadata file for each table, detailing schemas, partition specs, and snapshots.”

Three ideas are all an architect needs.

1. The table is a list. Iceberg’s metadata lists exactly which data files belong to the table right now, along with the table’s columns and types. Readers never go looking around a folder. They read the list.

2. Every change is a new snapshot. Iceberg’s specification is clear: “Once written, data and metadata files are immutable until they are deleted.” So a change doesn’t edit anything. It writes new files: new data files, or small “delete files” that mark rows as gone. Then it writes a new list, a snapshot, that includes them. Salesforce: “Modifications to a partition result in new Parquet files as snapshots.”

3. One pointer moves, all at once. The table’s current state is one pointer to the latest metadata. A write finishes by swapping that pointer in one step. Salesforce: “Every successful write operation (INSERT, UPDATE, DELETE) to an Iceberg table culminates in an atomic swap of the table’s current metadata file pointer.” Readers see the old snapshot or the new one, never half a write. Iceberg calls this “serializable isolation… readers never see partial or uncommitted changes.”

Inside one Iceberg table: files, snapshots and the pointer Inside one Iceberg table, in three layers. - **At the top:** Flights, a data lake object, the thing you name, map and query. It reads the table as it is now. - **In the middle, the table in Apache Iceberg format:** Snapshot 1, the list after the first load, kept as history; Snapshot 2, highlighted, the list after the second load and the table now; and the pointer, swapped in one step when a write finishes, pointing to the newest snapshot. - **At the bottom, files in cloud storage, written once and never edited:** the first load wrote flights files 1 and 2. The second load wrote flights file 3 and a delete file saying one row of file 2 is gone. Snapshot 1 lists the first load's files; Snapshot 2 lists files 1 to 3 and the delete file. - **The footnote:** old files aren't touched, so readers always see a whole snapshot, never half a write; two snapshots differ by a known set of files, which is the change feed. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.59.0", "built": "2026-11-09", "id": "DGM-D360-UH-03", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "layers 0.1.0", "theme": "stevetecharc 1.4.0"} Flights, a data lake object what you name, map and query The table (Apache Iceberg) Snapshot 1 the list after the first load, kept as history Snapshot 2 the list after the second load: the table now The pointer swapped in one step when a write finishes Files in cloud storage written once; never edited FIRST LOAD flights file 1 flights file 2 SECOND LOAD flights file 3 a delete file: one row of file 2 is gone reads the table now points to the newest lists its files lists files 1 to 3 and the delete file Old files aren't touched, so readers always see a whole snapshot, never half a write. And two snapshots differ by a known set of files: that's the change feed (module 4). HOW TO READ IT a change writes new files and a new snapshot; the pointer moves to it in one step what the second load wrote © 2026 Twinshark LLC · stevetecharc.com
Click to enlarge

Two layers: Parquet knows its file, Iceberg knows the table

Section titled “Two layers: Parquet knows its file, Iceberg knows the table”

Here’s the clearest way I know to tell them apart. Ask each one what it knows.

A Parquet file knows itself. Its columns, their types, how many rows it holds, the smallest and largest value in each column. That’s all. It doesn’t know other files exist. It doesn’t know it’s part of something called Flights. It has no idea what Flights looked like yesterday.

Iceberg knows the table. Which files make up Flights right now. The table’s schema, and every change to it. Every snapshot, and which one is current. Statistics across all the files, so a reader can skip whole files without opening them. None of that is inside any one Parquet file.

So Iceberg is a second layer of files, metadata files, sitting on top of the data files. Here’s an Iceberg table laid out, the Flights table from the diagram above, with its file names shortened:

flights/                                  one Iceberg table
├─ metadata/                              the Iceberg layer: knows the table
│   ├─ v1.metadata.json                   schema · snapshot 1
│   ├─ v2.metadata.json                   schema · snapshots 1 and 2 · current: 2
│   ├─ snap-2-manifest-list               the manifests that make up snapshot 2
│   └─ manifest-a, manifest-b             each data file, its row count,
│                                         its min and max per column
└─ data/                                  the Parquet layer: each file knows itself
    ├─ file-1.parquet                     first load
    ├─ file-2.parquet                     first load
    ├─ file-3.parquet                     second load
    └─ delete-1.parquet                   second load: one row of file 2 is gone

Read it from the top, the way an engine does. It opens the current metadata file, finds the current snapshot, follows it to the list of manifests, and the manifests list the data files and their statistics. Only then does it open Parquet files, and only the ones that can match. It never lists a folder to find out what’s in the table.

Open the current metadata file, and a trimmed version, with the IDs shortened, looks like this:

{
  "format-version": 2,
  "current-schema-id": 0,
  "schemas": [ { "schema-id": 0, "fields": [ "flight", "date", "origin", "delay_min" ] } ],
  "current-snapshot-id": 2,
  "snapshots": [
    { "snapshot-id": 1, "manifest-list": "snap-1-manifest-list" },
    { "snapshot-id": 2, "manifest-list": "snap-2-manifest-list" }
  ]
}

That’s the whole trick. The current table is one line: "current-snapshot-id": 2. When the next write finishes, a new metadata file is written with a new snapshot in it, and the table’s pointer moves to that file in one step. The Parquet files underneath don’t change at all.

Iceberg over Parquet: two layers, one table One Iceberg table in two layers, read from the top down. - **At the top:** an engine reading Flights, which starts at the metadata and never lists folders. - **The Iceberg layer, which knows the table:** the metadata file, highlighted, with the schema, every snapshot and which one is current; the manifest list, the manifests that make up this snapshot; and the manifests, listing each data file with its rows and its minimum and maximum values. - **The Parquet layer, which knows only its own files:** the first load's file 1 and file 2, and the second load's file 3 and delete file 1, each knowing its own columns, types and statistics. - **The reading order:** 1, the engine reads the metadata file first; 2, it follows the current snapshot to the manifest list; 3, the manifest list gives its manifests; 4, the manifests point only to the files that can match. - **The footnote:** a write adds data files, then new metadata files, then moves the pointer to the new metadata file; the Parquet files underneath never change. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.59.0", "built": "2026-11-09", "id": "DGM-D360-UH-07", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "layers 0.1.0", "theme": "stevetecharc 1.4.0"} An engine reading Flights starts at the metadata, never by listing folders Metadata files The metadata file the schema, every snapshot, which one is current The manifest list the manifests that make up this snapshot Manifests each data file, its rows, its min and max Parquet data files each one knows only itself: its columns, types and statistics FIRST LOAD file 1 file 2 SECOND LOAD file 3 delete file 1 1. reads first 2. the current snapshot 3. its manifests 4. only the files that can match A write adds data files, then new metadata files, then moves the pointer to the new metadata file. The Parquet files underneath never change. HOW TO READ IT the table now is one line here: the current snapshot the data itself, never edited Iceberg knows the table Parquet knows its own file © 2026 Twinshark LLC · stevetecharc.com
Click to enlarge

The names and the exact levels in the middle vary with the format’s version (the newest version simplifies them). The idea doesn’t: metadata files on top that know the table, Parquet files underneath that each know only themselves.

What Iceberg is good at

Section titled “What Iceberg is good at”
  • Safe change on immutable files. Writes are all or nothing, even with several writers.
  • History. Old snapshots remain until they’re cleaned up, so a table can be read “as of” an earlier state. Iceberg calls it time travel, and rollback.
  • Knowing exactly what changed. Two snapshots differ by a known set of files. Module 4 builds on this.
  • Fast planning. The metadata holds column statistics, so engines skip files that can’t match before reading them.
  • Schema changes as metadata. Iceberg can add, drop, rename and reorder columns, and widen a type, “so no data files need to be rewritten.”
  • Open, with a catalog. Any engine that speaks Iceberg can read the table through a catalog. Snowflake, Databricks, BigQuery and Data 360 all do.

What it doesn’t do

Section titled “What it doesn’t do”
  • It doesn’t make the lake transactional. Iceberg makes big writes safe; it doesn’t make one-row edits cheap. Each change is still new files and a new snapshot.
  • It doesn’t change a column’s type freely. The format allows only widening a type without rewriting the data (a whole number to a bigger whole number, for example). Turning text into a number would mean rewriting every file that holds that column.

What this explains in Data 360

Section titled “What this explains in Data 360”

Why you can’t change a field’s type. Salesforce states the rule plainly: “After a DLO or DMO is created, you can’t change the data type of any fields.” The fix is a new field. Salesforce doesn’t say why, so here’s our reasoning, labeled as ours.

  • Every Parquet file under the table carries that column’s type, and Iceberg itself only allows widening without rewriting them.
  • More importantly, everything downstream is bound to that field: mappings, transforms, insights, segments, data kits.

Note what this isn’t: Iceberg can rename a column and widen a type. Data 360 chooses not to offer those changes on its objects. It’s a product rule built on top of the format, and a sensible one, given everything that depends on a field. That’s why chapter 3 says to decide names and types before the first load.

I see what this costs on projects. When you find you need to fix or clarify a field, you can’t change it. You deprecate the one you have and create a second one. Now both live in your data model, and everyone who comes after you has to know which one to use. That’s baggage, and complexity your model didn’t need.

Why change arrives as whole loads. You don’t edit a row in Data 360’s lake; a stream or a transform writes new files and a new snapshot. That’s why Data 360 is feed-based: you change data by changing what you feed it.

Why “incremental” is possible at all. A snapshot knows exactly which files are new. So Data 360 can process just the change instead of the whole table. That’s module 4, and it’s behind every “incremental” setting in this course.

Why zero copy works. An Iceberg table published through a catalog can be read by another platform where it sits. Salesforce’s own layer on top of Iceberg “abstracts tables across various platforms like Snowflake and Databricks.” Chapter 15.

What a data lake object is. When you see a data lake object in Data 360, you’re looking at an Iceberg table: Parquet files, a list, snapshots and a pointer, with Salesforce’s names and relationships on top.

Won’t the layers pile up? Housekeeping

Section titled “Won’t the layers pile up? Housekeeping”

Here’s the question I’d ask next, and it’s a good one. If every change adds files and a snapshot, and nothing is ever edited, what happens after a year of hourly loads? Thousands of small files, thousands of snapshots, delete files on top of delete files. Doesn’t the table get slower and slower?

It would, if nothing cleaned it. So Iceberg plans for housekeeping. The project lists the jobs on its maintenance page, and an architect only needs the four ideas:

  1. Compaction: merge the small files. In the project’s words, frequent writes “may produce small data files that should be compacted into larger files”, because small files cause “an unnecessary amount of metadata and less efficient queries.” Compaction reads many small files and writes a few large ones in their place. A year of hourly loads becomes a handful of big files.
  2. Fold the deletes back in. The small delete files (“row 7 of file 2 is gone”) are merged too, and when data files are rewritten, the deleted rows are simply left out. Nothing is left to subtract when the table is read.
  3. Expire old snapshots. “Snapshots accumulate until they are expired.” Expiring them regularly “is recommended to delete data files that are no longer needed, and to keep the size of table metadata small.” Once no snapshot points to the old small files, those files are deleted for real.
  4. Tidy the metadata. Old metadata files are removed, files left behind by failed jobs are cleaned up, and the lists of files can be rewritten so finding data stays fast.

Here’s what that looks like in the files themselves. Each load lays a thin stripe on the first one, and a delete is a stripe of its own. A read has to open every stripe and stitch them together. Compaction folds them back into a few solid files, and the pointer moves to its snapshot.

Changes pile up as stripes, and compaction folds them in One table's files, growing over time, then compacted. - **First:** the first load, one big file, with the pointer on snapshot 1. - **Then:** each later load adds one more small file on top, a thin stripe, and the pointer moves to the newest snapshot. - **Every hour:** many more stripes; two of them, hatched in red, are delete files ("row 7 of file 2 is gone"). The pointer is on snapshot 18. - **A read:** a bracket down the whole stack: a read opens every file. - **Compaction:** the old stack fades, "the old files expire"; an arrow leads to two solid files, big file A and big file B, with the deleted rows left out, and the pointer is on snapshot 19: compaction is one more snapshot. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "concept 1.7.0", "built": "2026-11-12", "id": "DGM-D360-UH-10", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "stack 0.1.0", "theme": "stevetecharc 1.4.0"} Every load adds a file; compaction folds them back in the first loadThe first load: one big fileEach later load: one more smallfileEvery hour, more small filesa delete is a small file too: row 7 offile 2 is gone Compaction: one more snapshot big file A big file B deleted rows left out The pointersnapshot 19 The old files expire the first load a later load a delete file what compaction wrote © 2026 Twinshark LLC · stevetecharc.com
Click to enlarge

And here’s the part I like: compaction is just another write. It writes new, bigger files and a new snapshot, then moves the pointer in one step, exactly like a load. Nobody reading the table ever sees it half tidied. The same mechanism that layers the changes on is the one that collapses them.

Housekeeping: compaction is just another snapshot Housekeeping on one Iceberg table, in three layers. - **At the top:** Flights, a data lake object, which reads the table now, the same as always. - **In the middle, the table:** snapshots 1 to 8,760, a year of hourly loads, which expire; the compaction snapshot, highlighted, just another write that lists a few big files; and the pointer, which moves to the compaction snapshot in one step. - **At the bottom, files in cloud storage:** before, 8,760 small data files and hundreds of delete files, listed by the old snapshots; after, big file A and big file B with the deleted rows left out, listed by the compaction snapshot. The small files are deleted once no snapshot lists them. - **The footnote:** old snapshots expire and the small files no snapshot lists are deleted; in Data 360, Salesforce runs this housekeeping; in your own lake, your data engineers do. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.59.0", "built": "2026-11-09", "id": "DGM-D360-UH-08", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "layers 0.1.0", "theme": "stevetecharc 1.4.0"} Flights, a data lake object reads the table now, the same as always The table (Apache Iceberg) Snapshots 1 to 8,760 a year of hourly loads; they expire The compaction snapshot just another write: lists a few big files The pointer moves to it in one step Files in cloud storage the small files are deleted once no snapshot lists them BEFORE: MANY SMALL FILES 8,760 small data files hundreds of delete files AFTER: A FEW BIG FILES big file A big file B deleted rows left out reads the table now points to the newest listed them lists only these Then old snapshots expire, and the small files no snapshot lists are deleted. In Data 360, Salesforce runs this housekeeping; in your own lake, your data engineers do. HOW TO READ IT compaction writes a new snapshot, like any load what compaction wrote: fewer, bigger files © 2026 Twinshark LLC · stevetecharc.com
Click to enlarge

What that means in Data 360

Section titled “What that means in Data 360”
  • Salesforce does the housekeeping for you. Its guide describes it at a high level: incoming “full data loads and continuous change events (deltas)… are managed and merged with persistent data to maintain a current state”, and its table layer carries settings “like target file size and compression.” You don’t schedule compaction in Data 360.
  • Salesforce doesn’t publish the details, such as how often it compacts or how long it keeps old snapshots. So we won’t guess. What matters is the idea: the format needs housekeeping, and the platform runs it.
  • Old snapshots aren’t an archive. They’re there for safe writes and short-term recovery, and they expire. If the business needs years of history, model history as data (an engagement object, a status history), not as old snapshots.
  • Housekeeping is one reason tiny, constant loads aren’t free, even when ingestion is. Every small file is one more thing to merge later.
  • In your own lake, it’s your job. When Data 360 reads your company’s Iceberg tables in place (chapter 15), the housekeeping on those tables belongs to your data engineers. A table nobody compacts is slow, and expensive, to read from anywhere.

What an architect decides here

Section titled “What an architect decides here”
  • Names and types, before the first load, knowing they’re fixed in Data 360 even though the format could bend (chapter 3).
  • Whether a source should be read in place, if it’s already an Iceberg table in your company’s lake (chapter 15), and who does the housekeeping on it.
  • Where long-term history lives: as data you model, not as snapshots that expire.

Explore

Section titled “Explore”

All checked 2026-11-14.