Перейти к содержимому
SteveTechArcSteveTechArc Courses

Parquet: Data Stored by Column, Written Once

Это содержимое пока не доступно на вашем языке.

Module 2 in the chapter Under the Hood: What Data 360 Sits On, chapter 2 of the Data 360 Architect course. Each module stands on its own. Previous: What Data 360 Sits On · Next: Iceberg: A Table Over Files

The question this module answers: What is Apache Parquet, and what does it explain about Data 360?

At the bottom of Data 360’s lake are files. Salesforce: “storage consists of data lake files (for example S3 in AWS or Blob in Azure) in Parquet format. Data for a source table is stored across multiple partitions as Parquet files, with each table being a collection of these files.” So a year of flight events isn’t one file. It’s many Parquet files. To understand the lake, start with one of them.

What Parquet is

Section titled “What Parquet is”

The Apache project describes it in one line: “an open source, column-oriented data file format designed for efficient data storage and retrieval.” Two words in there do most of the work: column-oriented.

A spreadsheet saved as CSV is stored row by row: flight 1’s number, date, airport and delay, then flight 2’s, then flight 3’s. Parquet turns that around. It stores all the flight numbers together, then all the dates together, then all the airports, then all the delays.

Here’s the picture I’d use. A row-based file is like a pile of index cards, one card per flight with every fact written on it. A Parquet file is like a filing cabinet with one drawer per kind of fact: a drawer of dates, a drawer of airports, a drawer of delays. Ask “what was the average delay at each airport?” and you open two drawers. You never touch the other ten.

One table, stored by row and by column The same small flights table stored two ways, side by side. - **Stored by row, a CSV file:** in the file, flight 1's date, airport and delay come first, then flight 2's, then flight 3's. Asking for the average delay by airport reads every field of every flight. Good at adding or changing one record; poor at questions across millions of records. - **Stored by column, Parquet, as in Data 360's lake (highlighted):** in the file, every date comes first, then every airport, then every delay. The same question reads two columns, airport and delay. Good at questions across many records; poor at changing or fetching one record, now. - **The footnote:** the same three flights in two layouts; Parquet files are written once, and a change means new files. {"author": "Steve Simpson", "blueprint": "1.0.0", "builder": "standard 1.59.0", "built": "2026-11-09", "id": "DGM-D360-UH-02", "rights": "\u00a9 2026 Twinshark LLC. All rights reserved.", "template": "comparison 0.5.0", "theme": "stevetecharc 1.4.0"} Stored by row a CSV file IN THE FILE, IN ORDER Flight 1: date, airport, delay Flight 2: date, airport, delay Flight 3: date, airport, delay AVERAGE DELAY BY AIRPORT READS every field of every flight GOOD AT adding or changing one record POOR AT questions across millions of records Stored by column Parquet, as in Data 360's lake IN THE FILE, IN ORDER every date every airport every delay AVERAGE DELAY BY AIRPORT READS two columns: airport and delay GOOD AT questions across many records POOR AT changing or fetching one record, now Same three flights, two layouts. Parquet files are written once; a change means new files (module 3). © 2026 Twinshark LLC · stevetecharc.com
Click to enlarge

See it: the same flights, both ways

Section titled “See it: the same flights, both ways”

Here are six real flights as a CSV file, the way most of us have seen data all our careers: three flights into PHX on two days in August 2026, from BTS’s data on SkyData (opens in a new tab), with five of its 109 columns. You can open it in any text editor:

FlightDate,Reporting_Airline,Flight_Number_Reporting_Airline,Origin,ArrDelay
2026-08-14,AA,336,DFW,-3.00
2026-08-14,WN,1625,DEN,-10.00
2026-08-14,DL,2208,MSP,-4.00
2026-08-15,AA,336,DFW,-13.00
2026-08-15,WN,1625,DEN,-1.00
2026-08-15,DL,2208,MSP,-11.00

Notice two things. Each line is one flight, with every fact about it side by side. And it’s all just text: nothing in the file says that -3.00 is a number (minutes of arrival delay; negative means early) or that 2026-08-14 is a date. Whatever reads it has to guess (chapter 3 shows why a wrong guess hurts).

Now the same six flights as a Parquet file. You can’t open this one in a text editor; it’s a binary file, built for machines. But a tool can show you what’s inside, and laid out, it looks like this:

flights.parquet
├─ PAR1                                   marks the start of the file
├─ Row group 1  (rows 1 to 6)
│   ├─ FlightDate                        2026-08-14 ×3, then 2026-08-15 ×3
│   ├─ Reporting_Airline                 AA WN DL AA WN DL
│   ├─ Flight_Number_Reporting_Airline   336 1625 2208 336 1625 2208
│   ├─ Origin                            DFW DEN MSP DFW DEN MSP
│   └─ ArrDelay                          -3 -10 -4 -13 -1 -11
├─ Footer  (written last)
│   ├─ schema      FlightDate: date · Reporting_Airline: text · Flight_Number_Reporting_Airline: text
│   │              · Origin: text · ArrDelay: decimal
│   ├─ rows        6, in 1 row group
│   └─ statistics  FlightDate: min 2026-08-14, max 2026-08-15
│                  ArrDelay: min -13, max -1
└─ PAR1                                   marks the end

Read it top to bottom and you can see everything this module is about. The values are grouped by column, not by flight. The repeated dates are stored once with a count. The types are written down, in the footer. And the footer comes last, because the file is written start to finish in one pass. A real file holds millions of rows in many row groups, and a tool prints the format’s own type names (such as INT32 for a whole number), but the shape is exactly this.

Inside one file

Section titled “Inside one file”

You don’t need more detail than that, but four ideas from the format explain a lot:

  • Row groups and column chunks. A file is split into blocks of rows (“row groups”). Inside each block, each column is stored on its own (“a column chunk”). That’s how a reader can pick just the columns it needs.
  • The schema travels with the file. The file’s own metadata says what columns it has and what type each one is. A Parquet file describes itself.
  • Statistics. For each block, the file keeps the smallest and largest value of a column. A reader looking for 2026 flights can skip a block whose dates all fall in 2023 without opening it.
  • Written in one pass. The format puts its metadata at the end “to allow for single pass writing.” A Parquet file is written from start to finish, then read many times. Nobody edits a row in the middle of it.

And because similar values sit next to each other (a whole drawer of airport codes), each column compresses very well. The Apache project set this up on purpose: compression can be “specified on a per-column level.”

What it’s good at

Section titled “What it’s good at”
  1. Questions across many records. Totals, averages, counts, trends: read a few columns, skip the rest.
  2. Skipping. The statistics let engines skip whole blocks that can’t match.
  3. Small storage. Compressed columns take far less space than the same data as CSV.
  4. Open. The project says it’s “built to be used by anyone… not interested in playing favorites.” Snowflake, Databricks, BigQuery, Spark and Data 360 all read it.
  5. Typed and self-describing. No guessing what a column is.

What it’s bad at

Section titled “What it’s bad at”
  1. Changing one record. There’s no editing in place. A change means writing new files. (Module 3 shows how Iceberg keeps that orderly.)
  2. Fetching one record, now. “Show me this customer” means reading across many columns and files. Transactional databases are built for that; a Parquet lake isn’t.
  3. Lots of tiny writes. Every small load makes small files, and many small files slow reads down until they’re merged.
  4. Reading everything. If a job reads every column of every row, the column layout buys you little.

What it explains in Data 360

Section titled “What it explains in Data 360”

Here’s why an architect should care. Each strength and weakness shows up as Data 360 behavior.

  • Data 360 is built for analysis, not transactions. It unifies, segments and analyzes. It isn’t where an order gets edited, and it isn’t the system of record (chapter 1). A columnar lake is exactly the right tool for the first job and the wrong one for the second.
  • Change arrives as loads, not edits. Data comes in by streams and refreshes, and is written as new files. You change data in Data 360 by changing what you feed it, which is chapter 1’s “feed-based” point seen from the bottom.
  • One person, in milliseconds, needs another layer. That’s why Salesforce built a separate real-time layer for in-session answers (module 5, chapter 13).
  • The type is part of the data. Each file carries its columns’ types. That’s one reason a field’s type matters so much on day one (chapter 3 and module 3).
  • Open format, open doors. Because Parquet is a shared standard, other platforms can read Data 360’s data and Data 360 can read theirs (chapter 15).

One caution, because it’s easy to get wrong: Data 360 bills by rows, not columns. Reading fewer columns makes a query faster. It doesn’t make it cheaper in credits, because Salesforce counts rows processed for that work (chapter 21; a few usage types, such as unstructured data, count megabytes instead). What cuts the bill is processing fewer rows: filters, incremental runs and slower schedules.

What an architect decides here

Section titled “What an architect decides here”
  • Nothing to configure: Data 360 manages its Parquet files. What you take away is a test for any design: is this a question across many records (the lake is good at it), or about one record right now (look at the real-time layer)?
  • When a source system offers Parquet files instead of CSV, consider them: unlike a CSV, a Parquet file carries its own column types, and Data 360 uses them. Salesforce lists how each Parquet type converts (a string to Text, a timestamp to a date and time, an integer or a double to Number), and warns that a column of an unsupported type “is ignored and not ingested.” So check the file’s types against that list before the first load (chapter 4, the S3 door).

Explore

Section titled “Explore”

All checked 2026-11-14.