Formats, Old to New

Module 4 of 6 in the chapter APIs, Contracts and Data Across the Wire, chapter 3 of the Integration Architect course. Each module stands on its own. Previous: The Star Trek Transporter · Next: Schemas and Transformation

The question this module answers: Which formats does data travel in, what is each one good at, and where will you meet it?

Any communication between two systems has some form of serialization. You just need to know what it is. In the last module, the transporter turned a delivery into a stream of text. But which text? That depends on the format, and there are more of them than you’d think.

So let’s take one Steve Trucking delivery and send it through each format, from the oldest to the newest.

Fixed-Length: Every Field in Its Place

When I started, I read books on advanced COBOL and on JCL, the job control language. I’m dating myself. Back then, we had fixed-length formats. Every field had its own place and its own size. The first name had 20 characters. The last name had 20 characters. If the name was shorter, the rest was filled with spaces.

Here are two Steve Trucking deliveries, in a fixed-length file of our own design:

D-1042  Delivered North   Corner Hardware     0412.5
D-1043  In transitSouth   Hilltop Bakery      0098.0

The id takes 8 characters, the status 10, the depot 8, the recipient 20, and the weight 6. Every record is exactly the same length.

Fixed-length files were extremely fast to work with, because all you had to do was count. If every record is 53 characters, the 51st record starts at character 2,650. Just do the math, and jump straight there. But you could never change the structure later. Add one character to a field, and every position after it moves, for every system that reads the file.

And fixed-length isn’t history. It’s still alive in banking. In the United States, the files that move money between banks, ACH files, are fixed-length. In Nacha’s own words, an ACH file is “a fixed-width, ASCII file, with each line exactly 94 characters in length” (ACH File Overview (opens in a new tab), Nacha, checked 2026-11-01). If you work with banks, you may still have to write one.

Healthcare is a little different from what people often remember. The older HL7 standard, version 2, isn’t fixed-length: its fields are separated by a character, usually a pipe, like PID|1||PAT-0042||Rivera^Ana (HL7 Version 2 (opens in a new tab)). The current healthcare standard, FHIR, is REST with JSON or XML; its current published version is R5 (HL7 FHIR (opens in a new tab), checked 2026-11-01).

XML: Tags That Can Grow

Then XML came along, with its tags. An open tag, the value, a close tag:

<delivery>
  <id>D-1042</id>
  <status>Delivered</status>
  <recipient>
    <name>Corner Hardware</name>
  </recipient>
  <items>
    <item><sku>PAL-200</sku><quantity>3</quantity></item>
  </items>
</delivery>

What XML gave us was room to grow. Add a new tag, and the systems that don’t know it can skip it. Fields can hold other fields, so a recipient can sit inside a delivery, and lists can repeat. Structures, like the ones in the contract, became easy to send (XML 1.0 (opens in a new tab), W3C).

For a long time, XML was the standard. SOAP is built on it. Many older systems speak it, and some industries still do. XML is getting rarer in new integrations, but it’s typically legacy, and you don’t want to change legacy around. So you’ll meet it.

JSON: Why It Won

Then came JSON, and today it’s the usual format for web APIs, Salesforce’s REST API among them (RFC 8259 (opens in a new tab)):

{
  "id": "D-1042",
  "status": "Delivered",
  "recipient": { "name": "Corner Hardware" },
  "items": [ { "sku": "PAL-200", "quantity": 3 } ]
}

When I ask a class why JSON won, the first answer is usually readability. That’s one. The other is size. XML can be bloated: an open tag, a close tag, the name of every field written twice. JSON writes each name once.

That mattered a lot when phones went onto 3G networks. On a slow mobile connection, every packet counted, and that’s right when JSON showed up. I used to say a JSON payload could be 30 to 40% smaller. So we measured it, with the full delivery from the contract, the same fields in each format, written compactly, the way one system sends it to another: no spaces, no line breaks.

Click to enlarge

One delivery, by size. Measured, not guessed.

The JSON was 39% smaller than the XML. That’s one payload, not a law: the gap depends on how many fields there are and how deep they go. But it’s about a third. And notice why: the less a format repeats, the smaller it is. XML writes every field’s name twice, JSON once.

The same is true of spaces. When a person reads a payload, it’s indented, one field to a line, and the spaces and line breaks count too: indented, the same delivery is 477 bytes in XML and 337 in JSON. That’s fine for reading, and it’s why systems send it compact.

JSON and XML share one weakness, and fixed-length doesn’t have it. To find the 10th record in a big JSON or XML payload, you can’t do the math. You have to read from the start, through every open and close, until you get there. With a five-megabyte payload, that takes time, and memory. Chapter 10 comes back to it.

CSV: Rows and Columns

CSV, comma-separated values, is the spreadsheet’s format: a row of column names, then one row per record (RFC 4180 (opens in a new tab)).

id,status,depot,weightKg,deliveredAt,recipientName,recipientCity
D-1042,Delivered,North,412.5,2026-11-01T14:05:00Z,Corner Hardware,Springfield

It’s the smallest of the text formats, and it’s what you’ll use to load data in bulk. Salesforce’s Bulk API 2.0 takes CSV, and only CSV, for loading data (Create a Job (opens in a new tab), Salesforce Developers, checked 2026-11-01).

But notice what’s missing. CSV is flat. The recipient had to be split into two columns. And the list of items doesn’t fit in one row at all: it needs a second file, with the delivery’s id on every row to join them back together. That’s the price of being small. Chapter 10 is bulk.

YAML: Written for People

YAML carries the same structures as JSON, but it uses indentation instead of brackets and quotes, so it’s easier for people to read and write (YAML 1.2.2 (opens in a new tab)):

id: D-1042
status: Delivered
recipient:
  name: Corner Hardware
items:
  - sku: PAL-200
    quantity: 3

You met YAML in the contract: it’s how most OpenAPI files are written. You’ll also meet it in settings files. You’ll rarely see it as the payload between two systems: it’s written for people, so it’s always indented. Even so, it’s smaller than indented JSON, 222 bytes for the same delivery, because it has no brackets and few quotes. Its one catch: the indentation is the structure. Move a line two spaces, and you’ve changed its meaning.

Binary: Written for Machines

The last kind isn’t text at all. Binary formats pack data into bytes that only a machine can read. They’re the smallest and the fastest, and you need the schema to read them: without it, the bytes are meaningless.

You’ll meet one on Salesforce. The Pub/Sub API, the way to publish and subscribe to events at scale, carries each event’s payload in binary Apache Avro (Event Data Serialization with Apache Avro (opens in a new tab), Salesforce Developers, checked 2026-11-01; Apache Avro (opens in a new tab)). The subscriber first asks for the event’s schema, then uses it to turn the bytes back into an event.

Why Binary Still Rules

Everything a computer stores or sends is bits, zeros and ones. Eight bits make a byte. The question is what you spend those bytes on.

A text format spends them on characters. Every letter, every digit, every bracket and quote is a character, and in UTF-8, the encoding the web uses, a basic Latin letter or digit takes one byte, an accented letter like é takes two, a character like 中 takes three, and an emoji takes four (RFC 3629 (opens in a new tab)). So the number 40, written as text, is two characters: two bytes. The delivery time, 2026-11-01T14:05:00Z, is 20 characters: 20 bytes.

A binary format doesn’t write the characters. It stores the value itself, in as few bits as it needs. In Avro, the number 40 takes one byte. The same delivery time, stored as a count of milliseconds, takes six. And the field names aren’t sent at all, because both sides have the schema (Apache Avro specification (opens in a new tab), Binary Encoding). The whole delivery came to 84 bytes, against 238 in JSON.

Click to enlarge

Text or binary. The same values, measured.

Not every value is smaller. The weight, 412.5, is five characters as text, and eight bytes as a binary decimal number. But across a whole message, and across millions of messages, binary wins. That’s why it still exists, and why it rules wherever volume and speed matter most: events, streams, files, the machine-to-machine traffic nobody reads.

And there’s one more reason. Put binary content into a text format, and it gets bigger. A photo of a signed delivery note is binary already. To carry it inside JSON, it has to be turned into text first, usually Base64, which writes every three bytes as four characters (RFC 4648 (opens in a new tab)). A 2 MB photo becomes about 2.7 MB of text. Send the same photo as a file, as binary, and it stays 2 MB. If you try to encode something into text, it can get really big.

Two details from Salesforce’s own documentation show why the transporter matters. The Pub/Sub API doesn’t check the payload when it’s published, so a bad payload only fails when a subscriber tries to read it. And its dates arrive as a count of time since 1970, where the older Streaming API sends them as dates in text (Differences Between Change Events Received with Streaming API vs. Pub/Sub API (opens in a new tab), checked 2026-11-01). Same event, two formats, two ways to send the time. Chapter 11 is events.

Choosing a Format

So which one? Usually, you don’t choose. The server’s contract does, and the client follows. But when you’re designing the contract yourself, every format makes a trade.

Click to enlarge

Formats, old to new. Every format makes a trade.

For a call between two systems today, JSON is the usual answer. For a large load, CSV. For events at scale on Salesforce, Avro. For a file a person will edit, YAML. For a bank, whatever the bank says. And whatever goes out, the contract should say what comes back, in the same format.

What an Architect Decides Here

• For every interface, the format in each direction, written in the contract.
• For an older system, whether to speak its format, or translate it once, in one place. Module 5 is translation.
• For bulk loads, how a structure becomes flat rows, and how the rows join back together.
• For binary formats, where the schema lives, and who can read it.

Beyond the exam: what AI changes. AI is very good at turning one format into another: XML into JSON, a fixed-length layout into CSV, a sample payload into YAML. It’s a real time saver. But it guesses at what it can’t see: a field’s width, a date’s time zone, which rows join to which. Give it the layout, and check what it produced against the contract. Module 6 does this on a real payload.

Explore

• Nacha, ACH File Overview (opens in a new tab): the 94-character record.
• W3C, XML 1.0 (opens in a new tab); IETF, JSON (RFC 8259) (opens in a new tab) and CSV (RFC 4180) (opens in a new tab); YAML 1.2.2 (opens in a new tab); Apache Avro (opens in a new tab).
• HL7, FHIR (opens in a new tab) (R5) and Version 2 (opens in a new tab).
• Salesforce Developers, Event Data Serialization with Apache Avro (opens in a new tab), checked 2026-11-01.
• IETF, UTF-8 (RFC 3629) (opens in a new tab) and The Base16, Base32, and Base64 Data Encodings (RFC 4648) (opens in a new tab), checked 2026-11-01.
• Salesforce Developers, Bulk API 2.0 and Bulk API Developer Guide, Create a Job (opens in a new tab), and Differences Between Change Events Received with Streaming API vs. Pub/Sub API (opens in a new tab), checked 2026-11-01.
• The next module, Schemas and Transformation: describing a shape, checking it, and changing one into another.