Beyond the Builder

Module 6 in the chapter Data Transforms, chapter 7 of the Data 360 Architect course. Each module stands on its own. Previous: Streaming Transforms

The question this module answers: What else can a batch transform do, and when do you need code?

In my demo of a batch transform, I said you can do this all without writing code. For most reshaping, that’s still true. This module covers four things beyond the basic build: flattening JSON, pivoting, how often a transform really reprocesses your data, and, when the builder isn’t enough, Code Extension. Each is a decision for the architect more than a click path, so that’s how we’ll take them.

Flattening JSON

In chapter 3 we looked at a real flight in JSON (F9 3871, from BTS’s data on SkyData): the airports as objects inside the flight, and the delay causes as a list. Data 360 works in rows and columns, so somewhere that structure has to become tables. One choice was to land the JSON as a string, in one field, and flatten it inside Data 360.

That’s the Flatten JSON transformation, in a transform node. Salesforce’s reason for it: “To use data in downstream features, such as identity resolution and segmentation, the data must be stored as rows and columns in a data lake object or data model object.” What to know before you choose it:

• It needs a schema file that describes the JSON. You can write it by hand or generate it from a sample. Only the type, properties and items keywords are read.
• You define output tables. For that flight: a table of flights, with the airports and times as columns, and a table of delay causes, one row per cause (three rows for F9 3871: carrier 4 minutes, national air system 19, late aircraft 110), each carrying the flight’s key so it points back to its flight. Each list in the JSON is a table you design, with its own key (module 4).
• Bad rows drop out. “The row will not be processed and included in the results if the JSON data is malformed.” No error on the row; it just isn’t there. Count rows in and out.
• There are limits: “The maximum JSON level limit is 6,” and at most 50 properties.

If the source can send flat data, that’s still simpler (chapter 3). Flattening inside is for when it can’t.

Pivoting

In module 2, the PHX arrivals output had one row for each airline on each day. Sometimes the business wants it the other way around: one row per day, with a column for each airline. That’s a pivot, and it’s in the aggregate node. Salesforce: “You can also pivot dimensions as columns.” You add up to two dimension columns under Group Columns, then select the values you want as columns.

Day	AA	OO	WN	…
2026-08-14	125	63	168	…
2026-08-15	122	58	120	…

Flights landed at PHX, from BTS’s August 2026 data (SkyData bts-ontime v1.0, computed November 2026). Columns in alphabetical order. These are counts, not a comparison of airlines.

Notice that you select the values. The columns are the airlines you chose when you built it. A new airline that starts flying next month doesn’t add a column by itself. A pivot is good for a fixed, short list, such as phone types or a few regions, and a poor fit for a list that keeps growing. For that, keep one row per airline per day, and let the report or insight do the pivot.

Incremental or full: how much each run reprocesses

Every batch run is one of two kinds. An incremental run applies only to input data “that has been added, updated, or deleted.” A full run processes everything, new and existing. And the line to remember: “You can’t configure the run type.” The transform picks, from rules on Salesforce’s page.

A run can be incremental only when all of these are true:

• the transform is scheduled;
• the changes are less than 5% of the data;
• the write mode is Replace (see below);
• and every step is one that supports it: filters, dropped columns, single-row formulas, buckets, and the simple date, text and number transformations.

A full run happens when any of these is true:

• the run was started by hand, with Run Now, or by a flow;
• the transform has an aggregation, a join, or a multi-row formula;
• more than 5% of the data changed;
• the write mode is Upsert;
• an input is an external data lake object or data model object (zero copy, chapter 15);
• the transform, or its input’s mappings, were just edited.

Two more rules. “A transform run will be skipped if the source data or transform remains unchanged from the previous run.” And the first scheduled run of each day is always full, to correct data drift: in an incremental run, a record that stopped matching a filter can stay in the output until the next full run.

Now look at module 2’s arrivals transform. It has an aggregation, so every run is a full run, whatever the schedule. That’s not wrong; counting needs all the rows. But it’s what each run costs (chapter 21). When you design a transform, know which kind of run it will get, and schedule it to match.

Replace or Upsert

The output node asks for a write mode. Salesforce’s page on creating a transform puts the difference this way: “Upsert loads all input data in every run, merging it into the output. Replace attempts incremental transforms, loading only changes from previous runs into the output.” The run types page agrees: Replace is required for an incremental run, and Upsert always means a full run.

The two modes also differ in what you must map. Salesforce’s page on writing to a data lake object says Replace “requires all columns in the output object to be mapped,” while Upsert allows unmapped columns, which are left empty. One warning: that same page describes the run behavior the other way round from the two pages quoted above. Those two agree with each other, so this course follows them for how each mode runs.

When the builder isn’t enough: Code Extension

Sometimes the logic doesn’t fit nodes and formulas. For that, there’s Code Extension: “You can bring your custom Python code into Data 360 to extend Data 360’s native capabilities.” Salesforce’s own condition is the one to keep: “If the native Data 360 features don’t meet your business requirements.”

How it fits with what you’ve learned:

• A script runs as a batch transform. You deploy the Python code, then create a batch transform that uses it, and run it by hand or on a schedule like any other.
• Same rules for objects. “Your script must read from and write to the same object type: DLOs to DLOs and DMOs to DMOs,” and it writes to different objects, so the source stays as it was.
• Developers build it outside the org. The toolchain is Salesforce CLI, a Code Extension plugin and a Python SDK. You write and test locally against a sandbox, deploy to the sandbox, check it, then move it to production in a data kit (chapter 22).
• Watch the logs. Execution logs go to a logs data lake object, and “Data governance policies don’t apply to the Logs DLO.” Never print personal data or credentials from a script.
• Check your org first. It has to be turned on in Feature Manager, and Salesforce notes it “isn’t currently supported in orgs that have Bring Your Own Key (BYOK) enabled.”

Code Extension also has a second kind of custom code: functions, which “run as part of the search index pipeline, where they control how content is chunked for search and AI retrieval.” Chapter 18 covers them.

Two more things to plan for. Compute size: each script runs on a compute size you choose (Salesforce’s architect evangelist Von Clark McClendon lists sizes from L to 4XL, changeable later), and the Flex Credits rate card bills Code Extension per compute unit. Status: Code Extension arrived for batch transforms as a beta in February 2026 (Spring ‘26 release notes), and the Summer ‘26 notes describe it without a beta label, available from May 2026. Check your own org’s release before you plan on it.

Salesforce’s condition puts the native features first. Before choosing code, count what it adds: a developer toolchain, a sandbox-first release path, and logic that someone has to read in Python rather than in the builder.

So when is code worth it? Salesforce’s condition is the right place to start, but I wouldn’t stop at “only when the builder can’t do it.” I’ve seen builder work so convoluted that it’s fragile, where a short script would do the same job cleanly. My test is this: reach for code when it will be easier to maintain and support as code than as a huge amount of configuration. A chain of transforms that one clear script could replace is a good sign. So is logic that’s the business’s own secret sauce, like a scoring rule it owns and changes often, kept in source control with tests. Either way, count the costs above first: a developer to own it, the release path, and the compute.

What an architect decides here

• Whether JSON is flattened before it’s sent or inside Data 360, and the tables and keys it becomes.
• Whether an output is pivoted, only when its list of values is fixed.
• For each batch transform, which kind of run it will get, its write mode, and a schedule to match.
• Whether a transform is easier to maintain as configuration or as code, and who owns the code if it’s code.

Explore

All checked 2026-11-14.

• Write the Batch Data Transform Results to a Data Lake Object Output Node (opens in a new tab), Salesforce Help (checked 2026-11-14).
• Flatten JSON Data (opens in a new tab), Salesforce Help.
• Batch Data Transform Run Types (opens in a new tab), Salesforce Help.
• Code Extension in Data 360 (opens in a new tab), Salesforce Developers.