The Engines, and the Second Layer

Module 5 in the chapter Under the Hood: What Data 360 Sits On, chapter 2 of the Data 360 Architect course. Each module stands on its own. Previous: Change as a Feed · Next: What This Explains

The question this module answers: What does the processing in Data 360, how does it run, and why is real time a separate layer?

Files, tables and a feed hold the data. Something has to read it, join it, add it up and answer questions. And something has to answer “who is this visitor, right now?” in a few milliseconds. Those are different jobs, and Data 360 uses different machinery for each. This module opens the machinery: how the work is spread across many machines, who runs those machines, and which engines do which job.

Why one machine isn’t enough

Start with the size of the job. A segment over three years of orders, a transform over a billion web events, an identity resolution run across ten sources. No single computer reads that fast enough. So the work is split: the files are divided up, many machines each read their share at the same time, and the results are combined at the end. That’s called distributed processing, and it’s how every big data platform works.

Distributed processing raises three practical questions. Which machines do the work, and how many? Who starts them, watches them, and replaces one that fails? And which program splits the work and puts the answer back together? Data 360’s answers are Kubernetes, a controller Salesforce built, and two engines, Spark and Hyper. Let’s take them in that order.

Kubernetes: what “Kubernetes-native” means

Salesforce describes Data 360’s compute as an “Elastic Compute Fabric: Kubernetes-native, multi-tenant compute fabric. Runs Spark for distributed processing and Hyper for SQL workloads. Scales elastically across diverse jobs and support isolation for running untrusted code.” That sentence needs unpacking.

First, containers. A container is a program packed together with everything it needs to run, so it runs the same way on any machine. Think of a shipping container: what’s inside doesn’t matter to the crane, the ship or the truck.

Kubernetes is the open-source system that runs containers across many machines. Its own project calls it “a portable, extensible, open source platform for managing containerized workloads and services.” In plain words, you tell Kubernetes what you want running (this many copies of this program, with this much memory), and it makes it so, and keeps it so. What it does for a platform like Data 360, in its project’s words:

• Self-healing: it “restarts containers that fail, replaces containers”, so one broken machine doesn’t stop a job.
• Automatic bin packing: you say how much CPU and memory each container needs, and it fits containers onto machines “to make the best use of your resources.” That’s how many customers’ jobs share machines efficiently.
• Scaling and rollouts: it adds and removes copies as load changes, and rolls out new versions “at a controlled rate.”

So “Kubernetes-native” means Data 360’s processing runs as containers that Kubernetes starts, places, scales and restarts. Nothing sits on a fixed server waiting. When your job runs, the machines it needs are found for it; when it ends, they go back to the pool. That’s what “elastic” means here, and it’s also how one platform serves many tenants safely: each job is kept in its own space.

The conductor: the data processing controller

Kubernetes runs containers, but someone still has to decide which jobs run where, with what settings. Salesforce built that layer and calls it the data processing controller (DPC): “a comprehensive, multi-workload data processing orchestration service that provides Job-as-a-Service (JaaS) capabilities.” When you press Run on a transform, or a schedule fires, the job goes to the controller.

What it does, from Salesforce’s description:

• It picks and sizes the machines. Services ask for CPU, memory and counts; the controller chooses the machine types and mixes cheaper “spot” capacity with on-demand capacity.
• It handles failure. “Automatic failure handling, retries and dynamic load balancing.”
• It keeps tenants apart. It “inherently supports multitenancy, designed to understand and enforce Data 360 tenancy boundaries and resource separation”, with encryption “both in transit and at rest.”
• It tunes Spark for you. “Manually tuning Spark parameters requires specialized skills… The DPC team centralizes this expertise.” It also upgrades Spark and the cloud runtimes without downtime.
• It limits how much runs at once. Jobs are routed by tags, and “a maximum job concurrency setting” controls how much runs at the same time.
• It isn’t tied to one cloud or engine. It runs Spark on Amazon’s managed service (EMR, “on EC2 and EMR on EKS”, Amazon’s Kubernetes), and it’s built so new engines and other clouds can be added “without exposing underlying infrastructure details to users.”

Notice what’s missing from your side: there’s no cluster to size, no Spark setting to tune, no server to patch. That’s the trade you make with a managed platform. You give up the knobs; you get a team that turns them for everyone.

Spark: the engine that does the heavy work

Apache Spark is an open-source engine for processing data across many machines. Its own documentation describes the shape: an application has one coordinating program, the driver, that splits the work into tasks; a cluster manager (Kubernetes is one of the choices) provides machines; and executors, processes on those machines, “run computations and store data for your application.” The driver sends tasks to the executors, each reads its share of the files, and the results come back together.

In Data 360, Spark does most of the work you configure:

• Batch transforms run on a schedule or on demand, for “complex, resource-intensive restructuring”, either reading everything or in “a highly efficient incremental processing mode” that reads only what changed since the last run (chapter 7).
• Streaming transforms are a SQL query that “continuously executes against the incoming stream of record changes.” To keep costs down, Salesforce packs “multiple streaming transformation definitions for a single tenant into a single underlying compute job” (chapter 7).
• Calculated insights run as scheduled batch Spark jobs; streaming insights run on Spark’s streaming engine. Salesforce groups insights that read the same data “in the same batch CI job or streaming CI job” (chapter 10).
• Segment membership: “the Spark compute layer is responsible for calculating actual Segment membership”, with tricks like “partition pruning to minimize data scanning” (chapter 11). Spark reads Parquet and Iceberg directly, so it works on the lake where the data sits. Identity resolution has its own pipeline, which Salesforce describes as “built on a highly scalable, cloud-native framework”, with its own matching index (chapter 9); Salesforce doesn’t say which engine runs it, so we won’t either.

The query engines: answering questions fast

Running a transform and answering a question are different jobs. A transform reads a lot and writes a lot. A question (“how many people are in this segment?”, a dashboard, a SQL query from an API) wants an answer quickly. Data 360 uses SQL engines built for that:

• Hyper, Tableau’s SQL engine. Its own documentation calls it “Tableau’s blazingly fast SQL engine”, and it reads Parquet and Iceberg files. Salesforce uses it to accelerate “complex analytical queries and interactive dashboards.”
• Trino, which Salesforce lists among the workloads its controller runs (“Trino for Query”). Trino is an open-source “distributed SQL query engine for big data.”
• Segment counts use a separate query path from membership: “Data 360 Query handles segmentation count computations”, and “accelerated segment counts use a sampling-based approach” for fast estimates while you build. That’s why a count while building can differ slightly from the final membership.
• Query federation sends SQL to another platform’s engine, Snowflake’s or BigQuery’s, and pushes down as much of the query as it can (chapter 15).

Salesforce doesn’t publish which engine answers which query in every case, so treat the list as the toolbox, not a routing table.

And everything else on the same fabric

The same controller and Kubernetes fabric run Data 360’s other moving parts, in Salesforce’s own list: “Real-time processing workloads”, “Event delivery for Data Actions”, “Data Extraction Jobs for connectors”, and “Milvus (the vector database for unstructured data indexing)” behind search indexes (chapter 18). One fabric, many kinds of job.

Click to enlarge

What the engines explain

This is the part an architect uses.

• Why a small job still takes minutes. Every run is a job: the controller finds machines, Spark starts a driver and executors, reads its files, and writes new ones. That start-up work is the same whether you process ten rows or ten million. Don’t expect seconds from a batch job.
• Why schedules matter. Jobs share capacity, and the controller limits how many run at once. A hundred transforms set to the same minute wait for each other. Spread schedules, and chain work in order (chapter 14).
• Why cost follows rows read. Spark reads files to do the work, and Data 360 bills mostly by rows processed (chapter 21; a few usage types count megabytes or compute units). Incremental mode, filters early in a transform, and slower schedules are the levers on both time and credits.
• Why the build-time count and the published segment can differ. Counts while building are estimates from a sample; membership is computed in full by Spark (chapter 11).
• Why you can’t tune it, and don’t have to. No cluster settings are exposed. Your design choices are what you control: what runs, on how much data, how often.

Why milliseconds need a second layer

Now the other job. A visitor lands on your website. In the next half second you want to know who they are, what they did today, and which offer to show. Could the lakehouse do that?

Think back to module 2. Parquet is great at questions across many records and poor at “this one record, now.” Files are written in loads, and a new snapshot takes a write to land. That’s the right design for history and analysis, and the wrong one for a conversation that’s happening now.

So Salesforce built a second layer. In its words: “Data 360 Real-Time Layer processes real-time signals and engagement data in memory.” Memory is fast but limited, so there’s a store underneath it: “a petabyte scale NVMe (SSD) storage layer on the Lakehouse”, the low latency store. Salesforce calls it “a durable cache”, and is clear that “most data eventually makes it to the Lakehouse for long term persistence.”

Data moves between the layers. Recent agent conversation turns stay in memory and older ones move to the low latency store, “accessed within a few milliseconds.” A customer’s profile context is fetched from the lakehouse ahead of time and kept close. Salesforce describes “a true storage hierarchy” of memory, SSD and the lakehouse, “with data seamlessly migrating between these layers.”

Click to enlarge

What the second layer explains

• Why real time has its own limits. Real-time identity resolution uses exact matching only; real-time segments can’t use exclusions; real-time graph records have a size cap. The limits come from the latency it promises, not from its size: memory backed by a petabyte-scale store (chapter 13).
• Why “real time” costs differently. Salesforce meters the real-time layer by the events it processes, separately from batch work (chapter 21).
• Why most of Data 360 runs in minutes, not milliseconds. Streams, identity resolution, insights and segments work on the lakehouse. That’s by design, not a weakness; it’s where the history is.
• Why it matters for agents. Agents need both: the deep history in the lakehouse and this session’s facts in the real-time layer (chapter 19).

What an architect decides here

• For each use case, which layer answers it: the lakehouse (across many records, minutes or hours) or the real-time layer (one person, now). Most use cases need only the first.
• Where the cost will come from: rows processed by the engines, or events processed by the real-time layer.
• How jobs are scheduled and chained, knowing they share capacity and each run has start-up time.

Explore

All checked 2026-11-14.

• Data 360 Architecture (opens in a new tab), Salesforce Architects: the compute fabric, Spark and Hyper, the low latency store and the real-time layer.
• Apache Spark: Cluster Mode Overview (opens in a new tab), the Apache Software Foundation: the driver, the executors and the cluster manager.
• Overview (opens in a new tab), Kubernetes documentation (the Cloud Native Computing Foundation): what Kubernetes does.
• Hyper API (opens in a new tab), Tableau: Hyper, Tableau’s SQL engine.
• Trino (opens in a new tab): the distributed SQL query engine.