Checkpoint 1, Steve Trucking

Module 5 in the chapter From the Lake to the Model, chapter 8 of the Data 360 Architect course. Each module stands on its own. Previous: The Party, Ready for Identity Resolution

The question this module answers: Given three of Steve Trucking’s sources, which door, which category, which key, and which path to the model?

This is the end of Part 1, and the first time the course leaves the airports and the pilots. If you haven’t tried the Checkpoint 1 challenge, try it first. Here’s how I would architect it, and why.

The company, in one paragraph

Steve Trucking delivers appliances, TVs and furniture for big-box Retail Partners to the partners’ customers, the Recipients, at home. About 3,300 deliveries a day. In the org, each delivery is a Work Order, keyed by the partner’s own reference in one field, such as KEYSTONE-PHX-8843119-1. The Recipient is fields on the Work Order, not a contact, because a Recipient belongs to one Retail Partner, and no partner may ever see another’s (S2 in the brief).

Source 1: deliveries and their Recipients, from the org

The door: pull. It’s a Salesforce org, so the CRM connector brings it in (chapter 5). We set up the Work Order stream ourselves and choose its fields, as with any object a bundle doesn’t cover (chapter 5).

The category: profile. This is the one to think about. A delivery sounds like engagement: it happens on a day. But the planned day changes when a Recipient reschedules, and an engagement stream needs an event time that never changes (chapter 6). And because the Recipient lives on the delivery, each Work Order row is the only record of that Recipient for that delivery: one row, one person, which is what profile means.

The key: the partner key, already one field, already unique (chapter 3). No formula needed.

The path to the model: one lake object, several model objects, exactly as the pilots file did in module 3:

Work Order fields	Maps to
Partner key	Individual: Individual ID
Recipient’s first and last name	Individual: First Name, Last Name
Recipient’s email	Contact Point Email (key: partner key plus -EMAIL; Party: partner key)
Recipient’s phone, and in E.164	Contact Point Phone (key: partner key plus -PHONE; Party: partner key)
Delivery address	Contact Point Address (key: partner key plus -ADDR; Party: partner key)
The delivery’s own fields: partner, DC, status, dates	A Delivery object

The keys and fields are this course’s example.

For the Delivery object, look at the standard objects first (module 1); if nothing means “a delivery”, create a custom one, named for what it is, not for the org it came from (module 2). Delivery notes and gate codes stay out: they’re personal data, and nothing downstream needs them (S4).

Notice what this design does to people. A Recipient who gets three deliveries is three Individuals. That’s deliberate, and it’s where Part 2 starts.

Source 2: tracking-page events, pushed

The door: push. The tracking page’s back end sends each event as it happens through the Ingestion API, streaming (chapter 4). Nobody’s job drops a file; the page knows the moment something happens.

The category: engagement. Each row is something that happened at a moment: the link opened, the window viewed, a question asked. The event time is when it happened, and it never changes.

The key: an event ID the page makes for each event.

The path to the model: a custom engagement object, related to the Delivery by the partner key. And here’s the decision to see: no Party. The link’s token identifies a delivery, not a person. Whoever opens the link, the Recipient or someone in the household, the event belongs to the delivery. Linking it to the Individual would claim to know more than we do.

Source 3: manifest lines, pulled from Amazon S3

The door: pull. The integration layer writes each partner’s manifest set, and each revision, as a CSV file in Amazon S3; a scheduled stream picks it up (chapter 4).

The category: other. A manifest line is a plan. It’s not about a person, and its date can change, so it isn’t engagement.

The key: partner key plus the revision number, built by a formula field on the stream (chapter 6): compound values in a single key. With the partner key alone, a revision would overwrite the line it revised, and the requirement says it mustn’t. With the revision in the key, each version is its own row.

The path to the model: a custom Manifest Line object, related to the Delivery by the partner key. And the Recipient’s fields on the manifest stay out of the stream (chapter 3: choose the fields you need). People come from one source only, the org. A second copy of a Recipient’s name and phone would be one more place to protect and delete, and one more Individual to unify for no reason.

Click to enlarge

What waits for Part 2 and Part 3

• Unify the Recipients, or deliberately don’t? Three deliveries, three Individuals. Within one Retail Partner, unifying them could be useful; across partners, never (S2). Chapter 9.
• Keeping each partner’s data apart inside Data 360. Part 3.
• Status history and truck positions: ingest them, or query them where they are. Chapter 15.

Part 1 in one look

Part 1 was about bringing data in. Eight chapters:

1. Data 360 in One Picture: the whole flow first.
2. Under the Hood: what Data 360 sits on: Parquet files, Iceberg tables, the change feed, the engines.
3. Understanding Your Data Structures: the dictionary and the keys, before anything comes in.
4. The Doors In: pull, push, or leave it where it is.
5. Bringing In CRM Data: your orgs, your objects, your timing.
6. Streams and the Lake: the category, the lake object, the formula fields.
7. Data Transforms: reshaping data inside Data 360.
8. From the Lake to the Model: this chapter.

Every decision in this checkpoint came from one of them. Part 2 starts with what we just set up: Individuals from more than one place, waiting to become one.

What an architect decides here

• For each source: the door, the category, the key, and the path to the model, written down before the first stream is built.
• Which source owns the people, and which sources leave the people fields out.
• What each event belongs to: a person, or something else.