Retention and Masking

Module 3 of 9 in the chapter Reliability, Recovery, and Observability, chapter 14 of the Integration Architect course. Each module stands on its own. Previous: Capture Everything: the Integration Log · Next: Retries Without Duplicates

The question this module answers: How long do you keep an integration log, and what are you allowed to write in it?

A Log Fills Up

Module 2 said to capture everything. Here’s the other half: the log itself is data. It grows every day, it takes up storage, and it holds copies of whatever crossed the arrow, which often includes personal data. So you have two challenges to design in from the start, not to fix later: how long you keep it, and what you let into it.

When I first built an integration log, the very next question was: what’s going to happen when it fills up? You need a second batch process that cleans up after the first. And be ready, right from the start, to explain to the client why you’re cleaning up after yourself and not using up all their data storage. That’s a question you want to answer before they ask it.

Salesforce’s large-data-volume guidance says the same thing for any growing object: “A policy of archiving or discarding data at the same rate at which it comes into the system can prevent this effect.” For a log that grows by thousands of entries a day, that’s the design goal. What comes in each day, something goes out each day.

Two Phases, Then Three Tiers

The design I used was a two-phase purge (the Data Architect course has it too). A weekly batch, run on the quietest day of the week, deleted the successful entries after a few days and left the errors. A second pass came along after about a month and deleted the errors. The timings sat in settings, not in code, so they could change without a deployment. And an entry with a do not delete flag, because someone was investigating it, was left alone.

That still works. Today I’d make it three tiers, because one more thing has become clear: the people who report on the integration want the history long after anyone needs the payloads.

Every entry, for a short time. Successes and failures, with their payloads, for a few days. That’s long enough to answer “did last night’s run go through?”
Failures, for longer. Until they’re resolved and a while after: long enough to see a pattern, replay a payload (module 6), or answer the partner who calls next week.
The counts, for a long time. No payloads, no personal data: just how many calls per integration per day, how many failed, how long they took. That’s what module 8 reports on, and it’s small enough to keep for years.

And over all three, the flag. A do not delete flag holds an entry, whatever its tier, until whoever set it takes it off. That’s how an investigation or a legal hold survives the purge.

Click to enlarge

Three tiers and a flag.

As a labelled example, for Steve Trucking’s custom object log: every entry kept 7 days; failures kept 30 days after they’re resolved; daily counts kept 13 months, so this month can be compared with the same month last year. Those aren’t rules. Your numbers come from the business, the partners’ contracts and the law.

That last one matters, because retention cuts both ways. The OWASP Logging Cheat Sheet, written for every kind of system, puts it in one sentence: log data “must not be destroyed before the duration of the required data retention period, and must not be kept beyond this time.” Not too short, and not too long.

Purging at Volume

Deleting a few thousand log entries a week is easy. Deleting millions needs its own design, and it differs by where the log lives.

A custom object. Deleted records go to the Recycle Bin first and still count against performance while they’re there; at volume, the Data Architect course covers hard delete through Bulk API 2.0 (Archiving and Purging).

A big object ledger. Here’s a catch that surprises people. A big object is deleted with deleteImmediate(), and Salesforce’s guide is strict about how: “You must specify all fields in the index. You can’t include a partially specified index or non-indexed field, and wildcards aren’t supported.” There’s no “delete everything older than 30 days.” To delete a row, you have to name it by its whole index, and you can only query by the index too.

So the purge has to be designed into the index, before the big object is deployed, because the index can never change (module 2). If you’ll purge by age, the index needs a date in a position you can query on, so a job can find each day’s entries and then delete them by their full index, up to “50,000 records at a time.” Design the index for the question you’ll ask most, and for the day you’ll need to delete.

Test knowledge. Know that an integration log needs a retention plan from day one, and that Salesforce recommends archiving or discarding data “at the same rate at which it comes into the system.” Know the purge mechanics: custom object deletes go through the Recycle Bin (hard delete through Bulk API 2.0 at volume); big object rows are deleted with deleteImmediate(), naming every field in the index, up to 50,000 at a time.

Decide What the Log May Hold, Field by Field

Now the harder challenge. Integration logs can hold personal data, so we need to decide how to mask what we write.

Chapter 6 already gave every field a level, and said what each level means for a log (Classifying the Data): Public and Internal can be logged as they are; Confidential is masked; Restricted is never logged. This module is how you make that happen in the log itself.

The rule I’d put first: mask when you write, not when you read. If the full value goes into the log and you hide it on the screen, it’s still in the log, in every export, every backup, every copy someone pulled to look at a failure. Once a value is written, assume it’s been seen. Masking has to happen before the entry is saved.

The OWASP cheat sheet gives the list of what “should usually not be recorded directly in the logs, but instead should be removed, masked, sanitized, hashed, or encrypted.” It includes access tokens, passwords, encryption keys, “Bank account or payment card holder data”, and one line I like a lot: “Data of a higher security classification than the logging system is allowed to store.” That’s chapter 6’s levels, in OWASP’s words. And names, phone numbers and email addresses, it says, “may also need to be treated in some special manner.”

For each field on the arrow, the log does one of four things:

Keep it as it is. The status, the delivery window, the partner’s reference. You need these to find and fix a failure.
Mask it. Keep enough to recognize, not enough to use: the last four digits of a phone number, the postal code without the street.
Replace it with a keyed hash. When you need to match entries, “all the failures for this recipient”, without storing who the recipient is. More on this below.
Never write it. Secrets and tokens (chapter 13), card numbers, and anything Restricted.

Here’s one of Steve Trucking’s manifests, field by field.

Click to enlarge

Mask when you write, not when you read.

Look at the gate code. It’s on the manifest because the crew needs it at the gate, on the day. It never belongs in a log, where a support person reading a failure next week would see it.

A Hash to Match Without Exposing

Sometimes you need to connect log entries about the same person. A recipient complains that three deliveries never got their status; you want to find every failure for that recipient. But you don’t want their phone number sitting in the log.

So you store a keyed hash of the phone number instead: a one-way fingerprint made with a secret key, the same way chapter 8’s webhook signatures work (in Apex, Crypto.generateMac). The same number always gives the same fingerprint, so you can search for it. But the fingerprint can’t be turned back into the number, and without the key, nobody can make the fingerprint for a number they’re guessing. Keep the key where secrets live, never in the log.

It also helps on the day someone asks to be forgotten. Make the fingerprint of their number, and you can find every log entry about them, and delete it.

But be honest about what a hash is. It’s not anonymous. Under the EU’s GDPR, for example, data that has been pseudonymised but “could be attributed to a natural person by the use of additional information should be considered to be information on an identifiable natural person” (Recital 26). A hashed phone number is still personal data. It’s much safer, and it still needs the retention plan.

The List Lives in Settings, Not Code

Which fields get masked, hashed or dropped will change. A partner adds a field. A law changes. Someone finds a new place where a gate code was hiding in a notes field.

So the masking list should be something an admin can change without a developer: in Salesforce, custom metadata, one row per integration and field, saying keep, mask, hash or drop. Custom metadata is deployed with the rest of the configuration, so the list in production is the list you tested. And the integration layer can read the same classification from Salesforce’s field settings, so both layers mask the same way (chapter 6).

Two more places to check, because they’re easy to miss:

The ledger. A big object doesn’t support encryption: encrypted data copied into one is “stored as clear text on the big object” (module 2). So the masking has to happen before the payload reaches the ledger.
The debug log. Printing the whole response while you build is normal. Leaving it in production puts every payload in a debug log the moment someone turns one on (chapter 9 module 3).

The Law, Around the World

I want to be careful here, because this course is read in many countries, and the laws aren’t the same. But two ideas show up in privacy law almost everywhere, and the log needs both.

Keep only what you need. The GDPR calls it data minimisation: personal data must be “adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed” (Article 5(1)(c)). For a log, that’s the masking list.

Keep it only as long as you need it. The GDPR again: kept in a form that identifies people “for no longer than is necessary” (Article 5(1)(e)). For a log, that’s the three tiers.

Your own country’s law will say it in its own words, and your company’s privacy team decides what it means for you. The Data Architect course goes further into the laws (Classify, Then Protect). The architect’s part is to make sure the log can do whatever they decide: purge on time, hold when told, and find everything about one person.

Beyond the exam: what AI changes. AI tools are good at spotting personal data in a sample of log entries, including the gate code hiding in a notes field that nobody listed. Use that to review the masking list. Don’t send real log entries to an AI tool to find out: send masked samples, or the field list, for the same reason the log is masked in the first place.

What an Architect Decides Here

The retention for each tier, from the business, the contracts and the law, and the day of the week the purge runs.
Who can set the do not delete flag, and who takes it off.
How the log is purged at volume: hard delete for a custom object; for a big object, a date in the index, decided before it’s deployed.
For every field on every arrow: keep, mask, hash or never write, in custom metadata, masked when it’s written.
Where the hashing key lives, and who can use it to find one person’s entries.
What the counts tier keeps, so reporting survives the purge.

Explore

OWASP Cheat Sheet Series, Logging Cheat Sheet (opens in a new tab): what not to log, and keeping logs for exactly as long as required (checked 2026-11-07).
Salesforce Developers, Best Practices for Deployments with Large Data Volumes (opens in a new tab): archive or discard at the rate data arrives (checked 2026-11-07).
Big Objects Implementation Guide, Delete Data in a Custom Big Object (opens in a new tab): deleteImmediate() and the full index (checked 2026-11-07).
Regulation (EU) 2016/679 (opens in a new tab), the GDPR: Article 5 and Recital 26, one example of privacy law (checked 2026-11-07).
Classifying the Data: chapter 6, the four levels and what each means for the log.