Designing Out Lock Contention

Image generated with AI

The question this module answers: How do you stop UNABLE_TO_LOCK_ROW from becoming an architecture problem?

Back to 4 p.m.

It’s the first week live at Steve Trucking. At 4:12 p.m. the support queue fills up: crews tap Delivered and the app says the save failed. Tap again, and it usually works. The integration layer’s log shows the same thing on the partner side: UNABLE_TO_LOCK_ROW on status updates, then success on the second try. By 5 p.m. it’s quiet again. Nothing is broken at 10 a.m.

The first instinct is to add retries everywhere and move on. I don’t want to start there. Retry may end up being part of the answer, but if I start with it, I’m building increasingly clever ways to survive a collision I never understood. In module 1 we drew the lock footprint of a save. This module starts from that drawing and asks: why are these transactions colliding, and what can we change so they don’t?

Find the Shared Lock

Don’t stare at the hundreds of failing Work Orders. They’re all different records. Look for the thing they have in common. Maybe they all belong to the same Account. Maybe they all update the same roll-up. Maybe they point at the same lookup record. Maybe automation on each one updates one shared summary or configuration record.

At Steve Trucking it’s the Manifest. Each crew’s save has its own Work Order and its own Service Appointment, and those never collide. But every Work Order on one DC’s afternoon manifest also reaches for that one Manifest record, and if the Manifest counts its delivered Work Orders, every save also runs the Manifest’s own save while holding the lock. A few hundred crews, one record. That’s the hot spot: the place where the footprints overlap.

Click to enlarge

So my first troubleshooting question is always: where do the footprints overlap? Until I can answer that, I’m guessing.

Observe Before You Guess

The drawing is my theory. Production is the evidence, and the two have to meet. Which transactions fail? Which objects are involved? Which records do they share? When does it happen, and what else is running then: an integration, a batch job, a data load, a particular automation path?

Salesforce gives you a tool for exactly this. Scale Center’s Row Lock report “identifies transactions experiencing high row locks and the entities involved,” and Salesforce’s own suggested fixes start where module 1 did: look for “locks arising due to the data model itself,” look for deliberate FOR UPDATE locks, and “break large transactions into smaller batches.” Well-Architected says the same tool “names the specific trigger and object involved.” Scale Center started in 2023 with Unlimited Edition. Since February 2026, Salesforce says it is available “for all major editions including Enterprise Edition (EE) and Professional Edition (PE) orgs,” in production and full copy sandbox orgs.

But the tool isn’t the architecture. It gives us evidence. We still have to explain the collision. Trust, but verify: draw the footprints, then check them against what production is actually telling you.

The Usual Culprit: Parallel Incoming APIs

In my experience, the most common cause of lock errors is parallel processing at the same time, typically competing incoming APIs. And I understand why teams build it that way. We want throughput. If one worker is good, ten should be better.

Sometimes. Suppose each status update takes one second. One hundred of them, one at a time, take 100 seconds. Ten at a time, in the perfect case, take about 10. That’s a huge improvement, but the math assumes the updates are independent. If all ten workers need the same Manifest’s lock, we haven’t built ten lanes. We’ve built ten lanes that all merge into one intersection. More workers didn’t add capacity to the choke point. They added a longer line, more failures and more retries, and past a certain point, less finished work.

So don’t measure the design by “how many threads can we run?” Measure it by how much useful work completes successfully. Ten parallel workers with constant lock failures can be slower than two that finish cleanly.

Four Levers

When I see lock contention, I work through four levers, in this order, and only then decide how to retry what still fails.

Click to enlarge

Lever 1: Reduce the Footprint

If I can redesign so that independent transactions stop fighting over one shared record, that’s usually my favorite answer. Look at the shared records in your footprint and ask why each one is there. One default Account everything points to. One counter record. One status record. Sometimes there’s a real business reason. Sometimes it was just convenient: “we needed somewhere to put these.” That’s not a good enough reason to make every transaction in the system wait for the same lock.

At Steve Trucking, the Manifest is a real business record: it’s the partner’s list, the unit we receive, reconcile and replay. It stays. What we can question is what each Work Order save does to it. Does every tap of Delivered need to recalculate the Manifest’s counts inside the crew’s own save? The coordinator watching the Manifest needs those counts to be accurate: a few seconds behind is no big deal, but minutes can be. So the counts still have to keep up. They just don’t have to be calculated inside a few hundred crews’ saves.

This is where the relationship decision from module 1 becomes a design move. I’ve moved from master-detail to a lookup when the locking was critical, and coded the roll-ups myself. The lookup still locks the Manifest for a moment on each save. What changes is that the Manifest’s own save, its validation, flows and roll-ups, no longer runs inside every crew’s save.

Don’t trade master-detail casually, though. It brings real behavior with it: the detail’s access follows the master, deleting the master deletes the details, the relationship is required, and roll-up summaries come for free. So I don’t say, “We have locking. Convert everything to lookups.” That’s treating the symptom with brute force. I ask: which master-detail behavior is creating the contention? What do we gain by changing it? What do we then have to rebuild? If the contention is serious enough, I’ll make that trade. I have. But it’s an architecture decision, not a checkbox.

Lever 2: Shorten the Transaction

The second lever is to do less right now. Look at everything a crew’s save does synchronously: save the Work Order, update the Manifest, recalculate counts, notify the Recipient, tell the partner, prepare data for Snowflake. Some of that has to happen before the crew sees Saved. Much of it doesn’t.

When I move work out of the transaction, it’s usually because of large volumes, limits, or to break things up into smaller parts. At Steve Trucking, each status change publishes a platform event when the crew’s save commits, and an Apex trigger subscribed to those events updates the Manifest’s counts. Salesforce describes how that trigger works: it “runs asynchronously in its own process,” and it “can receive a batch of events at once,” up to 2,000. So a few hundred taps in the same minute become a handful of Manifest updates instead of a few hundred saves lining up for the lock, and the count stays seconds behind, not minutes. The partner’s status message goes out the same way, after the save commits.

Here’s what that looks like. The crew’s save no longer touches the Manifest’s count. It publishes an event, one per Work Order that changed status:

// In the Work Order trigger handler, after update.
// Work_Order_Status__e is set to Publish After Commit in its definition,
// so the events go out only if the crew's save commits.
List<Work_Order_Status__e> events = new List<Work_Order_Status__e>();
for (WorkOrder wo : (List<WorkOrder>) Trigger.new) {
    WorkOrder before = (WorkOrder) Trigger.oldMap.get(wo.Id);
    if (wo.Status != before.Status) {
        events.add(new Work_Order_Status__e(
            Manifest_Id__c   = wo.Manifest__c,
            Work_Order_Id__c = wo.Id,
            Status__c        = wo.Status
        ));
    }
}
EventBus.publish(events);

And a trigger on the event does the counting, in its own transaction, a batch at a time. Notice it doesn’t add one per event. It recounts each Manifest it touched, because an event can arrive more than once, and module 3 explains why that matters:

trigger WorkOrderStatusSubscriber on Work_Order_Status__e (after insert) {
    // The Manifests touched by this batch (up to 2,000 events)
    Set<Id> manifestIds = new Set<Id>();
    for (Work_Order_Status__e e : Trigger.new) {
        manifestIds.add((Id) e.Manifest_Id__c);
    }
    ManifestCounts.recount(manifestIds);
}

public with sharing class ManifestCounts {
    public static void recount(Set<Id> manifestIds) {
        // Still a lock: a coordinator may be saving the same Manifest
        List<Manifest__c> manifests = [
            SELECT Id, Delivered_Count__c
            FROM Manifest__c
            WHERE Id IN :manifestIds
            FOR UPDATE
        ];

        // Count the delivered Work Orders on each Manifest, from the records themselves
        Map<Id, Integer> delivered = new Map<Id, Integer>();
        for (AggregateResult row : [
                SELECT Manifest__c manifestId, COUNT(Id) total
                FROM WorkOrder
                WHERE Manifest__c IN :manifestIds AND Status = 'Delivered'
                GROUP BY Manifest__c]) {
            delivered.put((Id) row.get('manifestId'), (Integer) row.get('total'));
        }

        for (Manifest__c manifest : manifests) {
            manifest.Delivered_Count__c = delivered.containsKey(manifest.Id) ? delivered.get(manifest.Id) : 0;
        }
        update manifests;
    }
}

The FOR UPDATE from module 1 is still there, and it should be: the count still has to be right. What changed is who waits for it. Not a few hundred crews, but one subscriber, a batch at a time. To publish, “call the EventBus.publish method,” and with Publish After Commit, each call “is counted as one DML statement against the Apex DML statement limit” (Publish Event Messages with Apex (opens in a new tab), Platform Events Developer Guide, checked 2026-10-31). To subscribe, “write an after insert trigger on the event object type” (Subscribe to Platform Event Notifications with Apex Triggers (opens in a new tab), checked 2026-10-31). One thing to remember: that trigger “runs under the Automated Process entity or the user you select in the trigger configuration,” so give it a user with the access it needs.

“Seconds behind” is a design target, not a promise. Salesforce says plainly that “there can be a delay between when an event is published and when the trigger processes the event.” So measure it under 4 p.m. load in the scale test, and watch it in production. If the count drifts toward minutes, the coordinator is working from stale numbers, and that’s a defect.

But asynchronous isn’t automatically better, and it isn’t eventually correct by magic. Salesforce’s Well-Architected framework is blunt about it: asynchronous work “introduces eventual consistency, so the result of the work isn’t visible in the transaction that requested it,” it needs explicit error handling because “a failure surfaces in a job log rather than to the user,” and the advice is “to keep work synchronous when it comfortably fits.” So before I move something out, I ask: can another process act on the stale value in the meantime? What happens if the later work fails? Who watches for that? Can it safely run twice? That last question is module 3.

Lever 3: Partition the Work

Sometimes the work truly can’t avoid the shared record. Then I look at how the work is grouped. Updates on one Manifest collide with each other, but one DC’s Manifest doesn’t collide with another DC’s. So I don’t need to make all the work wait in one line. I need one line per Manifest.

Click to enlarge

That’s the smallest contention domain: what work has to be ordered together, and what can safely proceed independently? It’s the difference between “turn parallel processing off” and “partition parallel processing intelligently.” Well-Architected describes the same idea at volume: “partition data across logical boundaries to enable parallel processing.”

It’s the same fix I teach for data loads. When two files carry children of the same account and run in parallel, they lock that account against each other. The fix is to prepare the files carefully: identify all the children under that account and put them in one file. Don’t let them be spread around. Salesforce’s guidance is the same: “sort bulk loads by parent Id so that concurrent batches don’t fight over the same rows.” For live traffic, Salesforce has built the same idea into platform event triggers. With parallel subscriptions, available for custom high-volume platform events, one trigger can run as up to 10 partitions side by side, and you choose the partition key that decides which partition handles each event. Salesforce’s advice for avoiding locks is exactly our lever: “choose a partition key that ensures the placement of events that access the same records in the same partition.” Use the Manifest’s ID as the key, and “events with identical partition key values are processed by the same partition in their original received order.” One line per Manifest, with the lines running side by side. The setting lives in metadata, next to the trigger:

<?xml version="1.0" encoding="UTF-8"?>
<PlatformEventSubscriberConfig xmlns="http://soap.sforce.com/2006/04/metadata">
    <masterLabel>WorkOrderStatusSubscriberConfig</masterLabel>
    <platformEventConsumer>WorkOrderStatusSubscriber</platformEventConsumer>
    <numPartitions>3</numPartitions>
    <partitionKey>Work_Order_Status__e.Manifest_Id__c</partitionKey>
</PlatformEventSubscriberConfig>

numPartitions is “a number from 1 through 10,” and a custom partition key must be “a required custom field of the platform event,” written as EventName__e.FieldName__c (Parallel Subscription Configuration Examples in Tooling API and Metadata API (opens in a new tab), checked 2026-10-31). So Manifest_Id__c is required on the event. Every event for the same Manifest lands in the same line.

When the work is a list in Apex instead (a batch job, a nightly correction), the same idea is a few lines: group the Work Orders by their Manifest before you hand them on, so each Manifest’s children travel together, in the same order every time.

// Group the Work Orders by Manifest...
Map<Id, List<WorkOrder>> byManifest = new Map<Id, List<WorkOrder>>();
for (WorkOrder wo : workOrders) {
    if (!byManifest.containsKey(wo.Manifest__c)) {
        byManifest.put(wo.Manifest__c, new List<WorkOrder>());
    }
    byManifest.get(wo.Manifest__c).add(wo);
}

// ...then line them up one Manifest after another, in Manifest ID order
List<Id> manifestIds = new List<Id>(byManifest.keySet());
manifestIds.sort();
List<WorkOrder> sortedByParent = new List<WorkOrder>();
for (Id manifestId : manifestIds) {
    sortedByParent.addAll(byManifest.get(manifestId));
}

It’s Salesforce’s loading advice in code: “group records by the field ParentId in the same batch to minimize locking conflicts” (Loading Data from the API (opens in a new tab), Best Practices for Deployments with Large Data Volumes, checked 2026-10-27). And because every run sorts the same way, it’s the consistent lock order you’ll see below, too. For partner updates arriving through the API, the integration layer does the same sorting before they reach Salesforce.

Lever 4: Control Concurrency

Architects spend a lot of time making systems faster, so “run less in parallel” can feel backward. It isn’t. If the work competes for the same lock, fewer workers can finish more, because fewer transactions fail and retry.

That’s why serial processing is a legitimate tool, and why chapter 10 kept Bulk API serial mode in the toolkit. When I ask a class how they choose, the common answer is: go parallel first, and if there are locking errors, switch to serial. Grab the speed, wait till you run into the speed bumps, and then slow the car down. That’s fair. Serial is slower, but it’s predictable; parallel is faster, but it can work today and fail tomorrow, depending on what else is running. Not because serial is better. Because sometimes the data model has already told us: this work isn’t independent. Listen to it.

In Apex, the everyday dial is a batch job’s scope: how many records each run of execute gets, and so how many records each transaction locks and for how long.

// 50 Work Orders per transaction:
// smaller transactions hold their locks for less time
Id jobId = Database.executeBatch(new WorkOrderCorrectionBatch(), 50);

The scope is the “number of records to be passed into the execute method”; with a query locator it can go up to 2,000 (Database Class (opens in a new tab), executeBatch, Apex Reference Guide, checked 2026-10-31). Smaller isn’t always better (more transactions, more overhead), so measure it, the way chapter 10 tunes batch sizes. For data loads, the same lever is the Bulk API’s serial mode, set when the job is created (chapter 10, Loading at Volume).

Breaking work apart and controlling concurrency go together. If I split one giant job into 5,000 small jobs and launch all 5,000 at the same Manifest at once, I didn’t solve the contention. I made it worse.

Keep Deliberate Locks Narrow

Sometimes we need a lock on purpose: handing out the next number in a sequence, or checking and changing a status that two transactions must never decide at the same moment. That’s where FOR UPDATE fits, and I’ve used it many times. But typically with a select of only one record. A deliberate lock says: this piece of work must happen one at a time. Fine. Make that point as small as possible, and get out quickly.

And when a deliberate lock has to cover two records, take them in the same order everywhere. That’s how you avoid module 1’s two trucks stuck in the intersection. Salesforce’s own runtime does it for DML: it “first locks sObject parent records, then children,” and “locks sObject records in order of ID when multiple records of the same type are being edited,” and asks developers to do the same, “accessing tables and rows in the same order from all locations in an application” (Avoiding Deadlocks (opens in a new tab), Apex Developer Guide, checked 2026-10-31). At Steve Trucking, the rule is the Retail Partner’s Account first, then the Manifest:

// Our rule, in every class that locks both: parent first, then child.
// Account, then Manifest. Never the other way round.
Account partner = [
    SELECT Id
    FROM Account
    WHERE Id = :partnerId
    FOR UPDATE
];
Manifest__c manifest = [
    SELECT Id, Status__c
    FROM Manifest__c
    WHERE Id = :manifestId
    FOR UPDATE
];
// ... decide and update, then let the transaction finish

If every piece of code takes the Account before the Manifest, a transaction can wait behind another one, but two can never each hold what the other needs. And because a locking query can’t use ORDER BY, when you lock several records of the same type, sort the IDs yourself and lock them in that order.

Retry Is Not the First Fix

When a lock fails, retrying can be completely appropriate. But I don’t want retry to hide bad architecture. If a few hundred saves are fighting over one Manifest and half fail, aggressive retry turns those failures into more traffic at the exact intersection that’s already jammed. Well-Architected names it: exponential backoff “prevents retry storms that overwhelm recovering systems.”

So before I decide “retry three times,” I ask the lever questions. Why did we collide? Can we remove the shared lock? Shorten the transaction? Partition the work? Reduce parallelism? Defer what isn’t critical? Then, for the failures that still happen, and some always will: how do we retry them safely? That’s module 3.

Agents Need the Same Discipline

Agents don’t change any of this. They make it easier to break at scale. A coordinator updates a delivery once. An agent can try, fail, retry, re-query, try another action, across hundreds of deliveries, very quickly.

So I don’t want agents making broad, arbitrary writes. We need to give agents best practices on queries and a focused scope on updates, and make them call established processes for updates. Instead of teaching every agent how to change our data model, give it an action that represents a safe business operation: “reschedule this delivery,” not “update any field on any Work Order.” That action is where we enforce validation, authority, transaction boundaries, locking behavior, logging and, in module 3, safe retries. An agent doesn’t get permission to ignore the architecture because it has AI in the name.

Test knowledge. Know the standard ways to reduce lock contention and when each fits: remove or relieve the shared parent (distribute children, avoid roll-ups on very busy parents, or move to a lookup with roll-ups calculated later); shorten the transaction (move non-essential work to asynchronous processing, accepting eventual consistency); group work by parent (sort loads by parent ID so one parent’s children aren’t spread across parallel batches; for platform event triggers, a partition key that keeps events for the same record in the same partition); and reduce concurrency (Bulk API serial mode, fewer parallel workers). Retrying with backoff handles what remains; it doesn’t remove the cause.

Beyond the exam: what AI changes. Agents multiply the writers, and they retry without getting tired. Give them focused update scope through established, tested actions, so the levers in this module apply to them too.

What an Architect Decides Here

• Where competing transactions share a lock, and whether that shared record, or what each save does to it, is actually necessary.
• Whether relationship behavior, roll-ups or synchronous automation should be redesigned to shrink the footprint.
• Which work must complete synchronously and which can move later, with the eventual consistency that brings.
• Whether the workload should run fully parallel, one at a time, or partitioned so only competing work waits.
• Where a deliberate lock is necessary, and how narrow it can be.

Don’t Tune Around a Bad Collision

UNABLE_TO_LOCK_ROW is the symptom. The architecture question is: why are these transactions fighting over the same thing? Find the shared lock. Then work outward: can we remove it? Touch it less? Hold it for less time? Separate the work that doesn’t really compete? Stop launching competing work at the same instant? At Steve Trucking, the answers were: keep the Manifest, stop recounting it inside every save, and give each Manifest its own line.

Only after that do I ask what happens when a collision still occurs. Because some will. Networks time out. Transactions collide. Agents retry. That’s normal. The next question is whether the operation can survive that retry without doing the business action twice. That’s module 3.

Next: Safe to Retry

Explore

• Salesforce Developers blog: Analyze Concurrent Errors in Scale Center (opens in a new tab) (2025) and Salesforce Scale Center Is Generally Available (opens in a new tab) (2023), checked 2026-11-01; Salesforce Admins blog: Empowering Admins: Build Org Scalability With New Scale Center Features (opens in a new tab) (February 2026, the editions), checked 2026-11-01.
• Salesforce, Platform Events Developer Guide: Subscribe to Platform Event Notifications with Apex Triggers (opens in a new tab), Platform Event Processing at Scale with Parallel Subscriptions for Apex Triggers (opens in a new tab) and Choosing a Partition Key (opens in a new tab), checked 2026-11-01.
• Salesforce Well-Architected: Resource and Cost Optimization (opens in a new tab) (data skew, asynchronous trade-offs, Scale Center) and Reliability (opens in a new tab) (partitioning, exponential backoff), checked 2026-11-01.
• Salesforce, Platform Events Developer Guide: Publish Event Messages with Apex (opens in a new tab) and Parallel Subscription Configuration Examples in Tooling API and Metadata API (opens in a new tab); Apex Developer Guide: Avoiding Deadlocks (opens in a new tab), checked 2026-11-01.
• Salesforce, Apex Reference Guide: Database Class (opens in a new tab) (executeBatch and its scope), and Best Practices for Deployments with Large Data Volumes: Loading Data from the API (opens in a new tab) (group child records by parent), Winter ‘27, checked 2026-11-01.
• Salesforce engineering blog: Salesforce Platform Record Locking and Concurrency (opens in a new tab) (the Record Locking Cheatsheet, 2014), checked 2026-11-01.