Build for Today. Plan for Tomorrow. Plan for Failure.

🌐 Also in 28 languages

Español · 日本語 · Português (Brasil) · Français · Deutsch · Italiano · 한국어 · 简体中文 · Português (Portugal) · 繁體中文 · Bahasa Indonesia · Türkçe · Polski · Tiếng Việt · ไทย · Nederlands · العربية · עברית · Українська · हिन्दी · Русский · Svenska · Dansk · Norsk bokmål · Suomi · Español (México) · Français (Canada) · Deutsch (Schweiz)

Image generated with AI

Module 5 of 7 in the chapter Thinking Like a Data Architect, the first chapter of the Data Architect course. Each module stands on its own. Previous: Architecture Is the Art of Trade-offs · Next: Know Your Tools: The Nine Questions

The question this module answers: How do you design for today’s requirements while planning for growth and for failure?

One of the easiest mistakes in architecture is designing only for the requirements sitting in front of you. We have a project, a backlog and requirements. We build something that meets them. It works, and we declare victory. Six months later the business changes, or the data grows, or another business unit wants to use it, or we acquire a company, or a new system needs to integrate, or something fails that nobody planned for.

None of that should surprise an architect. We can’t predict everything that’s going to happen, but we can design knowing that something will. That’s why I look at architecture through three lenses:

Build for today. Plan for tomorrow. Plan for failure.

You need all three.

Click to enlarge

Build for Today

Architecture starts with the problem we actually have. That sounds obvious, but architects can make the opposite mistake from short-term thinking: we can over-architect. What if we acquire twelve companies, have a billion customers, expand into 40 countries, and every system needs real-time integration? Maybe. But are those real possibilities, or are we inventing problems? Architecture isn’t about building the most sophisticated system we can imagine. It’s about building the appropriate one.

So start with today: what the business needs, the actual volumes, users and transactions, how often the data changes and how fast it has to move, who needs access, and where the data comes from and goes. And most importantly: why? Those answers give us something concrete to design.

Look for the Implied Requirements

The requirements you’re given aren’t necessarily all the requirements you have. There are almost always implied requirements.

The business says, “We need customer data in Salesforce.” Okay. Then someone has to own that data and decide how fresh it must be. We need to know what happens when the source system is down, how we’ll notice when synchronization fails, and which side wins when Salesforce and the source disagree. We need a retention period, an access model, a decision about sensitive fields, and a way to correct bad data.

None of that was in the user story. It’s all still architecture. A big part of becoming an architect is learning to recognize the questions that haven’t been asked yet.

Know Your Numbers

I constantly push architects to quantify. Don’t tell me there’s a lot of data; tell me how much. Ten thousand, ten million, a hundred million, a billion? Those are very different architectures. Don’t tell me the integration runs frequently: every hour, every minute, a thousand times a second? Don’t tell me we’re growing quickly. How quickly?

When I don’t have a better number, my baseline is 30 percent annual growth. It’s not a prediction; it’s a forcing function. If someone tells me we have 10 million records today, I don’t want the discussion to stop at 10 million. At 30 percent a year we’re at roughly 13 million next year, 17 million the year after, and around 22 million the year after that. Now let’s talk about the design. If the business tells me 30 percent is ridiculous, great: give me the real number. The point is to make growth part of the architecture discussion.

I call this Steve’s Number of Zeros. One extra zero can completely change your architecture. Ten thousand, a hundred thousand, a million, ten million, a hundred million: the feature didn’t change, and the requirement may not have changed, but the architecture might.

Remember the appointments from module 1? Let’s put the real numbers on them. Over 1,200 locations, 100 appointments a location a day, six days a week, about 305 days a year. That’s over 36 million appointments in the first year. Each visit creates two records, so over 73 million records. Now keep five years of history, growing 30 percent a year, and it’s over 650 million records. Same feature, same requirement. A couple more zeros, and a very different architecture.

Plan for Tomorrow

Once I understand today, I look forward: what is reasonably likely to change? Volume is part of it, but tomorrow isn’t just more. It can also mean different: more products, business units and countries; different privacy requirements; new channels and integrations; a merger or acquisition; a new master-data strategy or analytics platform; an AI agent consuming information that was designed only for humans.

That’s why I care so much about whether the architecture can grow through addition rather than constant mutation. If the business adds a product, can the model take it? If another system needs the data, can we expose it cleanly? If volume jumps, do we have somewhere to go? If the business structure changes, will we have to tear the data model apart? We aren’t trying to predict the future perfectly. We’re trying not to paint ourselves into a corner.

Don’t Build Tomorrow Today

There’s a balance here. Planning for tomorrow doesn’t mean building tomorrow’s system today. If we have 100,000 records and might have 100 million in five years, that doesn’t mean we build the 100-million-record architecture now. It may bring cost and complexity we don’t need yet.

The question I want answered is: can today’s architecture evolve into tomorrow’s? If we can start simply with a clear path forward, that’s often better than paying the complexity cost years early. The danger isn’t starting small. The danger is starting somewhere with nowhere to go.

Plan for Failure

Now the part nobody likes talking about: things fail. I’ve never designed a system where every component stayed available forever, every integration succeeded, every user entered perfect data, and every external system behaved exactly as documented. So don’t design that system.

Ask what can fail, then ask what happens when it does. An integration times out. An API rejects a request. A nightly load processes 800,000 records and fails on 200,000. A source system goes down. A user enters invalid data. An external identifier turns up twice. An upstream system changes the format of a value without changing the field name. A process succeeds in one system and fails in another. These aren’t unusual edge cases. This is what production looks like.

Partial Success Is Still Failure You Need to Understand

Partial failures are especially dangerous in data architecture. Imagine moving a million records, and the process reports that it completed. Great. But did a million arrive, or 999,997, and which three are missing? Maybe 200,000 failed, some were created twice, or everything arrived and some fields failed transformation.

“Job completed” isn’t the same thing as the data is correct. A data architect thinks about reconciliation: how do we know what should have happened, what actually happened, and the difference? And what do we do about it?

Here’s what I do when the systems get complex. Records go from A to B, left to right. Then B sends something back: a confirmation for every record it received and saved. Now A can compare what it sent with what was confirmed. If every record has its confirmation, the run succeeded. If any confirmations are missing, it didn’t, whatever the job status says, and we know exactly which records to go after: retry them, route them to a person, or raise an alert.

Click to enlarge

Fail Gracefully

When failure happens, I want to know what the system does next: whether it retries, and automatically or not; what happens if the same transaction is processed twice; where failed records go, and whether good records keep flowing past a bad one; who gets alerted, with what information, and how they replay the failure; and how a user finds out something went wrong.

And most importantly: can we recover without making the problem worse? A retry strategy that creates duplicate data isn’t much of a recovery strategy. Good architecture assumes failure and gives us a way back.

Monitor What Matters

Planning for failure without monitoring isn’t really planning. If an integration fails at 2:00 AM and nobody notices until a customer calls three days later, we didn’t have an operational architecture. We had hope.

Decide what needs to be observable: did the process run, how long did it take, how many records were processed, how many succeeded and failed? Are volumes suddenly different from normal? Is the backlog growing, are errors increasing? Architecture doesn’t end when the deployment succeeds. Someone has to operate this system tomorrow, and that’s part of the design.

Think in All Three Directions

Keep applying the three lenses through the rest of this course. When we design a data model: does it work today, can it grow tomorrow, and what happens when the data is wrong or incomplete? An integration: does it move today’s data, can it handle tomorrow’s volume and systems, and what happens when one side is down? Large data volumes: what do we have today, where are the numbers heading, and how do we recover when a large operation only partly succeeds?

The technology changes from chapter to chapter. The questions don’t.

Architecture Exists Across Time

A developer can sometimes look at a requirement and ask, “Does this work?” The architect has to ask more: does it work today, will it still work tomorrow, and what happens when it doesn’t work at all? Those three questions take us beyond implementation and into the life of the system.

You won’t predict every future requirement or prevent every failure. That’s not the goal. The goal is to build something appropriate for what we know today, leave ourselves somewhere to go tomorrow, and have a plan for when reality doesn’t cooperate.

Build for today. Plan for tomorrow. Plan for failure. We’ll use those three lenses again and again.

What an Architect Decides Here

• What the real volumes and growth are, and which numbers would change the design.
• Which implied requirements the business hasn’t stated yet.
• How today’s design can evolve into tomorrow’s, without building tomorrow’s system today.
• What happens when something fails: recovery, reconciliation and monitoring.