Capturing Truth — and the Schema-Evolution Myth

The cleanest place to see the choice — and the cost of getting it wrong — is a phenomenon engineers treat as an inevitable fact of life: schema evolution. Change a field name, add a required property, remove something that downstream consumers depend on, and suddenly you have a coordination problem spanning multiple teams, multiple services, and potentially years of stored events.

Entire tools exist to manage this. Schema registries. Compatibility rules — backward, forward, full. Versioned event types. Migration frameworks. The complexity is real, and the tooling that has grown up around it is sophisticated and carefully designed.

But there is a question that rarely gets asked: why does the schema need to change in the first place?

Two ways to define an event

There are two fundamentally different ways to define an event.

The first way is to design it. You look at what downstream consumers need, gather their requirements, and produce an event schema that satisfies them — often a bundled event in disguise. An OrderPlaced event might contain the customer’s shipping address, the items with their prices and descriptions, applied discount codes, the payment method, and a dozen other fields that various consumers find useful.

The second way is to capture it. You ask what actually happened, at the finest grain at which it happened, in the language of the person or process that made it happen. An OrderPlaced event in this model contains almost nothing: an order identifier, a customer identifier, a timestamp. Because those are the irreducible facts of the act of placing an order. Everything else is context that existed before the order was placed, or consequence that followed from it.

The designed event feels richer. The captured event feels sparse. But they have radically different properties when requirements change.

Why designed events need to evolve

A designed event is a snapshot of your current understanding of what consumers need. That understanding changes constantly.

A new consumer needs a field that the current schema doesn’t include. A regulatory requirement introduces new mandatory data. A fraud detection system needs something that was never considered. An existing consumer changes how it uses a field.

Each of these is a reason to change the schema. And changing the schema creates a coordination problem: existing stored events don’t have the new fields, old consumers don’t understand new fields, the system has to handle multiple versions of the same event type simultaneously.

This is the schema evolution problem. It is real, it is painful, and it is the direct consequence of designing events around consumer needs rather than around facts.

Why captured events almost never need to evolve

Bundling as lossy compression

The OrderPlaced event that contains twenty fields is not capturing one fact. It is bundling many decisions into a single structure.

Consider what actually happened when an order was placed. The customer added an item — a choice. They removed it and added a different one — another choice. They selected a shipping tier — a choice. They applied a discount code — a choice. They reviewed the summary and confirmed — the final commitment.

Each of these is a separate, independent fact. Each could have happened differently. Each is a discrete moment in time.

When you bundle them into a single OrderPlaced event, you are performing lossy compression. The compression is optimised for the consumers you are aware of today. It discards the granularity that future consumers — consumers you haven’t imagined yet — might need.

This is premature optimisation in the most literal sense: optimising for known consumers before knowing what all consumers will need, at the cost of information that cannot be recovered.

The uncompressed version — individual events for each decision — is strictly more informative. You can always derive the bundle from the atoms. You cannot recover the atoms from the bundle.

AddedToCart       { orderId, productId, quantity, priceAtTime }
RemovedFromCart   { orderId, productId, quantity }
ShippingSelected  { orderId, tierId, quotedCost }
DiscountApplied   { orderId, code, discountAmount }
OrderSubmitted    { orderId }

OrderSubmitted is the last atomic decision — the moment of commitment. It is not a summary of the decisions that preceded it. Those decisions are already in the log. OrderSubmitted says only: at this moment, the customer committed. Everything before it remains visible, queryable, and available to any consumer that needs it.

The order document is not the ground truth

A legally binding order document — a PDF, a contract, a formal record — is immutable. Once issued, it does not change. This might seem like a counterexample: surely the order document is the ground truth?

It is not. It is a materialised projection, frozen at the moment of submission.

The ground truth is the sequence of decisions that led to it. The document is a human-readable rendering of the state at a specific point in time, produced for a specific purpose — legal validity, human readability, physical delivery.

Both are immutable. Neither invalidates the other. They are truths at different levels of granularity, for different audiences, with different purposes.

The document can be derived from the event log at any time and will always be identical. You could choose not to store the document at all and re-derive it on demand — and it would be provably correct because the log is the source. Storing the document is a performance optimisation. Re-deriving it on demand is the proof of correctness.

This is what makes the framework coherent: you never have to choose between the atomic events and the aggregate document. The atomic events are primary. The document is derived. Both exist. Neither changes. The relationship between them is permanent and traceable.

When schema change is genuinely necessary

The argument is not that schemas never change. It is that schema change is almost always a symptom of having drawn the truth boundary incorrectly.

There are genuine cases where the original capture was wrong — not incomplete, but incorrect. A timestamp captured in the wrong timezone. An identifier that turns out not to be globally unique. A field that conflates two things that should have been separate. These are errors in observation, not evolution of needs. The response is a correcting event — OrderTimestampCorrected — which is itself a new fact: a correction was made, at this moment, for this reason. The original event remains. The correction is appended.

And there are cases where the unit of truth was drawn at the wrong boundary. PaymentProcessed that conflates authorisation and capture is an example. Separating them is not schema evolution. It is a better identification of what the atoms actually are.

The practical test is simple: is this change driven by consumer needs, or by a better understanding of what actually happened? If consumer needs, the answer is a new event, not a changed one. If a better understanding of reality, the existing event was capturing the wrong thing, and the correction is warranted.

The practical implication