Your Event Schema Is a Public API

⏱ 18 min read

There’s a mistake teams make when they first get serious about event-driven architecture. It’s an understandable mistake, and a dangerous one.

They think the event is an internal implementation detail.

It isn’t.

The moment an event leaves a service boundary—published to Kafka, fanned out through a streaming platform, consumed by another team, loaded into a lakehouse, replayed into a new projection, inspected by auditors, or used by operations during an outage—that schema stops being “just JSON” or “just Avro.” It becomes a public API. And public APIs have a long memory. event-driven architecture patterns

That is the heart of the problem. Teams adopt events to decouple runtime dependencies, then accidentally create semantic dependencies that are harder to unwind than REST contracts ever were. A synchronous API breaks immediately and loudly. An event schema breaks slowly, quietly, and in production three months later when someone replays a topic into a new service and discovers that customerStatus changed meaning twice and nobody wrote it down.

Events are not messages in the casual sense. They are historical statements about the domain. If you publish them carelessly, you are not merely shipping data. You are publishing language. And language, once shared, is governance whether you admit it or not. EA governance checklist

This is why event schema evolution is not a serialization problem. It is a domain design problem with operational consequences.

Context

Most enterprises arrive at event-driven architecture through pain, not fashion. They are trying to break apart a monolith, integrate acquired systems, reduce synchronous coupling, support real-time decisions, or give downstream analytics something better than nightly ETL. Kafka appears, often sensibly, as the central nervous system. Microservices begin to emit events. A schema registry appears. A few standards are written. Teams feel modern. microservices architecture diagrams

Then reality arrives.

One service publishes OrderCreated. Another team consumes it to reserve stock. A third uses it for customer notifications. A fourth stores it for audit. Six months later the order domain changes. “Created” is no longer a single business moment; there is draft, submitted, approved, and accepted. One team wants to add fields. Another wants to rename fields. A third wants to split one event into three. Someone proposes versioning in the topic name. Someone else says “just make fields optional.” Data engineers ask whether replay still works. Compliance asks if historical events remain legally interpretable. Operations asks what happens when consumers lag during a dual-publish migration.

This is not a tooling defect. It is what happens when a business vocabulary starts carrying load.

The deeper point is that event schemas sit at the intersection of three architectural concerns:

  • Domain semantics: what happened, in whose language, and with what business meaning.
  • Compatibility evolution: how the schema changes without breaking producers, consumers, storage, and replay.
  • Migration: how an enterprise moves from old meanings to new ones without stopping the world.

If you don’t design for all three, Kafka simply gives you a very efficient way to spread confusion.

Problem

A public event schema fails in more ways than a request-response API.

With REST, consumers call a provider directly. Ownership is clearer. Versioning is visible. Testing often happens at the integration point. With events, a producer may not even know all consumers. Some are official. Some are “temporary.” Some are hidden in reporting pipelines, fraud engines, notebooks, or compliance exports. The event is copied, transformed, retained, replayed, and repurposed. In large enterprises, it becomes institutional sediment.

That creates four common pathologies.

First, schema-first thinking without domain-first thinking. Teams argue about Avro compatibility modes while publishing events whose names are really CRUD notifications: CustomerUpdated, OrderChanged, ProductModified. These are integration exhaust fumes, not business events. They reveal database movement rather than domain intent. They age badly because downstream consumers infer business meaning from structural deltas.

Second, semantic drift under stable syntax. A field can remain compatible at the serializer level while becoming incompatible in meaning. status = ACTIVE may once have meant “eligible to transact” and later mean “identity verified.” Both are strings. Both pass schema checks. One destroys trust.

Third, breaking evolution disguised as additive change. Teams are told to “only add optional fields.” Fine advice as far as it goes, but weak in practice. New fields often imply new invariants. A consumer built on old assumptions may behave incorrectly, not fail loudly. Quiet wrongness is the worst failure mode in enterprise systems.

Fourth, migration fantasy. Architects draw the target event model and imagine a clean cutover. Enterprises do not cut over cleanly. They dual-run. They reconcile. They replay. They discover edge cases. They keep old consumers alive longer than anyone planned. Event evolution without migration reasoning is architecture as illustration.

Forces

This is where architecture earns its keep: not by declaring purity, but by balancing ugly forces.

Stability versus domain truth

You want stable contracts because change is expensive. But the domain changes because the business changes. If you freeze events too early, they become lies. If you change them too freely, they become chaos. The job is not to avoid change. The job is to make change survivable.

Producer autonomy versus shared language

Microservice teams want independence. Fair enough. But events are shared assets. A team can own publication, but it does not own interpretation once the event is public. This is why domain-driven design matters here. Events should arise from bounded contexts, with explicit language and ownership, not from database tables with marketing names.

Backward compatibility versus semantic cleanliness

You can keep adding fields and preserve binary compatibility. Over time you accumulate a fossil bed of deprecated semantics, duplicate fields, and “do not use” documentation that nobody reads. Sometimes the right answer is not more compatibility. Sometimes it is a new event, a new stream, or a new context map.

Replayability versus real-time simplicity

A schema that works for live consumers may fail for replay, backfill, audit, and analytics. Reprocessing old events against new code is where semantic shortcuts come home to roost.

Governance versus speed

Too much central governance and teams route around it. Too little and you get twenty variants of customer identity emitted across the estate. The right governance is lightweight, domain-aware, and ruthless about meaning. ArchiMate for governance

Solution

Treat every externally consumed event schema as a public, versioned domain contract. Design it with the same seriousness you would give a partner-facing API—arguably more, because its consumers are harder to inventory and easier to surprise.

That leads to a handful of principles.

1. Model events as domain facts, not record mutations

In domain-driven design terms, an event should express something meaningful in the ubiquitous language of its bounded context. OrderSubmitted is better than OrderUpdated. PaymentAuthorized is better than PaymentStatusChanged. A good event tells the business story of a state transition. A bad event reports that some columns moved.

This matters because domain facts evolve more gracefully. They carry intent. Consumers can reason about them without reverse-engineering your aggregate internals.

2. Distinguish schema compatibility from semantic compatibility

Your registry can validate backward, forward, or full compatibility at the structural level. Useful, but insufficient.

You also need semantic compatibility rules, such as:

  • existing fields keep their business meaning
  • units do not silently change
  • enumerations are extended carefully
  • nullability changes are treated as business-impacting
  • identifiers remain stable in scope and meaning
  • timestamps preserve event time versus processing time semantics

Put bluntly: if effectiveDate switches from local business date to UTC timestamp, that is a breaking change even if every serializer on earth accepts it.

3. Prefer additive change, but don’t worship it

Additive evolution is the safest default:

  • add optional fields
  • add new event types for new business moments
  • deprecate rather than rename
  • preserve existing required fields if they remain valid

But additive change is a tactic, not a religion. If the old event is semantically wrong, stop extending the lie. Introduce a new event. Keep the old one alive during migration. Then retire it deliberately.

4. Separate internal models from published contracts

Your aggregate, database schema, and internal command model should not leak directly into published events. Use an anti-corruption layer or mapping layer at the service boundary. This gives you room to refactor internals without forcing downstream change.

This is one of those unglamorous decisions that pays for itself every quarter.

5. Design for migration on day one

Every public event will evolve. Assume dual-publishing, consumer adaptation, replay, and reconciliation will happen. Build observability, lineage, schema governance, and version deprecation into the platform before the first major migration, not during it.

Architecture

A healthy event architecture has clear boundaries, contract ownership, and explicit compatibility controls.

Architecture
Architecture

The critical element in that diagram is not Kafka. It is the event mapping layer between the bounded context and the public stream. That is where internal state is translated into domain events with stable semantics.

A practical event envelope often includes:

  • event type
  • schema version
  • event ID
  • aggregate or business entity ID
  • event timestamp
  • producer context
  • correlation/causation IDs
  • payload
  • optional trace metadata

But don’t overdo generic envelopes to the point of hiding the domain. Shared metadata is useful. Shared vagueness is not.

Compatibility evolution

There are three broad evolution moves in the real world.

  1. Compatible enrichment

    2. Add fields that are genuinely optional and semantically independent.

  1. Compatible branching

    2. Introduce a new event type for a new domain fact while continuing to emit the old type where still valid.

  1. Incompatible replacement

    2. Publish a new contract and migrate consumers progressively. This is more expensive but often cleaner.

Here is the compatibility evolution idea in a form most teams can actually reason about:

Diagram 2
Compatibility evolution

The key decision point is not “can the schema registry accept this?” It is “is the meaning still intact?”

That question is architectural. It requires domain stewardship, not just CI checks.

Bounded contexts and semantic ownership

An event should be owned by the bounded context whose language it represents. Customer identity should not be defined independently by every service that happens to know a customer ID. Likewise, a fulfillment team should not casually redefine what an order submission means just because they subscribe to order events.

This is where context mapping from domain-driven design helps. Published language should be explicit. Downstream services may conform, translate, or use anti-corruption layers. They should not quietly fork semantics and still call it the same thing.

Migration Strategy

Migration is where elegant event models meet enterprise reality and usually lose the first round.

The right pattern, in most large estates, is a progressive strangler migration. Not a big-bang cutover. Not “everyone move by Q3.” A managed transition in which old and new contracts coexist while consumers shift incrementally.

Step 1: Classify consumers

Before changing a public event, inventory consumers in categories:

  • operational microservices
  • analytical pipelines
  • reporting extracts
  • audit and compliance stores
  • machine learning feature pipelines
  • ad hoc or hidden consumers

This sounds obvious. It rarely is. Hidden consumers are the tax you pay for successful event streams.

Step 2: Introduce the new contract

Create the new event type or version with explicit semantic documentation:

  • what business fact it represents
  • what changed from the old event
  • field-level meaning
  • migration guidance
  • deprecation timeline

If the change is semantic, prefer a new event name over a silent version bump. Renaming reality is cleaner than pretending continuity where none exists.

Step 3: Dual publish

Publish both old and new events from the source boundary. Keep publication logic side by side, with clear ownership and tests. This is temporary complexity used to buy safe migration.

Step 4: Migrate consumers incrementally

Move high-value, low-risk consumers first. Leave brittle legacy consumers behind a translation layer if needed. Some consumers will need an anti-corruption adapter that maps the new event back into the old shape while they are being retired.

Step 5: Reconcile

This is the step architecture diagrams often omit because it looks mundane. It is not mundane. It is survival.

During migration, run reconciliation jobs and dashboards to compare:

  • event counts by type
  • entity-level correspondence between old and new streams
  • timing differences
  • missing or duplicate publications
  • downstream projection divergence

If you do not reconcile, you are not migrating; you are guessing.

Step 6: Replay and prove

Before retiring the old contract, replay historical data into at least one representative consumer or projection built on the new contract. This is how you discover semantic edge cases, not by reading Confluence.

Step 7: Deprecate and remove

Deprecation needs dates, owners, and enforcement. “Deprecated” without a retirement mechanism is just a museum label.

Here is a realistic migration shape:

Step 7: Deprecate and remove
Deprecate and remove

A note on topic versioning

Teams often ask whether to version in the schema, the event name, or the Kafka topic.

My opinion: avoid topic-per-version unless the change really represents a new stream with different retention, security, throughput, or ownership characteristics. Topic proliferation creates operational drag. If semantics are continuous and compatibility is preserved, evolve within the same stream. If semantics change materially, prefer a new event type and sometimes a new stream. The point is clarity, not dogma.

Enterprise Example

Consider a global retailer modernizing order processing across ecommerce, stores, and marketplace partners.

Originally, the monolith emitted a CDC-style integration event into Kafka:

  • OrderCreated
  • OrderUpdated
  • OrderCancelled

Downstream consumers multiplied:

  • warehouse allocation
  • customer email notifications
  • finance settlement
  • fraud scoring
  • customer service timeline
  • analytics and BI

The trouble started when the business introduced marketplace orders and manual approval for high-risk purchases. In the old world, “created” meant the order was ready for fulfillment. In the new world, an order could be drafted, submitted, risk-reviewed, approved, then accepted for fulfillment.

The team’s first instinct was typical: add fields to OrderCreated such as approvalStatus, channel, and fulfillmentEligibility. Structurally this was backward compatible. Semantically it was a mess. Existing warehouse consumers still treated OrderCreated as actionable. Fraud wanted a pre-approval signal. Customer service needed a timeline. Finance only cared after acceptance.

So the architecture was redesigned around domain semantics:

  • OrderSubmitted
  • OrderRiskReviewed
  • OrderApproved
  • OrderAcceptedForFulfillment
  • OrderCancelled

The old OrderCreated continued to be published for legacy consumers, but it was marked as a transitional compatibility event, not the canonical domain fact.

A mapping layer inside the order service emitted both contracts. New consumers adopted the explicit events. Legacy warehouse integrations kept reading OrderCreated through an adapter until they were replaced. A reconciliation service compared entity lifecycles across both representations and flagged mismatches.

What did they learn?

First, they discovered analytics pipelines were the hardest consumers to migrate because they had encoded business assumptions in SQL scattered across teams.

Second, replay surfaced historical ambiguity: some old orders had no clear distinction between “created” and “approved.” The migration required domain decisions, not just code changes. They had to define approximation rules and document them.

Third, dual publishing increased operational cost for six months, but it prevented a fulfillment outage during peak season. That is the kind of tradeoff serious architecture makes willingly.

This is the enterprise reality: event evolution is as much about institutional memory as software.

Operational Considerations

The platform side matters. If your operating model is weak, your contract discipline will collapse under pressure.

Schema registry and CI gates

Use a schema registry, yes. Enforce compatibility modes, yes. But pair that with semantic review for public events. A passing compatibility check should not be mistaken for approval.

Observability

Track:

  • producer publish rates and failures
  • consumer lag by contract
  • schema version adoption
  • dead-letter volume
  • reconciliation mismatch rates
  • replay duration and error profiles

During migration, these become executive metrics, not just engineering metrics.

Data retention and replay

Retention policies should reflect business needs, not just broker cost. If events are used for audit, rebuild, and backfill, short retention without durable archival is architectural negligence.

Idempotency and ordering

Consumers must assume duplicates and partial disorder unless your design gives stronger guarantees and you have tested them under failure. A public event API without idempotent consumers is an invitation to accidental side effects.

Documentation as product

For public events, documentation should cover:

  • business meaning
  • field semantics
  • examples
  • lifecycle expectations
  • ordering assumptions
  • deprecation policy
  • migration notes

The best event catalogs read like product docs, not serializer dumps.

Tradeoffs

There is no free lunch here. The good patterns cost something.

Separate published contracts add complexity

A mapping layer and explicit domain events are more work than dumping entity changes to Kafka. But that “simplicity” merely pushes complexity downstream, where it multiplies.

Dual publishing is expensive

It adds code, testing, observability, and support burden. But it is usually cheaper than breaking unknown consumers in a large enterprise.

Strong semantic governance slows local teams

Correct. It should. Public language deserves friction. If a team wants total freedom, they should keep the event private.

Rich domain events can be harder for generic platforms

Data teams sometimes prefer flatter, more uniform structures. Fair concern. Solve it with derived analytical models and stream processing, not by degrading the operational contract into table change noise.

Failure Modes

This is where architectures usually die: not at design time, but in the gap between what was promised and what was actually governed.

The “optional field” trap

A producer adds an optional field that actually changes interpretation. Old consumers don’t break. They just become wrong.

Semantic overload

One event name accumulates too many business scenarios. Soon every consumer implements a matrix of conditions to determine what the event “really means.” That is not decoupling. That is distributed archaeology.

Hidden consumer breakage

A reporting job, lake ingestion process, or compliance extractor silently depends on a field or meaning that was “internal.” There is no such thing as internal once the stream is public enough.

Irreversible replay surprises

Historical events cannot be interpreted under the new model without special-case logic. The migration looked fine in forward flow and failed in rebuild.

Topic sprawl as faux governance

Every change gets a new topic. Discovery becomes impossible. Consumers subscribe to several generations at once. Operational overhead explodes.

Ownership ambiguity

No one knows who can approve changes to a public event. In that vacuum, whichever team deploys first effectively governs by accident.

When Not To Use

Not every integration deserves a public event contract.

Do not publish broad enterprise events when:

  • there are no real external consumers and no replay need
  • the data is purely internal and volatile
  • the domain language is immature and changing weekly
  • the interaction is fundamentally request-response and needs immediate consistency
  • teams lack the operational discipline for schema governance and migration

This is worth saying plainly: if your organization cannot manage contract ownership, consumer discovery, deprecation, and reconciliation, a shared event stream may be worse than a well-designed API.

Also, don’t force domain events where a data replication use case is better served by CDC into a controlled integration zone. CDC has its place. Just don’t confuse database facts with business facts.

Several patterns sit naturally around this approach.

Event storming

Useful for discovering domain events and surfacing language before schemas harden.

Anti-corruption layer

Essential when consumers or legacy systems cannot adopt the new domain model directly.

Outbox pattern

A good choice for reliable event publication from transactional systems, especially during migration where consistency between state change and emitted events matters.

Strangler fig pattern

The right mental model for progressive replacement of legacy event contracts and legacy consumers.

CQRS and projections

Helpful when consumers need read models derived from canonical domain events rather than raw integration payloads.

Data products and event catalogs

Useful for treating public event streams as governed assets with discoverability, ownership, and lifecycle management.

Summary

An event schema is not a technical artifact with a bit of JSON wrapped around it. It is a public promise about the meaning of things that happened in your business.

That promise must survive team boundaries, consumer drift, Kafka retention windows, microservice rewrites, replay, audit, and migration. It must survive success. Success is what creates hidden consumers and long-lived dependencies.

So treat event schemas like public APIs, with one extra layer of seriousness: they are APIs made of history. You can patch an endpoint. You cannot unpublish yesterday.

Use domain-driven design to anchor events in bounded contexts and ubiquitous language. Distinguish structural compatibility from semantic compatibility. Prefer additive change when meaning remains intact, but don’t be afraid to introduce new events when the domain truth changes. Separate internal models from published contracts. Migrate with a progressive strangler strategy. Reconcile relentlessly. Replay before you declare victory.

And above all, remember this: in enterprise architecture, the most expensive bugs are not syntax errors. They are shared misunderstandings with excellent throughput.

Frequently Asked Questions

What is API-first design?

API-first means designing the API contract before writing implementation code. The API becomes the source of truth for how services interact, enabling parallel development, better governance, and stable consumer contracts even as implementations evolve.

When should you use gRPC instead of REST?

Use gRPC for internal service-to-service communication where you need high throughput, strict typing, bidirectional streaming, or low latency. Use REST for public APIs, browser clients, or when broad tooling compatibility matters more than performance.

How do you govern APIs at enterprise scale?

Enterprise API governance requires a portal/catalogue, design standards (naming, versioning, error handling), runtime controls (gateway policies, rate limiting, observability), and ownership accountability. Automated linting and compliance checking is essential beyond ~20 APIs.