⏱ 20 min read
There is a lie we tell ourselves when modernizing data platforms.
We say the old world was tangled, brittle, and slow. Then we move to cloud storage, streaming pipelines, microservices, Kafka, domain-aligned data products, and shiny orchestration layers—and we expect simplicity as a reward for courage. event-driven architecture patterns
But complexity is stubborn. It rarely disappears. It migrates.
Take a monolithic enterprise data warehouse and replace it with a lakehouse, a mesh of services, event streams, reverse ETL, and half a dozen operational stores. The nightly batch jobs may vanish. The giant SQL scripts may be broken apart. The centralized ingestion team may no longer be the bottleneck. Yet the essential difficulty remains: the business is still messy, the semantics are still contested, and systems still disagree about reality. What changed is the address of the pain.
This is the central architectural truth of modern data platforms: your data platform complexity moved. It did not shrink.
That is not a reason to avoid modernization. It is a reason to do it with your eyes open.
The good architectures are not the ones that promise the removal of complexity. They are the ones that put complexity where the business can afford it. They move it from hidden corners into explicit contracts. They shift it from fragile procedural glue into bounded contexts, event models, reconciliation processes, and operational guardrails. They trade one kind of pain for another, ideally a pain that scales with the enterprise instead of against it.
This is where architecture earns its keep.
Context
Many enterprises grew their data landscape in layers, each sensible at the time.
First came operational systems: ERP, CRM, policy administration, order management, claims, fulfillment, billing. Then came integration: ETL jobs, point-to-point mappings, enterprise service buses, file transfers. After that came analytics centralization: enterprise data warehouses, conformed dimensions, master data hubs, reporting marts. Then digital products arrived, and with them APIs, microservices, Kafka, customer-facing apps, machine learning features, and the need for data to move continuously rather than nightly. microservices architecture diagrams
Soon enough, the old “single version of truth” platform started to buckle under new demands. Operational teams wanted event-driven integration. Product teams wanted domain ownership. Data scientists wanted raw history. Regulators wanted lineage. Executives wanted real-time dashboards. Finance still wanted numbers to tie out exactly at quarter end.
So the enterprise modernized.
Data lakes became lakehouses. Batch became stream-plus-batch. Central ETL became self-service pipelines. Schemas became contracts. Warehouse teams became platform teams. Data was repackaged into products. Some organizations embraced data mesh language; others simply distributed ownership without adopting the full vocabulary. Meanwhile, Kafka became the bloodstream for events, and microservices became the operational fabric.
This is progress. Often necessary progress.
But replacing a centralized warehouse with a distributed data ecosystem does not remove complexity. It exchanges centralized coordination problems for distributed semantic problems. And distributed problems have a nasty habit: they fail in more creative ways.
Problem
The old platform’s complexity was visible.
You could point to the nightly jobs that overran. You could name the giant transformation layer everyone was afraid to touch. You could see the reporting backlog, the overloaded warehouse team, the duplicate extracts, and the fragile source-to-target mappings. Complexity sat in one place like a rusted machine: ugly, loud, but obvious.
In the new world, complexity fragments.
Now there are event topics, stream processors, CDC feeds, object storage zones, semantic layers, operational APIs, multiple serving stores, data contracts, domain ownership boundaries, and observability tools trying to explain what happened at 02:13 UTC. Instead of one overloaded warehouse team, there are ten domain teams, each making reasonable local decisions that may or may not add up globally.
The hardest part is semantic drift.
“Customer” in sales is not “customer” in billing. “Order” in e-commerce is not “order” in fulfillment. “Policy” in underwriting is not “policy” in claims. A centralized warehouse used to hide these differences behind canonical models and painful governance committees. A distributed architecture surfaces them. That is healthy, but expensive. Once domains publish events and data products, every semantic decision becomes architecture. EA governance checklist
The platform challenge is no longer just moving and storing data. It is preserving meaning while allowing autonomy.
That sounds elegant in a slide deck. In production, it means arguing over event granularity, identity resolution, late-arriving facts, replay semantics, change data capture ordering, exactly-once illusions, and whether a dashboard should show operational truth, accounting truth, or customer-experience truth.
Those are not implementation details. Those are business decisions wearing technical clothes.
Forces
A good architecture article should name the forces plainly. Here they are.
1. Domain autonomy versus enterprise consistency
Domain-driven design gives us a useful lens: the business is not one model but many bounded contexts. This matters deeply for data architecture. Trying to force all contexts into one canonical enterprise schema creates a brittle abstraction. But letting every domain publish anything in any shape creates chaos.
You need enough autonomy for teams to move, and enough consistency for the enterprise to reason across domains.
That balance is architectural, not procedural.
2. Real-time expectations versus correctness
Kafka and streaming architectures are seductive because they make movement visible. Events flow, dashboards refresh, systems react. But “real-time” does not mean “correct.” Streams can be delayed, duplicated, reordered, or semantically incomplete. Financial close, regulatory reporting, and audit often require reconciliation points that are more important than immediacy.
Many modern platforms fail because they optimize for motion over agreement.
3. Decentralized ownership versus operational maturity
Pushing responsibility to domain teams works only if those teams can actually own data products: schemas, SLOs, quality, lineage, security, and lifecycle. Without maturity, decentralization becomes distributed negligence.
4. Historical truth versus operational convenience
Operational systems care about now. Data platforms care about history, correction, and replay. These are not the same problem. CDC from a source database might capture row changes but miss business intent. Events might capture intent but not final state. Snapshots help with reconstruction but blur causality. You often need all three.
5. Reuse versus local optimization
The shared platform should provide paved roads—ingestion, storage patterns, contracts, observability, access control, metadata, and serving patterns. But too much standardization throttles progress; too little creates a support nightmare.
6. Migration urgency versus business continuity
No large enterprise gets to redesign its information landscape from scratch. You migrate while serving existing reports, preserving downstream interfaces, and keeping the quarter-end close intact. Every architectural recommendation worth anything must survive contact with that constraint.
Solution
The answer is not to chase a fantasy of a perfectly unified platform. The answer is to design for explicit complexity placement.
A strong modern data platform does five things well.
First, it organizes around domains, not technologies. Storage formats, pipelines, and brokers are implementation choices. The real unit of design is the domain boundary: orders, pricing, claims, customer servicing, billing, inventory, and so on. This is where domain-driven design matters. Each bounded context should own its language, event semantics, and data products.
Second, it separates operational integration from analytical consolidation. This is where many teams go wrong. They use Kafka as if it were the warehouse, or they use the warehouse as if it were an event backbone. Those are different jobs. Event streams move domain facts and reactions. Analytical layers reconcile, enrich, and present cross-domain views over time.
Third, it treats canonical enterprise models with suspicion. A universal schema is usually a bureaucratic coping mechanism. Better to use federated semantics: domain models at the edge, explicit cross-domain mappings in the middle, and governed enterprise views where the business truly needs them—finance, risk, compliance, executive metrics.
Fourth, it makes reconciliation a first-class architecture concern. Reconciliation is not a cleanup step. It is the mechanism by which a distributed enterprise regains trust. If customer counts differ between CRM, billing, and the lakehouse, the platform must explain why and how they converge. Every serious enterprise platform needs golden reconciliation paths for key business entities and metrics.
Fifth, it embraces progressive strangler migration rather than big-bang replacement. The old warehouse, ETL estate, and shared marts do not disappear overnight. You carve out domains gradually, replicate source facts into the new platform, validate parity, and move consumers one class at a time.
A data platform is not modern because it uses streaming. It is modern because it knows where meaning lives, how meaning moves, and how meaning is checked when systems disagree.
Architecture
The target topology usually looks less like a single platform and more like a layered ecosystem.
At the edge, operational systems and microservices emit events or CDC feeds. Those facts land in a streaming backbone such as Kafka and in durable raw storage. Domain teams own transformation logic that shapes operational facts into domain data products. Some products are event streams, some are curated tables, some are serving APIs, some are feature sets for ML.
Above that sits a cross-domain integration layer. This is where identity mapping, reference data alignment, and shared business calculations live. Not everything belongs here, only the few things that truly span domains.
Then comes the consumption layer: BI, regulatory reporting, data science, operational dashboards, APIs, and reverse ETL into SaaS systems.
The important thing is not the boxes. It is the responsibility split.
Before-and-after topology
The “before” topology in most enterprises is centralized but congested. Sources feed a common ETL layer. ETL feeds a warehouse. Warehouse feeds marts and reports. Every change queues behind a shared team. Semantics are hidden in transformations no one fully understands.
The “after” topology is decentralized but explicit. Domains own products. Events and raw facts are preserved. Shared concerns are isolated into specific services and semantic views rather than buried inside one monolith.
Domain semantics discussion
This is where architecture usually gets shallow. It talks about pipelines and says little about meaning.
Meaning is the heart of the platform.
In DDD terms, an OrderPlaced event belongs to the ordering bounded context. It means something precise there: an order was accepted according to ordering rules. In fulfillment, however, the important concept may be a shipment request. In billing, the important concept may be an invoiceable obligation. If you publish one “enterprise order” event and expect all downstream domains to read the same thing, you are sneaking in a false canonical model.
Better to publish domain facts honestly, then create explicit translations where needed.
That translation layer can be event-driven, batch-derived, or both. But it should be visible. Hidden semantic translation is where future outages breed.
Reconciliation as architecture
In distributed data platforms, reconciliation becomes the adult supervision.
You need processes that answer questions like:
- Did every invoiceable event become an invoice?
- Does the customer count in the lakehouse match the CRM after identity merging?
- Did a replay of CDC produce the same ledger totals as the source system?
- Which metrics are provisional, and which are finance-certified?
This is not glamorous. It is the reason people trust the platform.
A mature platform does not just publish data. It publishes confidence.
Migration Strategy
This is the part that separates architecture from aspiration.
No bank, insurer, retailer, or manufacturer with a serious reporting estate is going to switch off the warehouse on a Friday and celebrate on Monday. Data platform migration is a strangler fig, not a demolition charge.
Step 1: Map business capabilities and bounded contexts
Begin with domains, not datasets. Identify the bounded contexts that matter operationally and analytically. Orders, payments, claims, customer, product, pricing, inventory, billing, risk. Then classify their current sources, critical reports, downstream consumers, and quality pain points.
This reveals where migration can produce real value instead of just technical novelty.
Step 2: Establish the dual-running backbone
Stand up Kafka or equivalent event streaming where operational integration needs it. Add CDC for systems that cannot emit rich events. Persist raw data in durable storage. Build metadata, lineage, contract validation, and observability from day one. If you postpone these, you will recreate the old mess in a more fashionable shape.
Step 3: Carve out one or two high-value domains
Pick domains with clear ownership, painful bottlenecks, and measurable business outcomes. Orders and customer often work well. Avoid starting with finance-certified reporting unless your organization has the discipline for rigorous reconciliation from the outset.
Build domain data products in the new platform. Keep old outputs alive.
Step 4: Reconcile before redirecting consumers
Do not migrate dashboards or downstream APIs because the new data “looks about right.” Run the new and old pipelines in parallel. Compare row counts, balances, dimensional conformance, and business KPI outputs over meaningful periods. Investigate divergences. Some differences reveal bugs. Others reveal old assumptions nobody documented.
Parallel run is expensive. So are credibility collapses.
Step 5: Move consumers progressively
Migrate consumers by class:
- exploratory analytics first
- new product use cases next
- operational dashboards after confidence builds
- regulatory and finance-certified outputs last
This sequence respects risk.
Step 6: Retire old transformations surgically
Once a consumer set has moved and reconciliation is stable, decommission corresponding legacy ETL paths. Archive logic, lineage, and mapping documentation. Resist the temptation to leave old jobs “just in case.” Zombie pipelines become silent liabilities.
Step 7: Repeat domain by domain
The strangler pattern works because it compounds confidence. Each migrated domain teaches the organization where semantics were implicit, where controls were weak, and where platform abstractions need refinement.
Migration is not just moving workloads. It is teaching the enterprise to own meaning in a distributed world.
Enterprise Example
Consider a multinational insurer modernizing its data platform.
It had the classic estate: policy administration in one stack, claims in another, billing on a separate platform, CRM in a SaaS product, and an enterprise warehouse fed by nightly ETL. Reports for underwriting, claims operations, agent commissions, and finance all ran from the warehouse. Delivery times were glacial. Every new metric required warehouse team intervention. Digital products needed near-real-time data the warehouse could not provide.
The executive mandate sounded simple: move to a modern cloud data platform with streaming, support microservices, and reduce warehouse complexity.
What actually happened was more interesting.
The insurer adopted Kafka as the event backbone for new digital services and introduced CDC for legacy policy and claims systems. It created domain-aligned data product teams around policy, claims, billing, and customer. Raw events and change streams were persisted into object storage. Domain teams built curated datasets and event streams for their own consumers.
Immediately, semantics surfaced.
A “customer” in CRM represented a person or business relationship. In policy administration, the key concept was insured party. In claims, the important role might be claimant, witness, or beneficiary. The old warehouse had flattened these into one customer dimension with years of undocumented exceptions. In the new world, domain teams resisted a single canonical customer model—and they were right to resist.
Instead, the platform introduced an identity and party resolution service in the cross-domain layer. Domains retained their local concepts. Enterprise reporting consumed a governed party semantic view built from explicit mappings and survivorship rules. The complexity did not disappear. It moved from hidden SQL into an explicit service and semantic product.
Then came reconciliation.
Claims paid totals from the new claims domain product did not match finance reporting for three consecutive trial runs. The root cause was not a bug in Kafka. It was timing and semantics. The claims platform emitted payment authorization events when adjusters approved payments, while finance recognized payments only after settlement posting. The old warehouse had quietly encoded this distinction in a batch transformation. The new platform had to make it explicit.
The architecture changed. Two domain facts were modeled: ClaimPaymentAuthorized and ClaimPaymentSettled. Finance-certified views used the latter. Operational claims dashboards used the former. Reconciliation reports classified differences by lifecycle state.
That is what good modernization looks like. Not elegance by slogan. Clarity by confrontation.
Over eighteen months, the insurer strangled out major sections of the warehouse for claims analytics, digital servicing, and customer 360 APIs. But it did not eradicate the warehouse overnight. Some finance and regulatory workloads remained until the new reconciliation controls proved themselves through quarterly close cycles.
The outcome was not “less complexity.” The outcome was complexity placed where experts could reason about it: in domain contracts, identity rules, and reconciliation workflows. Delivery improved. Trust improved. Governance improved because semantics stopped pretending to be universal. ArchiMate for governance
Operational Considerations
A distributed data platform lives or dies operationally.
Data contracts
If domains publish events or curated datasets, they need contracts with versioning rules, compatibility checks, ownership metadata, and deprecation policy. Without this, Kafka becomes a firehose of accidental dependencies.
Lineage and discoverability
Self-service is a myth without metadata. Teams must be able to discover what exists, who owns it, what it means, how fresh it is, and whether it is certified. Catalogs matter. So do ownership pages and runbooks.
SLOs and data reliability engineering
Every serious data product should declare expectations: freshness, completeness, schema stability, and support path. This is where data reliability engineering earns its keep. If a pipeline misses an SLA, who is paged? If a contract breaks, how is blast radius assessed?
Security and access
Domain ownership does not remove the need for enterprise policy. PII, financial records, health information, and regulated datasets need consistent access control, masking, tokenization, and audit. Federated architecture with centralized security controls is usually the pragmatic choice.
Replay and backfill
Sooner or later you will need to replay Kafka topics, rebuild tables from raw storage, or reprocess CDC after fixing a bug. Design for this from the beginning. Idempotency, deterministic transformations, partition strategy, and retention policy are not footnotes.
Cost management
Modern platforms often trade people bottlenecks for infrastructure sprawl. Storage duplicates. Stream processing costs grow. Too many serving stores appear. Cost visibility by domain and product is essential, or the platform becomes financially opaque.
Tradeoffs
Architecture is the art of choosing your problems.
A centralized warehouse gives strong control, simpler governance, and one place to optimize. It also creates bottlenecks, hides semantics, and struggles with event-driven use cases.
A distributed domain-oriented platform improves autonomy, local relevance, and fit for real-time integration. It also increases the need for explicit contracts, stronger engineering discipline, and robust reconciliation.
Kafka is powerful for decoupling and event-driven design, but it can lure teams into overusing streams for problems better solved in analytical stores. CDC accelerates migration, but row-level changes are not always faithful business events. Domain ownership aligns data with business capability, but only if domains truly have accountable teams. Shared semantic layers reduce consumer burden, but they can become a new central bottleneck if overloaded.
There is no free lunch here. There is only a better menu.
My view is straightforward: if your enterprise is large, your business domains are meaningfully different, and your platform must support both operational and analytical use cases, then distributed domain-aligned architecture is usually the right direction. But do not sell it as simplification. Sell it as a healthier distribution of complexity.
Failure Modes
Modern data platform programs fail in recognizable ways.
1. Canonical model relapse
Teams say they support domain ownership but secretly rebuild a universal enterprise schema in the middle. This recreates the old bottleneck under a new brand.
2. Event theater
The organization adopts Kafka, publishes lots of events, and calls itself modern. But event definitions are vague, ownership is weak, and analytical consumers still cannot trust the data.
3. CDC absolutism
CDC is useful, not magical. If you treat database changes as equivalent to domain truth, you will miss intent, lifecycle meaning, and process boundaries.
4. No reconciliation discipline
This is the silent killer. Without reconciliation, mismatches accumulate until executives stop trusting the new platform. Trust dies long before the platform does.
5. Distributed ownership without platform support
If every domain team has to build its own ingestion, quality checks, lineage, and serving stack, the architecture devolves into artisanal chaos.
6. Premature warehouse shutdown
Turning off legacy outputs before parity and governance are proven is how modernization programs become career events.
7. Ignoring domain semantics
This is the oldest mistake in new clothes. Data engineering without domain semantics is plumbing without architecture.
When Not To Use
There are situations where this style of architecture is the wrong move.
Do not build a domain-distributed, Kafka-heavy data platform if your enterprise is small, your domains are simple, and your reporting needs are mostly centralized and batch-oriented. A well-run warehouse or lakehouse with modest governance may be enough.
Do not decentralize ownership if your engineering culture cannot sustain product ownership. If teams are thin, turnover is high, and operational discipline is weak, a shared central team may produce better outcomes for now.
Do not push streaming everywhere if the business value is low. Real-time dashboards are not automatically useful. Many business decisions tolerate hourly or daily latency perfectly well.
Do not introduce cross-domain semantics platforms prematurely. If domains are still unstable, forcing enterprise-wide semantic harmonization too early creates brittle agreements no one believes in.
And do not imagine data mesh language solves management problems. If incentives, accountability, and funding models remain centralized and contradictory, relabeling datasets as products changes very little.
Related Patterns
Several patterns sit naturally beside this architecture.
Strangler fig migration is the essential migration approach: gradually replace legacy data flows while preserving business continuity.
Bounded contexts from domain-driven design provide the right mental model for data ownership and semantic boundaries.
Anti-corruption layers help when legacy source semantics must be translated into cleaner domain models.
CQRS-style separation is often useful: operational event streams and analytical read models should not be forced into one shape.
Data products give teams a way to package ownership, quality expectations, discoverability, and interfaces.
Reconciliation services are a pattern in their own right for regulated or high-trust environments.
Lakehouse plus streaming backbone is increasingly the practical combination: durable history in open storage, movement and reaction in Kafka, curated serving layers where needed.
Used together, these patterns create a platform that can evolve. Used carelessly, they create a distributed monument to wishful thinking.
Summary
The promise of data platform modernization is often framed badly.
People say the legacy warehouse is too complex, so we need streaming, domains, data products, Kafka, microservices, and a lakehouse to simplify everything.
That is not what happens.
What happens is better, and harder.
Complexity leaves the old ETL monolith and reappears in domain contracts, event semantics, identity resolution, lineage, replay, platform tooling, and reconciliation. The centralized pain of transformation logic becomes the distributed pain of semantic coordination. The trick is not to deny this. The trick is to make the new pain worth having.
That means designing around bounded contexts. Separating operational events from analytical truth. Treating reconciliation as a core capability. Migrating progressively with a strangler approach. Preserving raw facts. Governing enterprise views where the business genuinely needs them. And resisting the false comfort of canonical universality.
In short: your data platform complexity moved. It did not shrink.
A good architecture admits that. A great one uses it to put complexity in places where the enterprise can finally manage it.
Frequently Asked Questions
What is a data mesh?
A data mesh is a decentralised data architecture where domain teams own and serve their data as products. Instead of a central data team, each domain is responsible for data quality, contracts, and discoverability.
What is a data product in architecture terms?
A data product is a self-contained, discoverable, trustworthy dataset exposed by a domain team. It has defined ownership, SLAs, documentation, and versioning — treated like a software product rather than an ETL output.
How does data mesh relate to enterprise architecture?
Data mesh aligns data ownership with business domain boundaries — the same boundaries used in domain-driven design and ArchiMate capability maps. Enterprise architects play a key role in defining the federated governance model that prevents data mesh from becoming data chaos.