Your Data Platform Needs Routing More Than Storage

⏱ 19 min read

Most enterprise data platforms fail for a boring reason dressed up as a technical one.

They don’t fail because the warehouse is too slow, the lake is too expensive, or Kafka is too complicated. They fail because the organization keeps treating data as if the hard part were where to put it. It usually isn’t. The hard part is deciding where it should go, why it should go there, and what meaning it carries on the way.

Storage gets all the budget because storage is visible. You can point at Snowflake, S3, BigQuery, Delta Lake, Kafka, MongoDB, Postgres, Elasticsearch. Procurement likes boxes with names. Architects like platform diagrams with neat zones. But the real leverage in a modern data platform is routing: the deliberate movement of data through bounded contexts, processing stages, trust boundaries, and usage models. event-driven architecture patterns

A data platform without routing is a warehouse with no loading dock plan. Pallets arrive. Forklifts improvise. Some goods land in the right aisle, some sit in the corridor, and some rot behind a locked door because nobody knows who owns them. Then leadership asks why analytics is slow, machine learning is inconsistent, operational reporting is wrong, and compliance reviews are terrifying.

This is not primarily a storage problem. It is a semantic routing problem.

That distinction matters. Once you see it, the architecture changes. You stop asking, “What should be our enterprise data store?” and start asking, “How do events, records, state changes, and derived facts move through the enterprise with their meaning intact?” That is a different conversation. It is more architectural, more domain-driven, and frankly more useful.

Context

For years, enterprise data architecture was dominated by storage-centric thinking. First came the enterprise data warehouse. Then the data lake. Then the lakehouse. Then streaming platforms. Then data mesh, which many companies interpreted as “everyone gets a bucket and a prayer.” enterprise architecture with ArchiMate

Each wave promised simplification. Instead, most large organizations accumulated more places to store data while becoming less certain about which one represented truth.

This happened because storage abstractions are seductive. They suggest permanence and order. They imply that if we can centralize enough data in one place, clarity will emerge naturally. It rarely does.

The modern enterprise runs on multiple tempos at once:

  • transactional systems need low-latency writes and strong consistency
  • customer-facing applications need event-driven integration
  • analytics workloads need batch and incremental processing
  • compliance functions need lineage and retention controls
  • machine learning needs curated feature pipelines
  • operational teams need reconciled, explainable numbers

One storage technology cannot serve all of these needs elegantly. More importantly, even if it could, centralizing the bytes would not solve the semantic confusion between domains such as customer, order, payment, inventory, policy, claim, shipment, or account.

This is where domain-driven design helps. DDD is not just for application services. It is crucial for data architecture because data meanings are born inside domains, not inside storage accounts. A “customer” in marketing is often a lead or household. A “customer” in billing is a liable party. A “customer” in support may be an authenticated user. If your platform routes all three into a canonical table called customer_master, you have not created order. You have created a future incident.

The platform’s job is not merely to store data. Its job is to route domain facts into the right processing and consumption paths while preserving provenance, ownership, and semantic boundaries.

Problem

Many enterprise data platforms are built as giant collection points.

Application teams dump CDC streams, flat files, API extracts, logs, and events into a central platform. A data engineering team cleans some of it, standardizes some of it, and republishes some of it. Downstream users consume whatever they can find. Meanwhile, source teams continue evolving their services, schemas drift, and business definitions diverge.

The result looks familiar:

  • Kafka topics become generic dumping grounds
  • data lakes become graveyards of half-understood parquet files
  • warehouses accumulate duplicate business entities
  • microservices publish integration events with no stable contracts
  • reconciliation becomes a monthly ritual of shame
  • everyone asks for a “golden record,” but no one agrees on what gold means

A storage-first design creates three structural pathologies.

First, semantic collapse. Distinct domain concepts are merged too early in pursuit of enterprise-wide standardization. This destroys useful nuance.

Second, coupling by accumulation. Downstream consumers attach themselves directly to raw datasets or source schemas because the platform has no disciplined routing layer. This turns every upstream change into an enterprise event.

Third, late trust formation. Data quality, ownership, and business interpretation are deferred until after ingestion. By then the data has already spread.

The platform ends up doing too much after the fact and too little at the point of movement.

Forces

Routing architecture exists because the enterprise is being pulled by competing forces. Any serious design has to admit them.

1. Domain autonomy versus enterprise consistency

Business domains want to own their data and evolve independently. The enterprise wants shared metrics, common identifiers, and integrated reporting. These desires are not enemies, but they are not naturally aligned either.

DDD gives us a useful frame: let bounded contexts own their language, while integration happens through explicit translation. Routing is the mechanism for that translation.

2. Real-time integration versus controlled consumption

Kafka and event streaming have made it easy to publish everything quickly. That is both power and danger. If every change event is treated as enterprise data, the platform becomes noisy and fragile. Not every internal event deserves broad distribution.

Routing lets you distinguish between:

  • domain events for cross-domain collaboration
  • CDC streams for replication and recovery
  • analytical extracts for historical analysis
  • operational projections for query use cases

They are not the same thing. Treating them as the same thing is one of the classic enterprise mistakes.

3. Flexibility versus governability

A central lake gives flexibility. A governed warehouse gives control. Enterprises usually need both. Routing architecture allows different controls at different stages rather than pretending a single platform layer can satisfy everyone.

4. Performance versus explainability

Highly optimized pipelines often hide business meaning inside transformations. But regulated industries need lineage, traceability, and reconciliation. Routing with explicit stages helps preserve explainability.

5. Migration speed versus production safety

No serious enterprise starts greenfield. There are mainframes, ERP systems, brittle ETL jobs, and reports that executives trust more than your new platform. Migration must be progressive. Routing creates seams where old and new can coexist.

Solution

The core idea is simple: design the platform around routing decisions, not storage destinations.

In practice, this means the platform has explicit layers that decide:

  • what data can leave a domain
  • in what form it leaves
  • who can consume it
  • whether it should be replicated, transformed, enriched, or aggregated
  • where reconciliation occurs
  • what becomes authoritative for a given use case

Storage remains important, of course. But storage is downstream of routing intent.

A routing architecture usually has four conceptual planes.

1. Source semantics plane

This is where domain systems emit facts: domain events, CDC changes, snapshots, APIs, files. The key question here is not format. It is meaning.

A payment service emitting PaymentCaptured is making a different semantic promise than a database CDC stream showing an updated row in payment_transaction. One expresses business intent. The other expresses persistence state. Both are useful. Neither should be confused for the other.

2. Routing and policy plane

This is the architectural heart.

Here the platform evaluates rules such as:

  • route customer-facing operational events to Kafka for downstream services
  • route CDC from ERP to a staging zone for reconciliation
  • route finance-approved aggregates to the warehouse semantic layer
  • route sensitive PII to tokenization before broad consumption
  • route low-trust data into quarantine and quality workflows

This plane may be implemented using stream processing, integration services, metadata policies, schema registries, data contracts, and orchestration tools. But the concept matters more than the tool.

3. Projection and serving plane

Once routed, data can be shaped into fit-for-purpose projections:

  • operational read models
  • domain data products
  • analytical marts
  • search indexes
  • feature stores
  • compliance archives

A projection is not truth. It is a useful view of truth for a consumer context.

4. Reconciliation and control plane

Enterprises do not live on event flow alone. They live on auditability.

A proper routing architecture includes reconciliation between:

  • source systems and routed streams
  • event-derived projections and batch-derived aggregates
  • operational truth and analytical truth
  • old platform outputs and new platform outputs during migration

This is where many “modern” architectures quietly fall apart. They move data quickly but cannot explain discrepancies.

Architecture

Here is the broad shape.

Architecture
Architecture

This architecture is opinionated in one important way: routing happens before universal standardization.

That means you do not build one giant canonical schema and force all domains through it. You classify flows according to usage and semantics first, then standardize where it is actually useful.

Domain semantics first

Suppose the order domain emits:

  • OrderPlaced
  • OrderCancelled
  • OrderShipped

That is a business narrative. It should likely flow through Kafka to fulfillment, notifications, fraud, and customer-service projections.

Now compare that to CDC from the order database:

  • updates to order_header
  • inserts into order_line
  • updates to status_code

That flow may be excellent for analytical replay, historical rebuilds, and technical recovery. But if you expose it broadly as your enterprise integration contract, you are coupling consumers to internal persistence design. That is not event-driven architecture. That is database gossip.

The routing layer separates these concerns.

Bounded contexts and translation

A good platform recognizes that integration is translation, not just transport.

Bounded contexts and translation
Bounded contexts and translation

Notice what is absent: there is no claim that “customer” means the same thing everywhere. Instead, the routing layer creates projections for specific enterprise purposes. That is a much healthier pattern than canonical overreach.

Kafka’s role

Kafka is often central in routing architecture, but it should be used with discipline.

Kafka is excellent for:

  • durable event transport
  • fan-out to multiple consumers
  • decoupling producers and consumers
  • replayable stream processing
  • near-real-time integration

Kafka is not excellent as:

  • your only system of record
  • a substitute for business ownership
  • a universal semantic model
  • a dumping ground for every table change in the enterprise

Use topics to represent intentional contracts. Use schema governance. Separate domain events from CDC topics. Define retention and replay rules according to business use, not only technical convenience. EA governance checklist

Serving stores are downstream choices

The same routed fact may land in multiple stores:

  • object storage for raw retention
  • lakehouse tables for analytical transformation
  • warehouse marts for governed reporting
  • Elasticsearch for search
  • Redis or document stores for operational views

That is fine. In fact, it is often right. But these are projections of routed data, not the platform’s organizing principle.

Migration Strategy

This is where routing architecture earns its keep.

Most enterprises cannot replace their data platform in a single move. They have legacy ETL, nightly batch jobs, tightly coupled reports, regulatory extracts, and a hundred invisible dependencies. The winning strategy is progressive strangler migration.

You do not rip out the old warehouse, the old integration bus, and the old ETL suite in one heroic quarter. Heroic quarters produce executive presentations and postmortems.

Instead, introduce routing as a control layer beside existing storage-centric flows. Let the new architecture coexist, prove value, and gradually absorb responsibility.

Step 1: classify current flows

Start by mapping existing data movement:

  • source system
  • transport mechanism
  • target store
  • business consumers
  • latency expectations
  • semantic type: event, transaction state, master data, aggregate, reference data

Most enterprises discover they have many pipelines but almost no explicit routing logic. Good. That means there is architectural leverage available.

Step 2: identify domain-owned outbound contracts

Work with domain teams to define what they are willing to publish intentionally:

  • domain events
  • approved CDC feeds
  • published snapshots
  • reference datasets

This is where DDD matters. The domain decides what facts are meaningful externally. The platform team should not infer domain semantics from table names.

Step 3: place a routing layer in front of broad consumption

New consumers should go through routed products, not directly to source schemas or random files. This is the first true strangler seam.

Step 4: build reconciliation between old and new outputs

Do not shut off legacy reports immediately. For each new routed projection, compare:

  • record counts
  • totals and balances
  • business keys
  • late-arriving correction behavior
  • historical restatement logic

Reconciliation is not a side concern. It is the bridge between architectural confidence and production reality.

Step 5: move consumer groups incrementally

Shift consumers one usage class at a time:

  • first operational read models
  • then selected analytical datasets
  • then executive reporting
  • finally decommission duplicated pipelines

Step 6: retire generic ingestion patterns

As routed contracts mature, shut down the worst anti-patterns:

  • direct table scraping by downstream teams
  • file drops with undocumented semantics
  • giant shared “enterprise” topics
  • all-purpose canonical entity pipelines

Here is a progressive migration view.

Step 6: retire generic ingestion patterns
retire generic ingestion patterns

This is strangler architecture applied to the data estate. It works because routing creates stable handoff points while legacy internals remain intact.

Enterprise Example

Consider a global insurer modernizing claims and policy data across 18 countries.

The legacy setup is typical:

  • policy administration on packaged core systems
  • claims on a mix of regional platforms
  • customer interactions in Salesforce
  • finance reporting built from nightly ETL into a central warehouse
  • hundreds of extracts for regulatory and actuarial teams

Leadership wants real-time claim status, unified customer reporting, and better fraud detection. The first instinct is predictable: build an enterprise lakehouse and load everything into it.

That would help some things. It would not solve the real problem.

The word “claim” itself means different things across contexts:

  • in claims operations, it is a case with lifecycle events
  • in finance, it is a liability with reserve and payment implications
  • in fraud, it is a behavioral pattern with linked entities
  • in customer service, it is a status inquiry anchored to policyholder communication

A storage-centric design would try to harmonize these too early. The result would be endless canonical modeling workshops and a parade of compromise columns.

A routing architecture starts differently.

Claims systems publish domain events like:

  • ClaimRegistered
  • ClaimAssigned
  • ClaimReserveAdjusted
  • ClaimPaymentIssued
  • ClaimClosed

Legacy platforms that cannot emit meaningful events provide CDC and snapshots instead. The routing layer classifies these feeds:

  • operational status events go to Kafka for customer and workflow services
  • reserve and payment facts go through finance-specific routing and reconciliation
  • historical snapshots go to analytical staging for actuarial modeling
  • suspicious claim interaction patterns are routed into fraud graph projections
  • PII-heavy payloads are tokenized before broader access

The insurer then builds three different projections:

  1. Customer claim journey for portals and call centers
  2. Finance claim movement mart for reserves, payments, and close-period control
  3. Fraud signal graph linking claims, addresses, devices, and providers

Same broad domain. Different semantics. Different consumers. Different trust requirements. One routing architecture.

The migration is progressive. For six months, the old warehouse reports and the new finance mart are reconciled daily. Differences are investigated:

  • late reserve updates arriving after cut-off
  • regional status mappings collapsing multiple operational states
  • duplicate policyholder identifiers from CRM merges
  • CDC gaps from one unstable legacy source

This is what real modernization looks like. Not magical replacement. Controlled coexistence, explicit translation, and relentless reconciliation.

By the time executives say the platform is “working,” the real victory has already happened: domains have clearer outbound contracts, consumers have better-shaped data products, and the enterprise finally knows which numbers are operational, which are analytical, and which are provisional.

Operational Considerations

Routing architecture is not a free lunch. It shifts responsibility into platform discipline.

Data contracts

Without contracts, routing becomes a glorified forwarding service. Every critical published flow needs:

  • owner
  • schema versioning policy
  • semantic description
  • quality expectations
  • deprecation path

Observability

You need to observe movement, not just storage.

Track:

  • event lag
  • routing failures
  • schema breaks
  • dropped or quarantined records
  • projection freshness
  • reconciliation deltas
  • consumer dependency maps

Lineage

Lineage matters more when you have multiple routed projections. Consumers must know:

  • source domain
  • transformation path
  • quality checks applied
  • reconciliation status
  • allowed usage

Security and privacy

Routing is where privacy control becomes practical. You can enforce:

  • tokenization before cross-domain sharing
  • field-level filtering by consumer type
  • regional routing constraints for sovereignty
  • retention policies by data class

Replay and correction

A mature routing architecture supports replay. But replay must respect semantics. Replaying CDC for backfill is different from replaying business events for rebuilding a projection. If you do not understand that distinction, you will rebuild the wrong truth very quickly.

Tradeoffs

Let’s be honest. Routing architecture is better than storage-first thinking, but it is not simpler.

Upsides

  • clearer domain ownership
  • reduced downstream coupling
  • fit-for-purpose projections
  • easier progressive migration
  • better control over trust and governance
  • improved reconciliation and explainability

Costs

  • more architectural design upfront
  • need for metadata and contract discipline
  • additional platform components
  • duplicated storage across projections
  • more explicit ownership discussions, which some organizations avoid because they are uncomfortable

There is also a political tradeoff. Routing architecture forces the enterprise to confront semantic disagreement. A warehouse can hide disagreement for a while by collapsing it into one table. Routing cannot. It makes ambiguity visible. That is painful, but healthy.

Failure Modes

Several patterns routinely derail this approach.

1. The fake routing layer

This is just another ingestion hub with nicer branding. It forwards everything everywhere without meaningful classification. Result: complexity without control.

2. Canonical model obsession

Teams spend a year inventing universal enterprise entities before shipping anything. The business moves on. Consumers keep using extracts. The program becomes a museum.

3. CDC as integration strategy

CDC is useful, but if all cross-domain collaboration is built on table changes, service boundaries rot. Consumers become hostage to source persistence design.

4. No reconciliation budget

The architecture goes live, but there is no systematic comparison between old and new outputs. Trust collapses at the first discrepancy.

5. Topic sprawl in Kafka

Every team publishes events with inconsistent naming, poor schemas, and no lifecycle policy. Kafka becomes a distributed junk drawer.

6. Platform centralization without domain participation

A central team cannot manufacture semantics alone. If domains do not own what leaves their boundary, the platform becomes speculative.

When Not To Use

Routing architecture is not always the right answer.

Do not build this if:

  • you are a small company with a handful of systems and one analytics team
  • your primary need is simply a well-modeled warehouse from a few stable sources
  • your domains are not actually autonomous and all data is already managed in one application boundary
  • you lack the organizational willingness to define ownership and contracts
  • your event volume and integration needs are trivial

In those cases, a simpler architecture is better. A well-run warehouse with modest ETL, clear conformed dimensions, and solid governance can outperform an overengineered routing platform. ArchiMate for governance

That is worth saying plainly: if your complexity is not semantic and organizational, do not solve it with routing machinery.

Routing architecture connects naturally to several established patterns.

Event-driven architecture

Use domain events for intentional business communication. But distinguish them from technical change streams.

Data products

A routed projection with clear ownership, SLA, semantics, and consumption model is often a better definition of a data product than “a table someone published.”

CQRS

Operational read models are a natural outcome of routing events into fit-for-purpose projections.

Strangler fig migration

A routing layer creates the seams needed to replace legacy data movement incrementally.

Outbox pattern

Useful when microservices need reliable publication of domain events without coupling directly to transport.

Lakehouse and warehouse semantic layers

These remain valuable serving patterns. They just should not carry the full burden of enterprise integration semantics.

Summary

The center of gravity in data architecture has shifted.

Storage still matters. Of course it does. You need durable systems, good economics, performant queries, and governable retention. But most enterprises already have more places to store data than they know how to govern. Their shortage is not storage. It is routing.

Routing is where architecture earns its salary.

It is where domain semantics are protected instead of flattened. Where events are distinguished from persistence noise. Where policy is enforced before uncontrolled spread. Where migration becomes progressive instead of theatrical. Where reconciliation turns “modernization” from a slide into an operational truth.

If you design your platform around storage, you will keep building bigger places to be confused.

If you design it around routing, you can let data move through the enterprise in ways that preserve meaning, support multiple use cases, and survive change.

That is the point. Not to create one place where all data lives forever, but to create an architecture that knows where data should travel, how it should be translated, and when it should stop.

That is what mature data platforms do. They route first, store second.

Frequently Asked Questions

What is a data mesh?

A data mesh is a decentralised data architecture where domain teams own and serve their data as products. Instead of a central data team, each domain is responsible for data quality, contracts, and discoverability.

What is a data product in architecture terms?

A data product is a self-contained, discoverable, trustworthy dataset exposed by a domain team. It has defined ownership, SLAs, documentation, and versioning — treated like a software product rather than an ETL output.

How does data mesh relate to enterprise architecture?

Data mesh aligns data ownership with business domain boundaries — the same boundaries used in domain-driven design and ArchiMate capability maps. Enterprise architects play a key role in defining the federated governance model that prevents data mesh from becoming data chaos.