Real-Time Data Without Semantics Is Faster Noise

⏱ 19 min read

Speed seduces architects.

A team gets a streaming platform running, events start flying through Kafka, dashboards light up in seconds, and everyone feels modern. There is a particular kind of executive smile reserved for real-time systems: the smile that says we are now ahead of the market. But then the harder truth arrives. The numbers in one service don’t match the ledger in another. “Customer” means three different things depending on which topic you read. A cancellation event lands before the order-created event because one upstream team changed retry logic. The pipeline is fast, but the business is confused.

That is the central mistake in many ingestion topologies: we optimize for motion before meaning.

Real-time data without semantics is not insight. It is just faster noise.

Ingestion architecture is often discussed as plumbing. Queues here, connectors there, some schema registry, maybe a lakehouse at the end. Useful, but incomplete. In the enterprise, ingestion topology is not just about moving bytes. It is about preserving business intent while data crosses system boundaries, organizational boundaries, and time itself. The shape of the topology determines whether your enterprise becomes loosely coupled or merely loosely understood.

This is where domain-driven design matters more than most streaming diagrams admit. If events are the language of the business, then ingestion is not a transport problem alone. It is a translation problem, a boundary problem, and, quite often, a truth problem.

A good ingestion topology does three things at once. It moves data quickly. It carries domain semantics intact enough that downstream consumers can make reliable decisions. And it allows old and new systems to coexist during migration without turning reconciliation into a permanent tax. That sounds obvious. In practice it is rare.

This article argues for an opinionated approach: design ingestion around bounded contexts and semantic contracts, not around infrastructure capabilities. Use Kafka and microservices where they fit, but don’t confuse topics with domains. Migrate progressively using a strangler pattern, and treat reconciliation as a first-class architectural concern rather than an afterthought for the data team. If you do that, your topology becomes an operating model for change. If you do not, your “real-time platform” becomes a very expensive rumor mill.

Context

Most enterprises did not arrive at streaming architecture by design. They arrived there by accumulation.

A core platform emits CDC from a relational database. A mobile app sends clickstream events. An ERP exports files every hour. CRM updates arrive from SaaS webhooks. IoT devices post telemetry. A data platform team stands up Kafka to unify ingestion, then discovers that “unify” really means “place all disagreements onto one durable substrate.” That is still progress, but it is not coherence. event-driven architecture patterns

Legacy estates make this worse. The old systems were usually built around transaction processing, often with hidden business rules embedded in stored procedures, batch jobs, or application services nobody wants to touch. New digital services are built as microservices with API-first instincts and event-first ambition. Both worlds need to coexist for years, not quarters. So the ingestion topology becomes the fault line between old truth and new truth. microservices architecture diagrams

In this setting, architecture is not a question of whether to use streams, APIs, or files. Enterprises use all three. The real question is how to organize ingestion so the business meaning of data survives the journey. A fast stream carrying ambiguous facts is more dangerous than a slow batch carrying well-governed state. Bad semantics scale beautifully.

Problem

Many ingestion topologies fail because they are optimized around sources rather than domains.

You can see the symptom in topic design. Teams create topics named after systems: sap_customer, crm_customer, billing_customer, ecomm_customer. At first this feels pragmatic. It reflects reality. But it also bakes source-system thinking into the platform. Downstream consumers are now forced to reverse-engineer the business across technical seams. Every consuming team becomes an amateur integration team.

Another version of the same failure appears in “generic event” models. A platform team standardizes on envelopes, metadata, and a broad event taxonomy, then congratulates itself on consistency while every payload means something slightly different. “StatusChanged” becomes the most dangerous phrase in the enterprise. Status of what? Changed from which lifecycle state to which one? According to whose rules? Triggering which obligations? Without domain semantics, standardization can become camouflage.

Real-time ingestion then amplifies the ambiguity. Consumers subscribe early because latency is attractive. They implement local assumptions because business pressure is immediate. Soon one topic supports fraud checks, customer notifications, warehouse release, finance accruals, and machine learning features. The event was never designed to satisfy all those uses. It becomes overburdened, then unstable. A source team changes one field, and five downstream capabilities wobble.

This is not mainly a technology failure. It is a modeling failure.

Forces

Several forces pull the architecture in different directions.

First, business wants low latency. Not because “streaming” is fashionable, but because some decisions lose value quickly. Fraud scoring, inventory promises, dynamic pricing, operational alerts, and customer notifications all benefit from timeliness.

Second, enterprise reality demands coexistence. Core systems are not replaced in one motion. They are surrounded, carved out, and retired gradually. During that period, data will be duplicated, transformed, replayed, corrected, and disputed.

Third, domains are not uniform. A payment authorization event has very different quality requirements from a product view event. One is legally and financially consequential. The other is analytical and disposable. Treating them with the same ingestion posture is lazy architecture.

Fourth, organizations are fragmented. Teams own systems, not enterprise semantics. Incentives reward local delivery. So unless the architecture creates clear semantic boundaries, each team exports convenience rather than clarity.

Fifth, compliance and auditability matter. Real-time data is not exempt from traceability. In regulated industries, you must explain not only what happened, but why a system believed it happened at that moment based on the information available then.

And finally, there is cost. Semantic enrichment, canonical contracts, lineage, reconciliation, and replay tooling are not free. Every architect eventually faces the same hard tradeoff: invest in meaning up front, or pay for confusion forever.

Solution

The right ingestion topology starts with bounded contexts, not brokers.

That line is worth defending. Kafka is powerful, but it does not tell you what a customer is, when an order is committed, or whether inventory reservation is a fact or an intention. Those are domain decisions. If you skip them, the platform will faithfully distribute your confusion at scale.

The core pattern is this:

  1. Ingest source-native facts at the edge
  2. Translate them into domain-semantic events inside a bounded context
  3. Publish curated, versioned contracts for downstream use
  4. Separate operational events from analytical feeds
  5. Build reconciliation paths alongside the real-time path

This creates a layered ingestion topology rather than a single universal bus of raw truth.

At the edge, you collect what the source can reliably provide: CDC, API events, files, telemetry. Do not pretend raw ingestion is domain truth. It is evidence. Sometimes good evidence, sometimes weak evidence.

Inside a domain-aligned ingestion service or integration microservice, you perform mapping, validation, deduplication, enrichment, reference resolution, and lifecycle interpretation. This is where source-system semantics are translated into business semantics. It is also where anti-corruption layers belong. A legacy ERP may represent order state with codes and flags accumulated over fifteen years. Downstream services should not inherit that mess just because Kafka makes copying easy.

Then publish semantic events that mean something stable in the domain: OrderPlaced, PaymentAuthorized, InventoryReserved, CustomerAddressValidated. Not every internal state change deserves publication. Events are promises. Be careful what you promise.

A good ingestion topology also acknowledges different consumption modes. Operational consumers need compact, durable, ordered-enough streams with explicit contracts. Analytical consumers often need append-only histories, snapshots, or denormalized data products. Forcing both onto the same topic design usually satisfies neither.

And there is one more rule, deeply unglamorous but essential: every real-time path that matters needs a reconciliation path. Streams tell you what should have happened. Reconciliation tells you what actually agrees.

Architecture

Here is the shape I recommend for a large enterprise adopting real-time ingestion without losing semantics.

Architecture
Architecture

This topology has three deliberate layers.

Raw ingestion layer. This is where you absorb the source heterogeneity. Keep it close to the source shape. Retain metadata such as source system, ingestion time, source transaction identifiers, file offsets, connector version, and trace identifiers. This layer is not for broad enterprise consumption. It is for containment, replay, diagnostics, and controlled translation.

Domain translation layer. This is where the hard work lives. It should be owned jointly by platform and domain teams, though domain ownership should dominate the semantics. Translation services apply domain rules, lookup reference data, normalize identities, detect duplicates, and reject malformed or semantically impossible messages. This is not “just ETL.” It is executable domain understanding.

Semantic event layer. This is what most downstream systems should consume. Topics should be aligned with meaningful domain events or state changes, versioned carefully, and documented as business contracts. The payload should explain the event in business terms, not in source-column terms.

A common mistake is to over-centralize translation in one giant integration platform. Better than chaos, perhaps, but still brittle. A healthier approach is federated governance: common patterns and tooling, local ownership of domain semantics. The platform team should provide rails, not theology. EA governance checklist

Domain semantics and event design

If a topic is called OrderUpdated, you have probably already lost.

Good semantic events have clear business meaning, explicit lifecycle position, and known consumers. They indicate whether they represent a fact, an intent, or a derived conclusion. They include identifiers from the bounded context and often carry references to upstream identities for traceability.

For example:

  • OrderPlaced is a business fact after commitment.
  • PaymentAuthorizationRequested is an intent.
  • FraudRiskAssessed is a derived conclusion with a model version and confidence.
  • ShipmentDispatched is a physical process milestone.

These distinctions matter because consumers behave differently depending on what the event means, not just when it arrived.

Here is how bounded contexts fit into the ingestion topology.

Diagram 2
Domain semantics and event design

Notice what this avoids. It avoids a universal “enterprise customer event model” pretending all domains agree. They do not. Sales, billing, support, and identity often mean different things by customer. DDD does not eliminate that tension; it gives you a disciplined way to contain it.

Kafka and microservices

Kafka fits very well here, but with discipline.

Use Kafka as the durable backbone for raw and semantic streams, especially where event history, replay, fan-out, and decoupling matter. Use microservices to own translation logic where domain rules are substantial and where anti-corruption boundaries must be explicit. Use schema registry, contract testing, and topic lifecycle governance. Prefer idempotent consumers and explicit partitioning strategies tied to business keys where order matters. ArchiMate for governance

But do not use Kafka as an excuse to avoid APIs. Some interactions are commands, not events. Some queries need synchronous consistency. A mature enterprise topology uses streams and APIs together, based on interaction style rather than ideology.

Migration Strategy

The migration story matters as much as the target architecture. In fact, in large enterprises, the migration is the architecture for the next three years.

A progressive strangler approach is the practical choice. You do not rip out old integrations and replace them with a pure event model in one move. You introduce semantic layers around existing systems, gradually shift consumers to the new contracts, and retire old paths only after evidence, not hope.

Here is a typical progression.

Migration Strategy
Migration Strategy

Step 1: Expose source-native changes safely

Start by capturing changes from legacy systems without changing their behavior. CDC is often useful here because it minimizes source impact and reveals actual committed state transitions. But CDC is not a domain model. It tells you what changed in tables, not what happened in the business. Treat it as a stepping stone.

Step 2: Build semantic translators

Introduce services that consume raw changes and produce domain events. Initially they may rely on ugly mapping logic. That is acceptable. The point is to isolate the ugliness behind a stable semantic contract. This is your anti-corruption layer.

Step 3: Run in parallel

Keep old integrations operating while new consumers adopt semantic topics. During this phase, reconciliation is your safety net. Compare outputs between legacy and new flows. Measure divergence. Investigate business-significant mismatches, not just technical discrepancies. Perfect parity is often impossible because the new model may intentionally improve behavior. What matters is understood difference.

Step 4: Strangle by capability, not by system

Retire old integration paths one business capability at a time: order notification, customer eligibility, inventory promise, invoicing trigger. Enterprises get into trouble when they define migration in terms of applications rather than capabilities. Systems are too entangled. Capabilities are tractable.

Step 5: Make reconciliation permanent where truth is distributed

Some architects treat reconciliation as a temporary migration concern. In distributed enterprises, that is fantasy. When multiple systems hold operationally relevant views of the same business object, reconciliation becomes a permanent control, especially for finance, inventory, customer balances, and regulatory reporting.

This is not a sign of architectural failure. It is a sign of adult supervision.

Enterprise Example

Consider a global retailer modernizing order processing across e-commerce, stores, and partner marketplaces.

The retailer had:

  • a legacy ERP for inventory and finance
  • a separate order management system
  • a modern e-commerce platform
  • regional warehouse systems
  • a CRM used by service agents
  • nightly batch feeds into analytics

Leadership wanted real-time order visibility and inventory availability. The first attempt was classic platform optimism: stream every source into Kafka, standardize envelopes, let teams consume what they need. Within months, the retailer had over a hundred topics and a thicket of consumer logic. One team interpreted order acceptance as “payment started,” another as “inventory reserved,” and finance only recognized the order after a later ERP state change. Dashboards became fast and contradictory.

The architecture was then reset around bounded contexts.

Sales owned the meaning of OrderPlaced and OrderAmended. Payment owned PaymentAuthorized and PaymentCaptured. Fulfillment owned InventoryReserved, BackorderDeclared, and ShipmentDispatched. Finance did not consume raw sales topics for accounting; it consumed semantically governed events enriched with posting references and reconciliation IDs.

Legacy ERP changes were captured by CDC into raw topics. Translation services mapped ERP line statuses and warehouse reservation flags into fulfillment events. The order management system still operated for months, but new microservices subscribed only to semantic topics. A process manager coordinated cross-context workflows where necessary, but long-lived truth remained in each bounded context.

The critical move was reconciliation. Every order carried a shared business key across semantic events and legacy records. A reconciliation service compared:

  • order header and line totals between e-commerce, OMS, and ERP
  • payment status between gateway and finance postings
  • reservation and shipment state between WMS and customer-visible status
  • cancellation/refund consistency across channels

The team discovered predictable failure modes during parallel run: duplicated CDC events after connector failover, out-of-order shipment updates from one warehouse vendor, and occasional semantic drift when a source team added a new status code without notice. None of these issues were catastrophic because the topology had places to contain them: raw retention, translator logic, quarantine queues, and reconciliation dashboards.

The result was not purity. It was operability. Real-time customer notifications improved. Inventory accuracy improved enough to reduce canceled orders. Finance gained a traceable chain from operational event to accounting consequence. And over two years, point-to-point integrations were retired one capability at a time.

That is what enterprise success looks like: less heroism, more clarity.

Operational Considerations

Real-time ingestion lives or dies in operations, not PowerPoint.

Contract governance

Schema evolution needs rules. Backward compatibility is not enough if business meaning changes. A field can remain structurally valid while becoming semantically dangerous. Version contracts when business interpretation changes, and make that visible in documentation and lineage.

Observability

Track more than throughput and lag. You need semantic observability:

  • event acceptance/rejection rates by domain rule
  • duplicate detection rates
  • reconciliation variance trends
  • source-to-semantic translation latency
  • out-of-order arrival patterns
  • consumer contract adoption by version

If all you can see is broker health, you are blind where it matters.

Replay and reprocessing

Replay is one of streaming’s superpowers, but replaying without semantic discipline can create historical fiction. Translation logic changes over time. Reference data changes. Domain policy changes. So replay tooling must specify whether it reproduces prior outcomes or recomputes with current rules. Both modes are useful. Mixing them casually is dangerous.

Data quality and quarantine

Not all bad records should stop the stream. Quarantine semantically invalid or incomplete messages with enough metadata for diagnosis and remediation. But be careful: a quarantine queue is not a garbage chute. If nobody owns triage, it becomes an archive of shame.

Identity resolution

Many semantic problems are really identity problems in disguise. Customer, supplier, order, account, and product identifiers often differ across systems. If identity resolution is weak, event semantics collapse downstream. Treat mastering and correlation as architectural concerns, not integration chores.

Security and compliance

Raw topics often contain more sensitive data than semantic topics need. Apply least privilege. Mask or tokenize where possible. Keep audit chains from source evidence through translation to downstream decisions.

Tradeoffs

This architecture is not free.

The biggest tradeoff is complexity up front versus confusion later. Domain translation layers, anti-corruption services, reconciliation engines, and governance processes all slow initial delivery. Teams used to dumping raw events onto Kafka will complain that this is heavy-handed. Sometimes they will be right. Not every use case needs this much care.

Another tradeoff is duplication of logic. Multiple bounded contexts may derive related semantics from the same source. That can feel inefficient. But the alternative is often hidden coupling through a shared “enterprise semantic layer” nobody can evolve safely. Some duplication is the price of autonomy.

There is also latency. Translation, enrichment, validation, and reconciliation checks add time. In many domains that is acceptable. In a few ultra-low-latency scenarios, it is not. Architecture should respect the economics of the decision being made.

And there is organizational friction. DDD-aligned ingestion asks teams to agree on context boundaries and business events. That is harder than provisioning clusters. Technology is easy to centralize. Meaning is not.

Failure Modes

Let’s be blunt. These systems fail in recurring ways.

Raw topics become de facto enterprise APIs. Teams bypass semantic layers “just temporarily,” then depend on source internals forever.

Topic names are semantic, payloads are not. CustomerUpdated sounds clean until you realize the payload is a direct dump from CRM with twelve fields that mean nothing outside that system.

CDC is mistaken for event sourcing. Table changes are useful, but they are not a substitute for intentional business events.

Reconciliation is underfunded. The platform works well enough in demos, then trust erodes because nobody can explain discrepancies across systems.

Ordering assumptions leak. A consumer silently assumes event order across partitions or sources, then makes incorrect decisions during failover or replay.

Too much canonicalization. An enterprise tries to define one universal model for customer, order, product, and account. The result is a bureaucratic artifact that satisfies no domain and slows all change.

No ownership for semantic drift. A source team changes a status code, reference mapping, or field meaning, and consumers discover it through production incidents.

These are not edge cases. They are standard enterprise weather.

When Not To Use

This approach is not always the right answer.

Do not build a semantics-heavy ingestion topology for small, low-risk analytical pipelines where occasional inconsistency is acceptable and no operational decisions depend on the stream. A simpler ELT approach may be enough.

Do not force domain event modeling onto use cases that are primarily synchronous command processing with strict immediate consistency requirements. Use APIs and transactions where they fit.

Do not introduce Kafka and translation microservices if your integration volume is modest, your source landscape is stable, and a well-governed batch or API integration meets the business need. Real-time is often purchased with complexity that the business does not actually value.

And do not attempt a grand enterprise semantic model before teams understand their own bounded contexts. Shared language emerges from practice. It does not arrive laminated from an architecture board.

Several patterns sit naturally beside this topology.

Anti-Corruption Layer. Essential when extracting semantics from legacy or external systems.

Strangler Fig. The right migration shape for progressive capability-by-capability modernization.

Outbox Pattern. Useful when modern services need to publish reliable domain events tied to state changes.

CQRS and Read Models. Often helpful for building optimized downstream views from semantic events.

Saga / Process Manager. Appropriate where cross-context workflows need coordination without centralizing all truth.

Data Mesh, cautiously interpreted. Domain-owned data products align well with semantic ingestion, provided governance is strong and operational event contracts are not confused with analytical sharing.

Summary

An ingestion topology is not just a route for data. It is a declaration of what the enterprise believes its data means as it moves.

That is why speed alone is a trap. Real-time platforms can make organizations feel informed while actually making them more inconsistent. Without domain semantics, events become fast-moving ambiguity. Without migration discipline, modernization becomes parallel complexity. Without reconciliation, trust decays.

The better path is more grounded and more opinionated: ingest raw facts at the edge, translate them inside bounded contexts, publish semantic contracts deliberately, migrate progressively with a strangler approach, and design reconciliation as a permanent control where distributed truth demands it. Use Kafka where durable streaming and fan-out matter. Use microservices where domain translation and ownership are explicit. Use APIs where synchronous intent matters. Above all, model the business before you optimize the pipeline.

In the enterprise, meaning is the real throughput constraint.

And once you see that, the topology changes.

Frequently Asked Questions

What is enterprise architecture?

Enterprise architecture aligns strategy, business processes, applications, and technology in a coherent model. It enables impact analysis, portfolio rationalisation, governance, and transformation planning across the organisation.

How does ArchiMate support architecture practice?

ArchiMate provides a standard language connecting strategy, business operations, applications, and technology. It enables traceability from strategic goals through capabilities and services to infrastructure — making architecture decisions explicit and reviewable.

What tools support enterprise architecture modeling?

The main tools are Sparx Enterprise Architect (ArchiMate, UML, BPMN, SysML), Archi (free, ArchiMate-only), and BiZZdesign. Sparx EA is the most feature-rich, supporting concurrent repositories, automation, scripting, and Jira integration.