Your Data Platform Is a Dependency Graph

⏱ 20 min read

Most data platforms are described as if they were storage systems. A lake. A warehouse. A mesh. A hub. A pipeline.

That language is convenient. It is also misleading.

What actually governs a data platform in a large enterprise is not where data sits, but what depends on what, who is allowed to interpret meaning, and how change propagates when one team “just” adds a field, retires an event, or redefines a customer. The real architecture is not the database engine, nor the stream processor, nor the cloud account hierarchy. The real architecture is the dependency graph.

Once you see that, a lot of familiar pain stops looking random. Your finance mart breaks because a CRM team changed lead qualification rules. Your machine learning feature store drifts because product telemetry semantics shifted under its feet. Kafka topics multiply like rabbits, but nobody can say which downstream services are safe to change. The platform looks modern. The operating model is still medieval. event-driven architecture patterns

A good enterprise architect learns to distrust diagrams that show boxes and arrows but hide semantic dependence. Data systems fail less often because of technology choices than because the organization cannot see the graph it has built: source systems, event streams, canonical models, bounded contexts, transformations, controls, reconciliations, reports, and decisions. Every edge in that graph is a promise. Every promise has a cost. Many are made casually. Few are retired cleanly.

So this article argues a simple, opinionated point: treat your data platform as a dependency graph first and a technology stack second. Use domain-driven design to decide where semantics belong. Use progressive strangler migration to move safely from brittle central platforms toward domain-aligned data products. Use reconciliation as a first-class discipline, not as an embarrassed cleanup step. And use Kafka and microservices only where they fit the shape of the graph, not because event-driven architecture looks good in conference talks. microservices architecture diagrams

The platform is not a bucket. It is a living map of obligations.

That is the architecture.

Context

In most enterprises, the data platform did not arrive as a designed whole. It accreted.

The ERP team exported nightly files. The digital team added clickstream events. Marketing bought a SaaS tool with its own customer IDs. A central integration team stood up an ESB, then APIs, then Kafka. A reporting team built a warehouse. Data science created its own ingestion paths because “the warehouse is too slow.” Compliance required lineage. Security required access control. Finance required immutable reconciliation. Eventually somebody declared the result a modern data platform.

And yet, when an executive asks a plain question — “where does revenue by customer segment actually come from?” — the answer is not plain. It snakes across operational systems, event logs, transformations, manually maintained mappings, business rule engines, and reports that carry old semantics because changing them would break three regulatory processes and a sales incentive model.

This is why architecture language matters. If you think of the platform as a pipeline, you will optimize for flow. If you think of it as a warehouse, you will optimize for storage and query. If you think of it as a dependency graph, you will ask harder and better questions:

• Which domains own which meanings?
• Which consumers are tightly coupled to upstream semantics?
• Which transformations are derived facts versus business policy?
• Which dependencies are stable enough to standardize?
• Which ones must remain local to a bounded context?
• How do we migrate without freezing the enterprise?

These are not abstract questions. They determine whether your next transformation program becomes a clean evolution or another layer of sediment.

Problem

The core problem is semantic coupling disguised as technical integration.

Enterprises usually discover this late. They see duplicated data, inconsistent metrics, brittle ETL jobs, and topic sprawl. They talk about data quality, interoperability, or governance. Those are symptoms. The disease is unmanaged dependency. EA governance checklist

A customer table is rarely just a customer table. In sales, a customer may be an account hierarchy. In billing, it may be a liable party. In servicing, it may be a contract holder. In digital channels, it may be an authenticated user or sometimes just a cookie stitched probabilistically to a known identity. When a central team tries to unify all of these too early into one canonical “Customer,” they often manufacture ambiguity at scale.

That is where domain-driven design matters. Not as ceremony. As survival.

Bounded contexts are a way to contain semantic blast radius. They tell you where a model is valid and where translation is required. A data platform that ignores bounded contexts becomes a giant semantic laundering operation: meaning goes in rich and local, comes out flattened and politically negotiated. Downstream teams then reverse-engineer what was lost.

Now add streaming.

Kafka and event-driven microservices can improve decoupling at the transport layer while quietly increasing semantic coupling at the organizational layer. Publishing an event does not eliminate dependency. It multiplies it if event contracts are vague, ownership is weak, and consumers begin to infer hidden behavior from event timing or sequence. I have seen enterprises with pristine topic naming conventions and utterly chaotic meaning. They had governance for schemas but not for truth. ArchiMate for governance

The result is a platform where no change is small.

Forces

Several forces make this difficult, and they pull in different directions.

Local domain truth versus enterprise consistency

Domains need freedom to model their world accurately. The enterprise needs some consistent cross-domain views for finance, risk, customer experience, and regulatory reporting. These goals are both legitimate and frequently in tension.

Speed versus semantic stability

Product teams want to evolve events and APIs quickly. Downstream analytics and controls want stable definitions. “Move fast” collides with quarter-end close.

Decentralized ownership versus platform efficiency

Domain-owned data products reduce central bottlenecks. But enterprises still benefit from shared infrastructure for ingestion, lineage, policy enforcement, storage, and observability. The trick is sharing platforms without centralizing semantics.

Real-time propagation versus reconciled truth

Streams are seductive. They promise everything now. But many enterprise processes need validated, complete, and reconcilable truth, not merely fast truth. A payment authorization event is useful immediately. Financial posting requires stricter controls.

Migration urgency versus operational continuity

Nobody gets to redesign a Fortune 500 data estate from scratch. Legacy warehouses, batch chains, MDM hubs, and hand-built extracts continue to matter. The new architecture must coexist with the old while gradually reducing dependence on it.

These forces explain why simplistic blueprints fail. The platform is not one pattern. It is a negotiated arrangement across many dependencies.

Solution

The solution is to architect explicitly around dependency graphs, bounded contexts, and staged truth.

That means several concrete things.

First, identify domains and their bounded contexts before you design shared data products. Domain semantics should be owned as close as possible to the business capability that creates and understands them. Sales owns sales semantics. Billing owns billing semantics. Risk owns risk semantics. This sounds obvious. In practice, enterprises centralize interpretation because they confuse data consolidation with semantic authority.

Second, model dependencies at three levels:

1. Operational dependency: service A needs event or state from service B to function.
2. Analytical dependency: report, metric, model, or data product depends on upstream facts and rules.
3. Control dependency: reconciliations, audits, regulatory outputs, and financial statements depend on governed transformations.

These dependencies have different tolerances for latency, change, and ambiguity. Treating them all the same is a category error.

Third, define staged truth. Not all data is equally final.

• Source truth: what a domain system records.
• Event truth: what the domain publishes as observable business facts.
• Integrated truth: what enterprise consumers derive by combining multiple domains.
• Reconciled truth: what has passed balancing, completeness, and control checks for high-stakes use.

This prevents a common anti-pattern: pretending streaming data is automatically authoritative because it is fresh.

Fourth, make reconciliation a first-class architectural concern. Reconciliation is not a finance-only annoyance. It is how a graph of dependencies remains trustworthy. If orders, payments, shipments, and returns do not reconcile across systems and derived datasets, the platform is not modern. It is merely busy.

Fifth, migrate with a progressive strangler strategy. Do not replace the whole graph in one move. Carve out dependency clusters, build new domain-aligned publishing and consumption paths, run parallel outputs, reconcile results, and then retire old edges deliberately.

A data platform evolves one dependency edge at a time.

Architecture

The target architecture is not a monolith and not a free-for-all mesh. It is a layered dependency graph with clear semantic ownership and controlled derivation.

At the edge sit operational systems and microservices aligned to business capabilities. Where events matter, they publish domain events to Kafka or an equivalent event backbone. These events are not “everything that happened in the database.” They are purposeful business facts shaped by the domain.

Above that sits a shared platform layer for transport, schema management, lineage capture, policy enforcement, cataloging, access controls, and observability. This layer standardizes mechanics, not meaning.

Then come domain data products. Each domain exposes datasets, streams, or APIs with explicit contracts, quality expectations, and semantic definitions. These are not just tables with nice names. They are owned products with lifecycle management.

Enterprise-integrated products sit above or beside them, depending on the organization. These combine domain outputs for cross-domain use cases: customer profitability, order-to-cash visibility, fraud analytics, liquidity monitoring, and so on. This is where translation between bounded contexts happens. It should be visible and governed, not hidden in dashboards or ad hoc notebooks.

Finally, there are control-grade outputs: financial marts, regulatory reports, immutable audit views, and reconciled analytical stores.

Here is a simple view of the dependency graph:

The key point is subtle but important: enterprise products depend on domain products, not directly on raw operational systems wherever possible. That is how you control semantic spread.

Domain semantics and translation

A healthy platform does not force one universal model where the business has multiple legitimate meanings. It creates explicit translation points.

Take “customer.” The billing domain may produce liable_party_id. Sales may produce account_id. Digital may produce person_profile_id. A customer 360 projection can map these into a usable integrated view, but it should never erase the original semantics. The integrated model is derived truth, not universal truth.

This is where many data lake programs go wrong. They centralize storage before clarifying meaning. They ingest first and negotiate semantics later. Later never comes.

Event backbone, but with restraint

Kafka is valuable when you need decoupled propagation of domain facts, multiple consumers, replayability, and near-real-time processing. It is especially effective for order lifecycle events, payment state changes, inventory movements, and customer interaction streams.

But Kafka is not a data strategy. It is a transport and log abstraction with strong utility. If a team uses Kafka to publish low-level CRUD changes from a legacy database and calls that domain architecture, they are exporting confusion at scale.

Use event streams where the business event matters. Use APIs where request-response semantics matter. Use batch where completeness windows and cost profiles matter more than latency. Good architecture is plural.

Migration Strategy

The migration path matters more than the end-state slide.

A large enterprise already has dependencies embedded in ETL jobs, warehouse schemas, interfaces, and spreadsheets hidden behind business processes. The goal is not to leap over this reality. The goal is to reshape it safely.

The most effective approach is progressive strangler migration applied to dependency clusters.

Start with a business flow, not a technology domain. Order-to-cash is a classic candidate because it crosses sales, order management, billing, payment, and finance. Map the current dependencies: source systems, extracts, jobs, reports, reconciliations, service calls, and manual interventions. You are looking for semantic choke points and unstable edges.

Then select a narrow slice to modernize. For example, publish order lifecycle events from the order domain, build a domain-owned order data product, feed a new order-to-cash integrated product, and run it in parallel with the existing warehouse pipeline. Reconcile outputs. Prove equivalence or explain deltas. Only then move consuming reports or services.

This is not glamorous. It is architecture that respects consequences.

Why reconciliation must sit in the middle

Parallel running without reconciliation is theater. You may show green dashboards while quietly changing the meaning of metrics.

Reconciliation should test several dimensions:

• record counts and completeness
• key balances, totals, and financial amounts
• event-to-state consistency
• timing windows and late-arrival effects
• semantic rule equivalence
• exception populations

In streaming architectures, one of the hardest issues is temporal mismatch. A warehouse fed nightly may show a stable order count by business date; a stream-fed product may show changes continuously with late corrections. Both can be “right” and still disagree. Reconciliation must account for this with clear cut-off rules and versioned business calendars.

Strangle dependencies, not just systems

A common migration mistake is to move ingestion to the cloud while preserving all legacy semantic dependencies intact. This is relocation, not modernization.

A better sequence is:

1. expose domain events or APIs from an authoritative context
2. create domain-owned products with explicit semantics
3. shift downstream consumers from raw legacy structures to new products
4. retire obsolete transformations and hidden extracts
5. only then decommission legacy storage or integration paths

Retiring a table is easy. Retiring a dependency is architecture.

Enterprise Example

Consider a multinational insurer modernizing its claims and finance data landscape.

The company had grown through acquisition. Claims processing existed in four regional platforms. Finance ran a central ledger. Customer data was split across policy administration, CRM, and broker portals. Reporting was driven by a large warehouse fed nightly, plus dozens of region-specific marts. Every quarter-end, finance and claims operations spent days reconciling reserves, payouts, recoveries, and claim statuses. Data quality was blamed. The deeper issue was semantic drift.

A “claim” was not one thing.

In one region, a claim record represented a case. In another, it represented an incident exposure. In finance, reserve calculations were attached to postings. In analytics, teams joined all of them into a single claim grain because the warehouse encouraged a universal fact model.

The modernization program initially proposed a global canonical claims schema and a cloud lakehouse. That design looked neat. It would have failed.

The better move was to model bounded contexts explicitly: First Notice of Loss, Claim Handling, Payment, Recovery, Policy, and Finance Posting. Each context published domain events through Kafka where timeliness mattered: claim opened, reserve adjusted, payment authorized, payment settled, recovery recorded. Domain teams also published governed batch snapshots where full-state completeness mattered.

A shared platform provided schema governance, lineage, quality checks, and policy enforcement. But it did not define the meaning of a claim globally. Instead, an enterprise claims-finance integrated product translated domain outputs into a set of reconciled measures needed for regulatory and management reporting.

The migration started with one region and one flow: reserve movement to finance. New events were published from the claims domain, consumed into a reserve movement product, then compared in parallel against the warehouse-derived finance view. Reconciliation uncovered three classes of discrepancy:

• timing differences due to same-day reversals arriving after finance cut-off
• semantic differences where “approved reserve” in claims did not equal “posted reserve” in finance
• reference mapping errors for acquired legal entities

This was not failure. This was discovery. The old platform had hidden these mismatches under manual spreadsheet adjustments. The new architecture made them visible.

Within 12 months, the insurer had moved several reporting domains off direct warehouse dependence and onto domain-aligned products. Quarter-end close improved not because Kafka was magical, but because semantic ownership and reconciliation were built into the graph.

Here is the kind of flow that emerged:

That is a real enterprise pattern. Messy, staged, and honest.

Operational Considerations

Once you accept the platform as a dependency graph, operations change.

Lineage is not metadata vanity

Lineage must show not just technical hops but semantic derivations. Which dataset calculates recognized revenue? Which upstream business rules shape exposure at default? Which reports use “active customer” based on login behavior versus contract status? If your catalog only knows table-to-table movement, it knows too little.

Observability must include contract health

Monitoring topic lag, job duration, and storage cost is table stakes. You also need to observe:

• schema evolution and breaking contract risk
• freshness by data product SLA
• null and cardinality anomalies
• business control thresholds
• reconciliation exception trends
• consumer concentration around fragile upstreams

A platform can be technically healthy and semantically sick.

Access control follows domain semantics

Security models should align with business meaning. Claims adjuster notes, health information, payment details, and regulatory classifications should not be abstracted into generic row-level policy after the fact if domain contexts already know the sensitivity. Domain ownership and policy-as-code should work together.

Versioning is a business matter

Schema versioning is necessary but insufficient. You must version semantics too: rule sets, reference data mappings, cut-off calendars, classification logic. Many “mysterious” report changes come from untracked business-rule evolution.

Data quality should be executable at the edge

Quality checks belong as early as possible, ideally near domain publication. That includes mandatory fields, referential assertions, sequence integrity, and allowed state transitions. Central quality teams can define standards, but domains must own validity of what they emit.

Tradeoffs

There is no perfect data platform, only chosen compromises.

A dependency-graph approach with domain alignment improves semantic clarity and change management. It also introduces coordination overhead. Teams must own contracts. Translation layers become explicit artifacts rather than hidden convenience. Some stakeholders will complain that the architecture is less “simple” than a giant centralized warehouse with one enterprise model.

They are half right. Centralization looks simpler on the whiteboard. It just charges interest later.

Kafka-based event propagation improves timeliness and decouples transport. It also raises the bar for contract design, replay handling, idempotency, ordering assumptions, and temporal reasoning. If teams are not mature in these disciplines, streaming can amplify confusion.

Domain data products distribute ownership. They can also create uneven quality if platform standards are weak or if domains are under-resourced. A federated model without strong platform enablement becomes a polite version of chaos.

Reconciliation increases trust. It also adds cost and delays cutover. Executives often want the migration story without the reconciliation budget. That is how they end up with fast new pipelines and old manual controls still lurking underneath.

The right architecture admits these tradeoffs plainly.

Failure Modes

A few failure modes show up repeatedly.

The canonical model trap

The enterprise creates a grand unified business model too early. Domains comply superficially, then smuggle local semantics into overloaded attributes and custom extensions. The platform becomes consistent only in naming, not in meaning.

Event exhaust masquerading as domain events

Teams publish database changes rather than business facts. Consumers infer intent from table mutation patterns. Every schema change becomes an enterprise incident.

Data mesh without product discipline

The organization adopts the language of data products but not the responsibilities. No SLAs, no quality metrics, no lifecycle ownership, no deprecation policy. Decentralization becomes abandonment.

Migration by duplication

The new cloud platform simply mirrors the old warehouse logic. Technical debt is copied at higher compute cost. The dependency graph remains unchanged, just more expensive.

No reconciliation during parallel run

Differences emerge after cutover, in production, under audit. Confidence collapses. The old platform lingers forever because nobody trusts the new one enough to turn it on fully.

Hidden control dependencies

A report seems easy to migrate until someone discovers an end-user macro, a risk adjustment spreadsheet, or a month-end manual booking process depending on the old output. These are not edge cases. They are part of the system.

When Not To Use

This approach is not universally necessary.

If you are a smaller organization with a handful of systems, one analytics team, and modest regulatory pressure, a straightforward warehouse or lakehouse with well-managed ingestion may be entirely sufficient. You do not need a rich dependency-graph operating model when the graph itself is small and understandable.

Likewise, if your domain boundaries are weak, your operational systems are deeply entangled, and your organization lacks product ownership discipline, then introducing domain data products and event-driven contracts may simply expose dysfunction faster than you can fix it. Sometimes the first right step is consolidating systems and clarifying business capability ownership, not adding Kafka.

Do not use streaming where business latency does not justify the complexity. A nightly batch for supplier master updates is often perfectly respectable. Architecture should have a sense of proportion.

And do not force domain-driven design vocabulary where the business does not have meaningful bounded contexts. Some support functions are genuinely simpler with shared reference data and straightforward stewardship.

The point is not to romanticize decentralization. The point is to align architecture with the shape of meaning and change.

Related Patterns

Several related patterns sit naturally alongside this approach.

Domain-driven design provides the core semantic framing: bounded contexts, ubiquitous language, context mapping, and anti-corruption layers.

Data products provide the operating model for exposing domain-owned, consumable outputs with clear contracts and quality expectations.

Event sourcing can be useful in some domains where the event history itself is primary, though it is often overused. Most enterprises need event publication more often than full event-sourced persistence.

CQRS helps when read models differ materially from transactional models, especially for integrated projections and operational analytics.

Progressive strangler is the migration workhorse. It lets old and new coexist while you shift dependency edges safely.

Reconciliation architecture deserves recognition as a pattern in its own right: balancing records and amounts, managing timing windows, surfacing exceptions, and proving control equivalence across old and new flows.

Lakehouse and warehouse patterns still matter. They remain useful implementations for integrated and reconciled data stores. The graph view does not replace storage choices; it gives them proper context.

Summary

Your data platform is not a lake, not a warehouse, not a mesh, and certainly not a pile of tools held together by naming conventions. It is a dependency graph of facts, meanings, promises, and controls.

Once you design for that reality, better choices follow.

You stop pretending one canonical model can own every business concept. You use domain-driven design to place semantics where they belong. You publish domain facts deliberately, with Kafka where event propagation adds real value and with APIs or batch where they fit better. You build integrated data products as explicit translations across bounded contexts. You treat reconciled truth as a distinct and necessary stage for enterprise control. And you migrate progressively, strangling old dependencies edge by edge instead of staging heroic rewrites.

The memorable line is this: storage is where data rests; architecture is how meaning depends.

That is why so many data transformations disappoint. They move data without redesigning dependency.

The enterprises that succeed do something harder and more useful. They make the graph visible. They decide which edges deserve to exist. They govern semantics without centralizing everything. They reconcile before they celebrate. And over time, they turn a platform shaped by accident into one shaped by intent.

That is not fashionable architecture.

It is the kind that survives quarter-end, audit season, acquisitions, and the next “strategic” replatforming.

Which, in enterprise terms, is the only kind that counts.

Frequently Asked Questions

What is a data mesh?

A data mesh is a decentralised data architecture where domain teams own and serve their data as products. Instead of a central data team, each domain is responsible for data quality, contracts, and discoverability.

What is a data product in architecture terms?

A data product is a self-contained, discoverable, trustworthy dataset exposed by a domain team. It has defined ownership, SLAs, documentation, and versioning — treated like a software product rather than an ETL output.

How does data mesh relate to enterprise architecture?

Data mesh aligns data ownership with business domain boundaries — the same boundaries used in domain-driven design and ArchiMate capability maps. Enterprise architects play a key role in defining the federated governance model that prevents data mesh from becoming data chaos.