⏱ 22 min read
Everyone wants to talk about models.
The board asks about model accuracy. Product teams ask whether we can add recommendations, anomaly detection, forecasting, copilots, fraud scoring, demand sensing. Vendors arrive with polished demos and words like real-time, autonomous, and AI-native sprinkled over everything like parsley on bad pasta.
But in real enterprises, the hardest part of an AI platform is rarely the model.
It is freshness.
Not freshness in the hand-wavy sense of “near real-time.” Freshness as a hard architectural property: how quickly the platform can turn business events into trustworthy features without breaking semantic consistency across systems. Freshness is not just latency. It is the gap between reality and what your AI system believes reality to be.
That gap is where projects quietly fail.
A model can be elegant and still useless if the customer profile is six hours old, the payment state is stale, inventory updates arrive out of order, and the feature store says a user is “active” because a nightly batch job has not yet caught up with a cancellation event. By the time the model scores, the world has moved on. The decision is mathematically precise and operationally wrong.
This is why feature pipeline topology matters. Not as diagram theater. Not as platform branding. But as the actual shape of truth moving through the enterprise.
If you build AI platforms long enough, you learn a blunt lesson: the architecture of data movement determines the practical value of machine learning more than the sophistication of the model itself. A mediocre model on fresh, semantically correct features often beats a brilliant model fed by stale, inconsistent ones.
So this article is about that harder problem. How to think about data freshness through the lens of domain-driven design, event streaming, reconciliation, and migration. How to design feature pipeline topology for large enterprises where data does not live in one clean place, where systems disagree, where microservices publish half-true events, and where Kafka can either save you or amplify your chaos. event-driven architecture patterns
Context
AI platforms in enterprises are usually born from an uncomfortable compromise.
Operational systems run the business: order management, payments, claims, CRM, ERP, subscription billing, contact center, e-commerce, logistics. Analytical systems explain the business after the fact: warehouses, lakes, marts, BI models, semantic layers. AI systems sit in between and demand something both worlds are bad at providing simultaneously: current data with stable meaning.
This is the root tension.
Operational systems are fresh but fragmented. Analytical systems are integrated but late. AI workloads want low-latency features, historical consistency, replayability, lineage, and online serving. They want a training set, an online feature vector, and an audit trail that all mean the same thing. That sounds reasonable until you try to build it in a company that has grown by acquisition and where “customer” means one thing in billing, another in sales, and a third in support.
This is where domain-driven design matters. Not because bounded contexts are fashionable, but because freshness without domain semantics is just fast confusion.
A feature like customer_lifetime_value_90d is not a mere calculation. It carries assumptions:
- What is a customer?
- What counts as revenue?
- Is a refund netted?
- Which timestamp matters: authorization, settlement, invoice, shipment, or cash application?
- What happens when an order is amended after shipment?
- Which events are authoritative?
The technical pipeline cannot answer these questions on its own. These are domain questions. If they are left implicit, teams create feature drift disguised as implementation variance.
In other words: the pipeline topology is not merely an integration design. It is a map of domain truth.
Problem
Most AI platforms inherit one of two broken patterns.
The first is the warehouse-first feature platform. Teams ingest everything into a lake or warehouse, transform with batch jobs, and publish features from curated tables. This works well for experimentation, model training, and broad reporting. Then someone wants real-time personalization, dynamic fraud scoring, or operational next-best action. Suddenly “daily refresh” becomes “every 15 minutes,” then “micro-batch,” then “streaming,” and the platform starts bolting low-latency paths onto a batch spine that was never designed for operational immediacy.
The second is the event-everything platform. Every microservice emits events to Kafka, platform engineers build stream processors, and the organization congratulates itself for achieving event-driven AI. Then reality arrives. Event contracts are inconsistent, services emit technical events instead of business facts, reference data changes are poorly propagated, replay semantics are vague, and the online features diverge from training features because one path is streaming and the other is SQL over snapshots.
Both patterns miss the same point: freshness is a business property, not a transport property.
A feature is fresh only if:
- the relevant source event has been captured,
- the event has the right domain meaning,
- ordering and deduplication have been resolved appropriately,
- derived state has been recomputed,
- the online serving layer reflects it,
- the training and inference definitions remain aligned.
If any of these fail, the feature may be low-latency and still not be fresh in any meaningful sense.
This is where teams get trapped. They optimize ingestion latency while ignoring reconciliation. They obsess over stream processing frameworks while avoiding semantic ownership. They install a feature store and assume consistency comes free with the license.
It does not.
Forces
There are several forces pulling the architecture in conflicting directions.
1. Freshness versus correctness
The faster you compute features, the less time you have to resolve late arrivals, corrections, reference-data joins, and cross-domain dependencies.
A fraud model may tolerate a provisional device-risk feature that is corrected later. A credit decision probably cannot. A recommendation engine can live with eventual consistency. Financial exposure calculations should be much more conservative.
This is why one universal freshness target is nonsense. Different domains carry different business tolerances.
2. Domain autonomy versus enterprise consistency
Microservices encourage local ownership. That is good. But AI features often cut across domains: customer behavior, product interactions, payment history, support sentiment, fulfillment reliability. No single service owns the full truth.
If every domain emits events independently without a shared semantic contract, the feature platform becomes an archaeological dig. Teams spend more time inferring meaning than building value.
DDD helps here. Bounded contexts should retain autonomy, but key business events need explicit semantics and translation rules between contexts.
3. Online serving versus offline training parity
Training pipelines often use large historical tables. Inference pipelines often use stream-derived materialized views or key-value lookups. The two drift apart subtly:
- different null handling,
- different window boundaries,
- different identity resolution,
- different event correction logic,
- different source-of-truth timestamps.
This training-serving skew is not just a model issue. It is usually an architecture issue.
4. Throughput versus replayability
Kafka gives you durable logs and replay. That is excellent. But replaying a high-volume topology with stateful enrichments, changing schemas, and downstream side effects is where architectural innocence goes to die.
Replayability is easy in slides, hard in production.
5. Central platform control versus federated delivery
A central AI platform team wants reusable standards, feature definitions, governance, lineage, observability, and controlled serving. Domain teams want speed and local adaptation. If the platform becomes too centralized, it becomes a bottleneck. If it becomes too loose, feature semantics fragment. EA governance checklist
The trick is to centralize contracts, capabilities, and observability, while federating domain-specific feature production.
Solution
My preferred answer is a layered feature pipeline topology built around domain events, semantic feature definitions, and explicit reconciliation.
This is not a pure batch architecture. It is not a pure streaming architecture either. It is a topology where different paths serve different freshness and correctness needs, but all paths are anchored in the same domain model.
The essential ideas are these:
- Use domain events, not service internals, as the backbone of freshness.
- Separate canonical business facts from derived AI features.
- Support both streaming and batch derivation from the same semantic definitions where possible.
- Treat reconciliation as a first-class architectural capability, not an afterthought.
- Design for progressive migration using a strangler approach rather than big-bang feature platform replacement.
The topology usually has four layers:
- Operational event capture
- CDC from systems of record where events are missing.
- Native business events from microservices where event contracts are reliable. microservices architecture diagrams
- Reference data and master data propagation.
- Domain fact layer
- Durable, semantically governed streams or tables representing business facts.
- Identity resolution and key mapping.
- Event normalization into bounded-context vocabulary.
- Feature computation layer
- Streaming features for latency-sensitive use cases.
- Batch/backfill features for historical consistency and replay.
- Shared transformation logic or at least shared declarative definitions.
- Serving and reconciliation layer
- Online feature serving for inference.
- Offline historical store for training.
- Reconciliation jobs to compare expected versus observed state and repair drift.
That last point matters more than teams expect. Freshness is not maintained only by fast movement. It is maintained by continuous correction.
Architecture
A sensible topology starts with a distinction many organizations resist: events are not features.
Events are facts that happened in the domain. Features are interpretations of those facts for a predictive purpose.
Conflating the two creates brittle systems. If one model wants “cart additions in 10 minutes” and another wants “net cart value excluding coupon-only SKUs,” those are both derived from commerce events, but they are not the events themselves. Keep that separation clean.
Reference topology
This looks straightforward. It never is.
Domain semantics first
In a DDD sense, the domain fact layer is where bounded contexts meet the platform. An OrderPlaced event in commerce is not the same thing as InvoiceIssued in billing or PaymentCaptured in payments. They may refer to the same customer journey, but they mean different things and have different timing characteristics.
A healthy feature platform does not flatten these distinctions too early.
For example:
- The Commerce context owns browsing, carting, and ordering intent.
- The Payments context owns authorization, capture, reversal, and chargeback facts.
- The Fulfillment context owns allocation, shipment, and delivery.
- The Customer context may own identity preferences and segmentation.
- The Support context owns complaint and interaction history.
A feature like “likelihood to churn” might draw from all of them. But the semantics should be assembled intentionally, not by blindly joining whatever landed in the lake.
This is why I favor a canonical fact layer, not a grand canonical data model. There is a difference. The layer standardizes business facts and IDs enough for reuse, while respecting bounded contexts and preserving source meaning.
Streaming where it matters
Not every feature should be streaming.
Use streaming when:
- the decision is operational and time-sensitive,
- the domain event cadence is high enough to justify it,
- provisional answers are acceptable or reconcilable,
- the business value of reduced staleness is material.
Examples:
- fraud scoring during payment authorization,
- dynamic customer support routing,
- recommendation updates during browsing,
- logistics exception prediction during fulfillment.
Use batch when:
- historical consistency is more important than instant response,
- source systems issue frequent corrections,
- event completeness is poor,
- cost and complexity of stream state would exceed the value.
Examples:
- monthly propensity refresh,
- risk segmentation using slowly changing financial attributes,
- workforce planning forecasts,
- strategic demand planning.
Feature parity by design
The architecture should not permit online and offline features to become cousins who barely speak.
The practical pattern is:
- define feature semantics declaratively where possible,
- generate or share transformation logic across batch and stream engines,
- version feature definitions explicitly,
- store timestamps for event time, processing time, and effective business time,
- materialize both online and offline outputs from the same fact layer.
If exact code reuse is impossible, semantic tests become mandatory. A feature definition is only credible if batch and stream outputs can be compared over representative windows.
Reconciliation is the safety net
Enterprises love the fantasy of perfect event-driven truth. The real world is messier:
- a source system misses an event,
- CDC lags or duplicates,
- a producer deploy changes payload shape,
- a lookup table updates late,
- identity mapping changes after the fact,
- downstream consumers drop state during rebalance,
- backfills overwrite materialized views in strange ways.
Reconciliation is how adults build AI platforms.
You need jobs and services that ask:
- Does the feature store reflect the current domain fact state?
- Do counts and aggregates match source-of-truth windows?
- Did late events alter previously served features?
- Are online feature values within tolerance of offline recomputation?
- Which entities require repair or replay?
Here is the key line: reconciliation is not evidence of failure; it is evidence of realism.
Migration Strategy
Most enterprises already have a mess that works just enough to be dangerous:
- nightly ETL into a warehouse,
- model training on curated marts,
- ad hoc Redis caches for online features,
- some Kafka topics from newer microservices,
- point-to-point APIs filling gaps,
- a feature store implementation somewhere in the middle, often underused.
You do not replace this in one move. You strangle it.
Progressive strangler migration
Start with the feature use cases that have the clearest business need for freshness and the smallest semantic blast radius. Do not begin with “enterprise customer 360” unless you enjoy slow disappointment.
A good migration sequence often looks like this:
- Identify one bounded context with reliable events
- e-commerce interactions,
- support case updates,
- payment authorization outcomes.
- Build a domain fact stream or table
- not generic raw ingestion,
- explicit business facts with ownership and schema governance. ArchiMate for governance
- Compute a small set of high-value features
- one online use case,
- one offline training flow,
- one reconciliation process.
- Compare against the legacy batch path
- measure freshness,
- compare values,
- resolve semantic differences before scaling.
- Route one model or decision service to the new path
- canary release,
- shadow scoring,
- business outcome monitoring.
- Expand context by context
- add supporting domains only as needed,
- avoid boiling the ocean.
Migration topology
The migration hinge is parity and reconciliation. Teams often skip this because they are under pressure to show movement. That is a mistake. If the new topology is fresher but semantically different, the model behavior can change in ways that look like model degradation but are actually feature definition changes.
CDC is often the bridge
Where source systems cannot publish proper business events, change data capture can be an excellent transitional tactic. Purists complain that CDC is not domain-driven enough. They are partly right. CDC gives you state changes, not business intent.
But for migration, CDC is often the only practical way to bootstrap freshness from legacy systems. The trick is to promote CDC records into domain facts rather than exposing raw table changes directly to feature consumers. That translation layer is where semantics are recovered.
Enterprise Example
Consider a global retail bank building an AI platform for fraud detection, customer retention, and next-best action across cards, deposits, and digital channels.
Like many banks, it has:
- a mainframe core for accounts,
- card authorization systems,
- a digital banking platform built as microservices,
- CRM and case management products,
- a central data warehouse,
- Kafka used heavily by newer channels but not by the core.
The initial AI platform was warehouse-led. Fraud analytics trained on daily snapshots. Marketing models refreshed weekly. Customer service recommendations were rebuilt every few hours. All of this looked respectable until the bank tried to use AI in operational channels:
- stop likely fraud during authorization,
- intervene during a digital session when a customer appears at risk of attrition,
- route support calls using current account distress signals.
The existing platform could not do it. The data was “integrated” but old. Worse, “account status” meant one thing in card systems and another in core banking. Chargeback events arrived through a different path from card authorizations. Support case closure lagged by hours. Training data stitched all of this together after the fact, but online scoring had no equivalent view.
The bank adopted a layered feature topology.
Step one: establish domain fact streams for card authorization outcomes, digital session events, support case status changes, and account lifecycle changes. Some came from microservices, some from Kafka wrappers, some from CDC off legacy databases.
Step two: define canonical facts with business ownership:
CardAuthorizationEvaluatedChargebackRegisteredDigitalSessionAuthenticatedSupportCaseOpenedSupportCaseClosedAccountDelinquencyStatusChanged
These were not generic table changes. They were named as domain events with clear semantics and timestamps.
Step three: compute a narrow set of features:
- failed authorizations in last 30 minutes,
- customer support interactions in last 7 days,
- digital login anomalies over trailing sessions,
- unresolved dispute count,
- days since delinquency transition.
Step four: run shadow scoring against the existing warehouse-derived features and compare outputs. This was sobering. Some features differed by more than 20% because the warehouse path used posting date while the streaming path used authorization time. The architecture work uncovered a domain problem, not a technical bug.
That is how these programs really go. The topology exposes the organization’s semantic debts.
Eventually the bank cut over fraud scoring first, because freshness had direct economic value. Retention and next-best action followed later, with more tolerance for eventual consistency. The result was not one grand AI platform victory. It was a sequence of domain-specific wins built on a more honest data architecture.
That is how mature enterprise architecture earns trust.
Operational Considerations
A feature pipeline topology fails operationally long before it fails conceptually.
Observability must be end-to-end
You need telemetry for:
- source event lag,
- Kafka topic backlog,
- consumer delay,
- state-store growth,
- feature materialization latency,
- online serving p99,
- parity drift between online and offline outputs,
- reconciliation backlog,
- schema change events.
If you cannot tell how stale a feature is for a given entity, you are not operating a freshness-aware platform. You are just moving data quickly and hoping for the best.
Time semantics matter
Store and reason about at least three clocks:
- event time: when the business fact happened,
- processing time: when the pipeline saw it,
- effective time: when the fact became authoritative for the domain.
This distinction matters enormously in finance, healthcare, supply chain, insurance, and telecom, where corrections and retroactive adjustments are common.
Identity resolution is often the hidden bottleneck
Feature freshness collapses if entity identity is unstable. If customer IDs differ across channels, if householding rules change, if device graphs evolve, or if account merges happen after the fact, then “fresh features” may be attached to the wrong entity.
Identity is not a utility function. It is a domain capability. Treat it accordingly.
Schema governance must be pragmatic
Use schema registry, compatibility checks, event versioning, and contract tests. But avoid the fantasy that governance alone creates semantic quality. It only prevents certain classes of breakage.
A beautifully versioned meaningless event is still meaningless.
Tradeoffs
There is no free lunch here. Feature pipeline topology is all about choosing which pain you can afford.
Streaming increases value and complexity
Streaming can materially improve operational AI. It can also produce elaborate failure chains, expensive stateful infrastructure, and difficult replay behavior. Use it where freshness changes business outcomes, not because platform engineering enjoys low-latency systems.
Canonical facts improve reuse and create governance load
A domain fact layer reduces duplicate interpretation and helps feature parity. It also requires business ownership, schema discipline, and cross-team negotiation. Some organizations are culturally incapable of this. In such places, a lighter-weight federated model may be healthier.
Reconciliation improves trust and delays closure
Teams want to declare a pipeline “done.” Reconciliation refuses that fantasy. It creates recurring work, repair flows, and operational overhead. But without it, silent drift accumulates until trust disappears.
A central platform accelerates common patterns and risks becoming a chokepoint
The more standards you create, the more reusable and governable the platform becomes. The more teams must wait for you. The art is to provide paved roads, not permission bottlenecks.
Failure Modes
These platforms usually fail in familiar ways.
1. Fast but semantically wrong
The platform ingests events in seconds but computes features on unstable or mismatched domain concepts. The model is fresh and wrong. This is the most dangerous failure because dashboards look good.
2. Online/offline divergence
Training features and serving features drift. Model performance degrades in production. Teams blame the model, retrain repeatedly, and never fix the topology.
3. Event overproduction
Microservices emit too many technical events and not enough business facts. Consumers become tightly coupled to service implementation details. The platform becomes brittle.
4. No replay discipline
Backfills and replays mutate downstream state inconsistently. Some features are recomputed, some are not. Materialized views become historical accidents.
5. Reference data blindness
Product hierarchy, customer segment definitions, currency rules, branch mappings, and risk classifications update out of band. Feature values become inconsistent even when event streams are healthy.
6. Reconciliation absent or toothless
The organization discovers discrepancies through business complaints rather than platform controls. By then, trust is already gone.
A useful way to frame failure is this:
The point is simple. A stale or wrong feature is not a data issue in isolation. It becomes a business action issue.
When Not To Use
This architecture is not universally appropriate.
Do not build a sophisticated streaming feature topology when:
- your use cases are primarily batch scoring,
- source systems are so poor that semantic normalization would dominate all progress,
- business decisions tolerate hours or days of lag,
- your model lifecycle maturity is still low,
- you do not yet have stable domain ownership,
- the organization lacks operational discipline for event-driven platforms.
In many enterprises, a well-run batch feature platform is better than a half-built streaming one. There is no shame in that. Architecture is not a race to the most fashionable topology.
Also, if your domain is heavily document-centric, low-frequency, and manually adjudicated—certain legal, HR, or strategic planning scenarios, for example—the cost of freshness infrastructure may not pay back.
A memorable rule: if the business does not monetize minutes, do not engineer milliseconds.
Related Patterns
Several related patterns often accompany this topology.
Data mesh, but with restraint
A federated ownership model fits naturally with domain-oriented feature production. But data mesh slogans can become an excuse for semantic fragmentation. Domain ownership works only when shared contracts and interoperability are real.
Lambda and Kappa ideas, minus the dogma
The old batch-plus-speed-layer thinking still has practical value. So does the “log as system of record” mindset. But most enterprises need a hybrid. Dogma is less useful than disciplined dual-path design with clear parity checks.
CQRS for feature serving
Separating write-side domain fact ingestion from read-optimized feature serving is often helpful. Especially where online inference requires low-latency lookups and point-in-time consistency.
Event sourcing in narrow domains
For some bounded contexts, event sourcing can be a good source for reconstructable feature histories. But applying it universally across the enterprise is usually overreach.
Master data and reference data management
Not glamorous, but indispensable. Feature topology without reference-data discipline is a high-speed route to inconsistent inference.
Summary
The hardest part of AI platforms is not model selection, vector databases, GPUs, or orchestration frameworks.
It is getting the right business truth to the right decision point before that truth goes stale.
That is what data freshness really means. Not low transport latency. Not a streaming badge. Freshness is the timely availability of semantically correct, reconcilable domain facts and derived features.
Feature pipeline topology is how you make that happen. The good architectures share a few characteristics:
- they start from domain semantics, not tool choices,
- they distinguish facts from features,
- they use streaming selectively where freshness changes outcomes,
- they preserve training-serving parity,
- they build reconciliation into the platform,
- they migrate progressively through strangler patterns rather than big-bang replacement.
Kafka helps, but Kafka is not the answer. Microservices help, but only if they emit meaningful business events. Feature stores help, but they do not remove the need for semantic ownership. DDD helps most of all because it forces the platform to respect the shape of the business rather than flattening it into anonymous data exhaust.
In enterprise architecture, the winning designs are usually less magical than people hope and more disciplined than people expect.
AI platforms are no exception.
Models may be the engine. But freshness is the fuel line. And in production, fuel lines matter more than slideware.
Frequently Asked Questions
What is cloud architecture?
Cloud architecture describes how technology components — compute, storage, networking, security, and services — are structured and connected to deliver a system in a cloud environment. It covers decisions on scalability, resilience, cost, and operational model.
What is the difference between availability and resilience?
Availability is the percentage of time a system is operational. Resilience is the ability to recover from failures — absorbing disruption and returning to normal. A system can be highly available through redundancy but still lack resilience if it cannot handle unexpected failure modes gracefully.
How do you model cloud architecture in ArchiMate?
Cloud services (EC2, S3, Lambda, etc.) are Technology Services or Nodes in the Technology layer. Application Components are assigned to these nodes. Multi-region or multi-cloud dependencies appear as Serving and Flow relationships. Data residency constraints go in the Motivation layer.