Kafka Architecture Explained for Enterprise Architects

⏱ 6 min read

The mental model: Kafka as a distributed commit log

Kafka’s official introduction explains that topics are implemented as partitioned logs—append-only sequences of records—where offsets represent record positions. This log foundation is why Kafka supports replay, scalable parallelism, and durable history-based processing. turn16view0

Core cluster components: brokers, partitions, leaders, and replication

Kafka clusters distribute log partitions across servers and replicate partitions across multiple servers for fault tolerance. The leader handles read/write requests; followers replicate and can be promoted on failure. turn16view0

Figure 1: Kafka architecture layers — producers, cluster, consumers, and supporting components — **Figure 1:** Kafka architecture layers — producers, cluster, consumers, and supporting components

From an enterprise architecture perspective, these mechanics map directly to non-functional concerns: Sparx EA best practices

Availability is tied to replication and leader election
Throughput and scalability are tied to partitioning
Data locality and disaster recovery may involve replication across regions (e.g., mirror replication concepts) turn16view0

Producers and partitioning strategy

Producer responsibility includes selecting partitions, either round-robin for load balance or key-based for semantic grouping. Key-based partitioning is typically how enterprises preserve ordering for a business entity (order ID, customer ID) while still scaling throughput. turn16view0

Figure 2: Message lifecycle — from producer through partitioning and replication to consumer — **Figure 2:** Message lifecycle — from producer through partitioning and replication to consumer

This is a critical architectural decision: Kafka provides total order only within a partition, not across partitions. turn16view0

Consumers, offsets, and consumer groups

Kafka’s consumer model is built on consumer groups: each record is delivered to one consumer per group, and partitions are assigned so each partition is consumed by exactly one consumer within a group at a time, preserving per-partition order while enabling parallelism. turn16view0

The architecture also enables multiple independent groups to consume the same topic (broadcast across groups), which is a common enterprise integration pattern for decoupling domains. turn16view0 integration architecture diagram

Metadata management and KRaft controllers

Recent Kafka operations documentation describes KRaft mode, where Kafka servers can act as brokers, controllers, or both via configuration, and controllers participate in a metadata quorum. The documentation also provides operational guidance: combined broker/controller is simpler but not recommended for critical deployments because controllers are less isolated and cannot be rolled/scaled independently. turn18view0

It also notes typical controller quorum sizing (commonly 3 or 5 controllers) and explains that a majority must be alive for availability, providing explicit failure-tolerance examples. turn18view0

Architecture impacts on governance and enterprise operating model

Kafka can become a “shared enterprise nervous system,” which introduces governance needs: schema evolution, compatibility, and data contracts. Documentation for schema governance tooling describes schema registries as centralized repositories with compatibility checking and versioning, and it describes “data contracts” concepts supported via tags, metadata, and rules. turn19view0turn19view1

For enterprise architects, this means Kafka architecture decisions (topic naming, keying, retention, ownership) must be paired with governance decisions (ownership, contracts, access controls). turn19view1turn17view0 free Sparx EA maturity assessment

Frequently asked questions

Does Kafka replace all messaging?

Not automatically. Kafka’s architecture is optimized for durable event history and scalable stream processing, whereas traditional brokers may be stronger for complex routing patterns. Architectural selection depends on workload semantics and routing needs. turn16view0

Topic and partition internals

Figure 3: Topic internals — partitions with leaders, replication across brokers, and multiple consumer groups — **Figure 3:** Topic internals — partitions with leaders, replication across brokers, and multiple consumer groups

Understanding topic internals is essential for enterprise architects making capacity and reliability decisions. A topic is divided into partitions — each partition is an ordered, immutable sequence of records. Each partition has one leader broker (handles reads and writes) and N-1 follower brokers (replicate from the leader). The replication factor (RF) determines how many copies exist — RF=3 means each partition is stored on three brokers.

Why this matters for enterprises: If RF=3 and one broker fails, two copies remain available — no data loss, no service interruption. If two brokers fail simultaneously, one copy remains but write availability may be affected depending on min.insync.replicas configuration. For banking, RF=3 with min.insync.replicas=2 ensures that a write is acknowledged only when at least two replicas have it — providing durability guarantees that meet regulatory requirements.

ZooKeeper vs KRaft: the metadata evolution

Historically, Kafka depended on Apache ZooKeeper for cluster metadata management (broker registration, topic configuration, partition leadership). KRaft (Kafka Raft) replaces ZooKeeper with a built-in Raft consensus protocol, simplifying deployment and eliminating a separate distributed system dependency. For new enterprise deployments, KRaft is the recommended path — it reduces operational complexity and improves startup time. For existing ZooKeeper-based clusters, plan a migration during your next major version upgrade.

Kafka Connect for enterprise integration

Kafka Connect provides a framework for moving data between Kafka and external systems without writing custom code. Source connectors ingest data into Kafka (from databases via CDC, file systems, APIs). Sink connectors push data from Kafka to external systems (data warehouses, search indexes, notification services). For enterprises, Connect eliminates hundreds of point-to-point integrations and centralizes data movement through the Kafka backbone — making data flows visible, monitorable, and governable. modeling integration architecture with ArchiMate

If you'd like hands-on training tailored to your team (Sparx Enterprise Architect, ArchiMate, TOGAF, BPMN, SysML, Apache Kafka, or the Archi tool), you can reach us via our contact page.

Frequently Asked Questions

What is enterprise architecture?

Enterprise architecture is a discipline that aligns an organisation's strategy, business operations, information systems, and technology infrastructure. It provides a structured framework for understanding how an enterprise works today, where it needs to go, and how to manage the transition.

How is ArchiMate used in enterprise architecture practice?

ArchiMate is used as the standard modeling language in enterprise architecture practice. It enables architects to create consistent, layered models covering business capabilities, application services, data flows, and technology infrastructure — all traceable from strategic goals to implementation.

What tools are used for enterprise architecture modeling?

Common enterprise architecture modeling tools include Sparx Enterprise Architect (Sparx EA), Archi, BiZZdesign Enterprise Studio, LeanIX, and Orbus iServer. Sparx EA is widely used for its ArchiMate, UML, BPMN and SysML support combined with powerful automation and scripting capabilities.