Kafka Fundamentals
Before you write a line of Spring Boot code, you need the vocabulary and mental model Kafka is built on. Get this module wrong and every later module will feel like memorized incantations instead of engineering decisions.
Learning objectives
- Beginner: Create a single-partition topic for a low-volume audit-log use case where strict global ordering across every event matters more than throughput.
- Intermediate: Key library-events by libraryEventId so that a NEW event and its later UPDATE for the same book are guaranteed to be processed in order by the consumer.
- Advanced: Design partition count around your consumer group's expected max scale-out (e.g. 12 partitions to allow scaling to 12 consumer instances under load), while keying by a business ID whose cardinality is high enough to spread load evenly and avoid a "hot partition."
◆ The problem
Imagine an e-commerce platform: placing an order needs to update inventory, charge payment, notify shipping, update analytics, and email the customer. Do it all synchronously inside one request and the order endpoint is now only as fast and as reliable as its slowest, flakiest downstream dependency. Add a new consumer of "an order happened" (say, a fraud-detection service) and you have to modify the order service again. Every producer ends up coupled to every consumer.
Apache Kafka is a distributed event streaming platform: producers publish records of things that happened to named streams called topics, and any number of independent consumers read those streams at their own pace, without the producer knowing or caring who's listening. It solves the coupling problem above by turning "when X happens, tell every interested service" into "when X happens, write one record; anyone interested reads it."
The core idea is deceptively simple: Kafka is fundamentally a distributed, append-only log. Producers can only append new records to the end of a topic. Consumers read forward through the log at their own pace, tracked by a position called an offset. Nothing is removed when it's read — the same record can be read by ten different consumer applications, or replayed from the beginning by a new one that didn't exist yet when the record was written.
| Property | Traditional queue (e.g. classic JMS/RabbitMQ queue) | Kafka |
|---|---|---|
| Once a message is read | Typically removed from the queue | Stays in the log until retention expires — replayable |
| Multiple consumers of the same message | Usually one consumer gets it (competing consumers) | Every consumer group gets its own full copy of the stream |
| Ordering guarantee | Often global FIFO within the queue | Guaranteed only within a partition, not across a topic |
| Scaling reads | Add more consumers competing for the same queue | Add more partitions + consumers in a group, each owning a slice |
This is why Kafka is described as "event-driven architecture" infrastructure rather than just "another message queue": the log is a durable source of truth about everything that happened, not a transient inbox that gets emptied.
These terms recur on every page of this site. Get comfortable with them now — most of Kafka's complexity is just combinations of these primitives.
| Term | What it is |
|---|---|
| Broker | One Kafka server. Stores partitions, serves reads/writes. |
| Topic | A named logical stream of records (e.g. library-events). |
| Partition | An ordered subdivision of a topic's log; the unit of parallelism. |
| Producer | A client that writes (appends) records to a topic. |
| Consumer | A client that reads records from a topic, tracking its position via offsets. |
| Offset | A sequential integer identifying a record's position within its partition. |
| Consumer group | A named set of consumers sharing the work of reading a topic. |
| Replica | A copy of a partition kept on another broker for fault tolerance. |
| Leader / Follower | Of a partition's replicas, one leader serves all reads/writes; followers replicate it. |
| ISR | The subset of replicas fully caught up with the leader right now. |
A 3-partition, replication-factor-3 topic spread across 3 brokers. Each partition has exactly one leader broker (handles all traffic) and the rest as followers.
__ committed record __ next write position (offset) __ broker
◆ Under the hood
A "topic" isn't a file or a single physical thing — it's a naming convention over a set of partition directories. On disk, each broker literally has a folder per partition it hosts (e.g. library-events-0), containing segment files that store the actual records. When you "create a topic with 3 partitions," you're asking the controller to decide which brokers will host each partition's replicas and to create those directories.
Kafka exposes several distinct client APIs — knowing which one you're using (and which this site focuses on) avoids a lot of confusion when reading Kafka documentation elsewhere.
| API | Purpose | Covered on this site? |
|---|---|---|
| Producer API | Write records to topics | Yes — extensively (Modules 03, 06, 13) |
| Consumer API | Read records from topics | Yes — extensively (Modules 03, 09, 14, 15) |
| Streams API | Build stream-processing topologies (transform/aggregate topics into other topics) | Not covered — beyond this course's scope |
| Connect API | Framework for connecting Kafka to external systems (DBs, S3, etc.) via connectors | Not covered |
| Admin API | Programmatic topic/ACL/config management | Touched on via KafkaAdmin in Module 06 |
◆ The problem
If a topic were a single, unpartitioned log, only one consumer could productively read it at a time (more readers just re-read the same stream, they don't share the work), and all writes for that topic would be bottlenecked on a single broker.
A topic is split into one or more partitions, and this split is Kafka's entire answer to scalability. Each partition is independently ordered and independently assigned to a broker as its leader. That gives you two kinds of parallelism at once: producers can write to different partitions in parallel, and —critically for scaling consumption— a consumer group can assign one partition per consumer, so a topic with 6 partitions can be actively processed by up to 6 consumers in the same group simultaneously.
How a record gets assigned to a partition
When you produce a record, you can supply a key. Kafka's default partitioner hashes that key and maps it deterministically to one of the topic's partitions — the same key always lands on the same partition. If you don't supply a key, records are spread across partitions round-robin (or in "sticky" batches in newer clients, for batching efficiency).
◆ Under the hood
This is why keys matter for ordering: Kafka only guarantees order within a partition, never across an entire topic. If you need "all events for library card #482 processed in the order they happened," you must key your records by that card ID — that pins every event for that key to the same partition, which is strictly ordered. Two different keys can land on two different partitions and be processed in any relative order to each other; that's expected, not a bug.
// No key: round-robin across partitions — max throughput, no ordering guarantee kafkaTemplate.send("library-events", libraryEventJson); // Keyed: every event for this libraryEventId lands on the same partition, // so updates for the same book are always processed in order kafkaTemplate.send("library-events", libraryEvent.getLibraryEventId().toString(), libraryEventJson);
▲ Pitfall
Adding partitions to an existing topic changes key-to-partition mapping for future records. The hash is computed against the current partition count, so growing a topic from 6 to 12 partitions can send a key that used to land on partition 2 to a different partition afterward, silently breaking your per-key ordering assumption for anything produced after the resize. Plan partition counts up front; treat changing them later as a breaking change for ordering-sensitive topics.
✓ Quick recap
Does Kafka guarantee ordering across an entire topic? No — only within a single partition. Across partitions, relative order between records is undefined. What determines which partition a keyed record goes to? A hash of the key, modulo the current partition count (via the default partitioner). Why would you deliberately use more than one partition? To parallelize both writes and, more importantly, reads — a consumer group can have as many active consumers as there are partitions. What happens if you add partitions to a topic that's in production? New records may hash to different partitions than before, breaking per-key ordering guarantees for anything produced going forward.
💻 Code example
// No key: round-robin across partitions — max throughput, no ordering guarantee kafkaTemplate.send("library-events", libraryEventJson); // Keyed: every event for this libraryEventId lands on the same partition, // so updates for the same book are always processed in order kafkaTemplate.send("library-events", libraryEvent.getLibraryEventId().toString(), libraryEventJson);
Want a visual for this concept?
Generate a diagram tailored to “Kafka Fundamentals” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.
Sign in to generate a visual →