Running Kafka Locally

KRaft (Kafka Raft) removes that external dependency by having a subset of the Kafka brokers themselves run the Raft consensus protocol to agree on cluster metadata. Brokers can be configured with a controller role, a broker role, or both — in a local single-node setup it's common to run one process with both roles combined.

services:
  kafka-broker:
    image: apache/kafka:latest
    container_name: kafka-broker
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_LISTENERS: 'PLAINTEXT://:9092,CONTROLLER://:9093'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://localhost:9092'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-broker:9093'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

services:
  kafka-broker:
    image: apache/kafka:latest
    container_name: kafka-broker
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_LISTENERS: 'PLAINTEXT://:9092,CONTROLLER://:9093'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://localhost:9092'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-broker:9093'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

# create a topic with 3 partitions, replication factor 1 (single broker)
kafka-topics.sh --create --topic library-events --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# produce a message with no key
kafka-console-producer.sh --topic library-events --bootstrap-server localhost:9092

# produce with a key (format: key:value)
kafka-console-producer.sh --topic library-events --bootstrap-server localhost:9092 \
  --property "parse.key=true" --property "key.separator=:"

# consume from the beginning of the log
kafka-console-consumer.sh --topic library-events --from-beginning --bootstrap-server localhost:9092

# consume showing keys and partitions — makes key→partition routing visible
kafka-console-consumer.sh --topic library-events --from-beginning --bootstrap-server localhost:9092 \
  --property print.key=true --property print.partition=true

Run the keyed producer with the same key several times, then consume with print.partition=true — you'll see every record with that key lands on the same partition number, which is the most convincing way to internalize §01.4's partitioning rule.

# create a topic with 3 partitions, replication factor 1 (single broker)
kafka-topics.sh --create --topic library-events --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

# produce a message with no key
kafka-console-producer.sh --topic library-events --bootstrap-server localhost:9092

# produce with a key (format: key:value)
kafka-console-producer.sh --topic library-events --bootstrap-server localhost:9092 \
  --property "parse.key=true" --property "key.separator=:"

# consume from the beginning of the log
kafka-console-consumer.sh --topic library-events --from-beginning --bootstrap-server localhost:9092

# consume showing keys and partitions — makes key→partition routing visible
kafka-console-consumer.sh --topic library-events --from-beginning --bootstrap-server localhost:9092 \
  --property print.key=true --property print.partition=true

Everything so far ran with replication-factor 1 — one broker, zero fault tolerance. Production Kafka runs with 3+ brokers and a replication factor of 3, and the concepts below only become real once you run that shape locally and kill a broker mid-flight.

Replication

Each partition's records are copied to replication.factor brokers. One is the leader — it alone serves all produce/fetch requests for that partition. The others are followers, which continuously fetch from the leader to stay caught up, purely for durability; they don't serve client traffic directly (in the default configuration).

ISR & min.insync.replicas

A replica is in the ISR (in-sync replica set) only if it has fetched up to (near) the leader's latest offset within a configured lag threshold. A slow or disconnected follower falls out of the ISR. min.insync.replicas (a topic/broker setting, exercised fully in Module 13) sets a floor on how many ISR members must acknowledge a write before it's considered durable when the producer uses acks=all — this is the setting that actually enforces "durable," not acks alone.

Leader election

When a partition's leader broker fails, the controller picks a new leader from the remaining ISR members (never from an out-of-sync replica, by default — that would silently lose acknowledged data). This is why ISR membership matters so much: it defines the pool of "safe" leader candidates at any moment.

Leader election picks the new leader only from brokers that were in the ISR — never from a replica that had fallen behind, since that could silently drop already-acknowledged records.