Apache Kafka Event-Driven Architecture in Production
Production guide to Apache Kafka partitions, consumer group rebalancing, dead-letter queues (DLQ), idempotent producers, and exactly-once semantics.
Architectural Summary (AEO)
Apache Kafka achieves horizontal throughput by splitting topics into ordered, append-only partitions distributed across brokers. Production reliability requires keying messages for per-entity ordering, enabling idempotent producers (acks=all), isolating poison-pill messages into Dead-Letter Queues (DLQ), and using CooperativeStickyAssignor to prevent stop-the-world consumer group rebalances.
Reference System Flow
Producer (acks=all, enable.idempotence=true)
│ key = tenant_id (Murmur2 hash -> partition)
▼
┌──────────────────────────────────────────────────────────┐
│ Kafka Topic: orders.events (6 Partitions, RF=3, minISR=2)│
│ P0 [0,1,2] P1 [0,1] P2 [0,1,2,3] ... P5 [0,1] │
└──────────────┬───────────────────────────────────────────┘
│ Consumer Group (max 6 active consumers)
▼
┌──────────────────────────────────────────────────────────┐
│ Worker Pool (Manual Offset Commit after Idempotent Write)│
│ ├─ Transient Error ──► Retry Topic (exponential backoff)│
│ └─ Poison Pill ──► orders.events.dlq (alert + replay)│
└──────────────────────────────────────────────────────────┘1. Partitioning, Ordering Guarantees & Replication
Kafka guarantees strict FIFO ordering only within a single partition, never across an entire topic. Choosing the right partition key (such as `tenant_id` or `order_id`) keeps related state transitions ordered without creating hot partitions. In production, configure Replication Factor `RF=3` and `min.insync.replicas=2` paired with `acks=all` so a broker crash never loses acknowledged writes.
2. Consumer Group Rebalancing Without Latency Spikes
Eager rebalancing revokes all partitions across every consumer whenever a pod scales or restarts, causing noticeable lag spikes. Use `CooperativeStickyAssignor` with static group membership (`group.instance.id`) so rolling Kubernetes deployments only migrate affected partitions incrementally.
3. Dead-Letter Queues (DLQ) & Poison-Pill Isolation
Never block a partition indefinitely on a malformed payload. Route transient failures (HTTP 503, DB deadlock) to delayed retry topics (`retry-1m`, `retry-10m`), and after N attempts publish the raw record plus exception stack and original offset headers to a `.dlq` topic for inspection and replay.
4. Idempotent Consumers & Outbox Pattern
Even with Kafka's `enable.idempotence=true` and transactional `read_committed` isolation, side effects against external databases require an idempotent consumer (`INSERT INTO processed_events(event_id) ON CONFLICT DO NOTHING`) or the Transactional Outbox pattern via Debezium CDC.
KafkaJS Producer & Idempotent Consumer with DLQ Routing
const producer = kafka.producer({
idempotent: true,
maxInFlightRequests: 5,
retry: { retries: 8 },
});
await producer.send({
topic: "orders.events",
acks: -1, // acks=all
messages: [{ key: order.tenantId, value: JSON.stringify(order) }],
});
// Consumer with manual offset commit & DLQ fallback
await consumer.run({
autoCommit: false,
eachMessage: async ({ topic, partition, message }) => {
const eventId = message.headers?.["x-event-id"]?.toString();
try {
await db.transaction(async (tx) => {
const inserted = await tx.query(
"INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING event_id",
[eventId]
);
if (inserted.rowCount === 0) return; // Already processed
await applyOrderState(tx, JSON.parse(message.value.toString()));
});
} catch (err) {
await producer.send({
topic: `${topic}.dlq`,
messages: [{ key: message.key, value: message.value, headers: { "x-error": err.message } }],
});
}
await consumer.commitOffsets([{ topic, partition, offset: String(Number(message.offset) + 1) }]);
},
});Architectural Trade-Offs
| Delivery Semantic | Configuration | Latency Impact | Best Fit |
|---|---|---|---|
| At-Most-Once | acks=0, autoCommit=true | Lowest (< 3ms) | Non-critical clickstream / metrics |
| At-Least-Once + Idempotent DB | acks=all, manual commit, dedup table | Low (8–15ms) | 95% of billing, order & webhook pipelines |
| Exactly-Once (EOS) | transactional.id + read_committed | Moderate (+20–35ms) | Kafka-to-Kafka stream transformations |
Production Benchmarks & Field Observations
- 3-broker Kafka cluster (c6i.xlarge, NVMe): ~180,000 msgs/sec sustained at 1KB payload with acks=all and LZ4 compression.
- CooperativeStickyAssignor reduces rolling-deploy rebalance pause from ~12s (eager) to < 400ms.
Related Commercial Services
Engineering FAQs
How many partitions should a Kafka topic have?
Size partitions to `max(target_throughput / single_partition_throughput, max_expected_consumer_pods)`. For most mid-scale services, 6 to 24 partitions per topic provides ample parallelism without excessive file handles.
When should we choose Kafka over RabbitMQ or SQS?
Choose Kafka when you need high-throughput ordered event logs, multiple independent consumer groups reading the same stream, or event replay. Choose SQS/RabbitMQ for simple task queues with per-message routing.