Kafka is often introduced as if it were an architectural destination. It is a powerful platform, but it solves a narrower problem: durable, ordered streams of records that multiple consumers can read, process, and replay according to their own progress.
That capability changes what a system can do. It does not automatically solve workflow, consistency, ownership, or service boundaries.
What Kafka gives you
Kafka provides a durable log organized into topics and partitions. Producers append records; consumers read them and track their position. Within a partition, records have an order. Consumer groups allow several instances to share the work for a topic, while separate groups can independently consume the same stream.
This makes Kafka useful when a team needs to:
- ›decouple producers from several downstream consumers;
- ›absorb and process traffic asynchronously;
- ›retain an event history for a configured period; or
- ›rebuild a derived view by replaying records from a known position.
The useful abstraction is not “a faster queue.” It is a durable stream with explicit partitioning, consumer progress, retention, and delivery semantics.
What it does not give you for free
Kafka does not decide who owns a business fact. A topic named orders can still
be an integration mess if several teams publish incompatible meanings into it.
It does not turn a collection of services into a coherent workflow, and it does
not remove the need to decide how failures and compensating actions work.
Kafka also does not make distributed transactions disappear. A producer may successfully publish a record while a database transaction fails, or a consumer may process a record and fail before recording its progress. Exactly-once processing can be valuable in a bounded topology, but it is not a general promise that every external side effect happens once.
Begin with the event and its owner
Before creating a topic, write down the fact being represented and the system that owns it. A useful event describes something that happened, rather than an instruction disguised as a fact:
{
"eventType": "OrderPlaced",
"eventId": "01J...",
"occurredAt": "2026-10-02T18:30:00Z",
"orderId": "order-123",
"schemaVersion": 1
}
The example is intentionally incomplete. A real contract needs an agreed schema, compatibility policy, keying strategy, privacy review, and a decision about whether consumers may rely on each field. Do not publish a database row as an event merely because serializing it is convenient.
Partitioning is a domain decision
The record key influences partition placement and therefore the order available to a consumer for that key. Choose it from the ordering and scaling needs of the domain, not from whichever identifier happens to be present in the payload.
If all records use one key, the topic may have many partitions but only one hot partition. If the key is too broad, related records can arrive out of the order the consumer needs. If the key is too fine-grained, the consumer may have to reconstruct ordering across keys.
State the invariant explicitly: “events for one account are processed in order” is a design requirement; “the topic has twelve partitions” is only an implementation detail.
Retries and replays are part of the contract
A consumer should expect a record to be delivered again. Handlers should be idempotent where possible, and the system should distinguish transient failure, poison records, and permanent business rejection.
Bounded retries, backoff, and a deliberate dead-letter or quarantine policy are more useful than endlessly retrying a record in the main flow. Replaying a topic also requires an answer to a harder question: is the consumer rebuilding a derived view, or is it repeating an external side effect?
Those operations need different safeguards. A rebuild can often write to a new store and swap it into service. Repeating an email, payment, or partner API call needs an idempotency key and a business-specific reconciliation strategy.
When Kafka is the wrong tool
Use a simpler mechanism when the requirement is a single short-lived work item, a request/response interaction, or a transactional update inside one service. Kafka's retention, partitioning, consumer coordination, schema evolution, and operational monitoring are costs. They are justified by the problem, not by a desire to make an architecture look event-driven.
Likewise, do not use Kafka as a database, a workflow engine, or a replacement for an API that needs immediate confirmation. It can participate in each of those architectures, but it does not eliminate the other component's job.
The decision
Choose Kafka when durable streams, independent consumers, asynchronous work, or replayable history are central requirements. Define ownership, keys, schemas, failure handling, and side-effect safety before choosing clients and deployment topology.
Kafka is most valuable when its boundary is clear: it carries facts and work between systems while the surrounding architecture remains responsible for meaning, consistency, and business decisions.
Sources: Apache Kafka documentation, Kafka design documentation, Kafka consumer groups