How does Kafka address the issue of duplicate consumption?

Kafka utilizes offsets to address the issue of duplicate consumption.

The offset is a persistent identifier used to mark the position of a consumer in a specific partition where messages have been consumed. Each partition has its own offset.

After a consumer successfully processes a message, it submits the offset to Kafka. Kafka then saves the submitted offset in an internal topic. This allows the consumer to determine where to resume consumption from in case of a restart by reading the saved offset.

By using offsets, Kafka ensures that consumers do not consume messages repeatedly. When a consumer rejoins the consumer group, it will continue consuming from the latest committed offset.

In addition, Kafka also offers the following two mechanisms for handling duplicate consumption:

1. Offset committing automatically: Consumers can be configured to automatically commit offsets. In this case, Kafka will automatically commit offsets periodically. However, if a consumer crashes before committing offsets, it may result in duplicate consumption.
2. Manually submitting offsets: Consumers have the ability to manually control when to submit offsets. In this scenario, consumers can decide when to submit offsets based on their own logic. This allows for more precise control over the timing of submissions and avoids issues with duplicate consumption.