In this new article about Exactly Once with Kafka in Spring Boot, we will see how to achieve exactly once message delivery between producer and consumer using the Kafka Transactional API.

In a previous post we explored Kafka configuration with Spring Boot.

Types of Message Delivery in Kafka?

Kafka supports different types of message delivery:

1. At most once-messages: The producer does not retry sending the message when it encounters an error in the ACK response or exceeds the ACK timeout. Method risks message loss, as it may not deliver to the topic, preventing consumer consumption.
2. At least once-messages: Producer retries on errors or ACK timeout, assuming message not written to Kafka topic, ensuring at least once delivery.
3. Exactly once: In this case, which is the focus of this article, if the producer retries sending a message, it delivers the message one and only one time time to the final consumer.

When implementing Exactly Once in our application, we ensure that the application delivers each message exactly once, regardless of how many times the delivery is retried.

Configuring Kafka in Spring Boot for Exactly Once Message Delivery

Add Maven Dependencies

Start with Spring Initializr, add Kafka dependencies for easy Kafka integration in Spring Boot applications:

<dependency>
  <groupId>org.springframework.kafk<groupId>
  <artifactId>spring-kafka</artifactId>
</dependency>

Add the main dependency for the project:

<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka-test</artifactId>
  <scope>test</scope>
</dependency>

Enabling Transactional Behavior in the Producer

To ensure that we deliver each exactly once, we must configure some properties in our Spring Boot application.properties or application.yaml file:

spring.kafka.producer.enable-idempotence: true
spring.kafka.producer.transactional-id: prod-A
spring.kafka.producer.transaction-id-prefix: transaction

Idempotence enabled with transactional-id (prod-A) avoids producer duplicates, ensuring message idempotence in Kafka.. Each transactional-id should be unique for each producer.

Next, we need to send the messages within a transaction using the KafkaTemplate and the executeInTransaction method:

kafkaTemplate.executeInTransaction(kt -> {
  ListenableFuture<SendResult<String, String>> result = kt.send(event);
  result.addCallback(
      successResult -> log.debug("Message {} sent ok with result: {}", getEventAsString(event), successResult),
      exception -> {
        log.error("Non retriable exception found in send operation: {}", exception.getClass().getName());
      });
  return result;
});

By using the above code snippet, the broker recognizes that we are sending each message within a transaction identified by its transactional-id and a sequence number or timestamp.

Enabling Transactional Behavior in the Consumer

The consumer consumes messages from a broker in the order they arrive. However, to ensure transactional behavior, we need to inform the broker to wait for transactional reads. We can achieve this in the application.properties or application.yaml file as follows:

spring.kafka.consumer.properties.isolation.level: read_committed

The default isolation level is read_uncommitted, and it reads all messages from the offset in order without waiting for the transaction to commit.

Conclusion

In this article, we have seen how to apply Kafka with Spring Boot using exactly once message delivery. This ensures transactionality and guarantees that each message is delivered exactly once to the consumer.