In this post, we’re going to talk about how to use Kafka in a Microservices Architecture, not Franz Kafka the Bohemian writer, but the Apache tool in Microservices, and how useful and important it can be for building a good Microservices architecture.

Event-driven programming is a widely implemented pattern in Microservices, as discussed in a previous post. This is an event-based paradigm, meaning there is a loop that listens for any action and performs processing based on that action. It can be summarized as a system sending events or messages to another system within the same domain (business context).

The goal is full asynchrony in this architecture, eliminating synchronous HTTP calls with waiting for responses.

How does Kafka work?

Kafka is a durable messaging system based on publish-subscribe that exchanges data between processes, applications, and servers. It is a distributed and horizontally scalable system.

Its operation is similar to any messaging system. On one hand, we have Producers that send a message to a Kafka node (broker), and on the other hand, that message is processed by another application which is the consumer. These messages are stored in a topic to which the consumer is subscribed and can listen to those messages.

One of the greatest advantages Kafka provides is its speed, but it does not validate messages. So, if the Producer sends the message in one format and the consumer expects a different format, there will be compatibility issues.

To avoid compatibility problems between the message sent by the producer and received by the consumer, we can use the Schema Registry. This is a repository of schemas where the message format to be sent is stored. This way, the message can be validated, and an error can be thrown if the format is not as expected.

Event-driven with Kafka

For the use of the Event-driven pattern, a consumer and a producer are required. They communicate through messages, making use of Kafka, for example.

In this Architecture Pattern, a message will be sent when a change of state occurs. The consumer subscribed to that topic will receive and process the message.

Let’s explain this point with a simple example. Create a banking system with two Microservices: one for user balance and another for credit card operations.

Every time the user makes a payment with the card, the microservice that manages the card will produce a message that asynchronously arrives at a Kafka topic. Then the consumer microservice that handles the balance will be responsible for consuming that message and deducting the balance.

This is a very basic and simple example of using Kafka, as there are more parameters to consider, such as whether there is a balance or handling consecutive messages that require compensation due to payment failure, etc. However, these topics will be covered in future posts, with the Saga pattern.

Conclusion

In this post, we have seen how we can use Kafka in a Microservices architecture using the Event-driven pattern.

For an example using Quarkus and implementing the Saga pattern, check out the link provided.