In this new article, we will practically see fault tolerance and resilience. So let’s go to see an example of Circuit Breaker with Resilience4j in Spring Boot and and Time Limiter.

In a microservices architecture, it is vital to use the Circuit Breaker pattern to achieve a much more robust and fault-tolerant architecture.

What is the Circuit Breaker Pattern?

In our microservices architecture, we will make calls to other services frequently synchronously. It is for this reason that these calls may result in the circumstance that the service being called may not be accessible, or the latency of the call may be too high. In these cases, we are using valuable resources or threads, and on the one hand, we are leaving a call blocked and, on the other hand, we may cause a cascade failure.

An implementation of Circuit Breaker will help us in these types of scenarios, where the call to a third party has been blocked, avoiding repeated calls to that service and preventing that expenditure of resources.

The above diagram graphically represents a Circuit Breaker as a state machine (Open, Closed, Half_Open). When the system tries to communicate with another service that is not responding, these errors are tracked. When these errors in communication exceed a previously established threshold, we are in the OPEN state. After a waiting period, the state passes to HALF-OPEN, and the system tries to reconnect. If communication problems persist, it returns to the OPEN state, and if everything works correctly, it is in the CLOSED state.

Hands-on with Resilience4j and Spring Boot

Next, we will see an example of Circuit Breaker with Resilience4j in Spring Boot.

What is Resilience4J?

Resilience4j is a library that allows us to make our applications resilient. It is oriented towards functional programming, easy to use, and lightweight since it has hardly any dependencies.

Resilience4j Dependencies in Spring Boot

We will start creating our Spring Boot application, for which we will go to the Spring Initializr page. We will tell it the libraries we want for our project, including resilience4j:

		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-circuitbreaker-reactor-resilience4j</artifactId>
		</dependency>

Resilience4J Properties Configuration

In our resilience4j configuration, we can add different configurations for different services, as well as configurations for Circuit Breaker and Time Limiter.

In our example, we will add the following configuration, in which we establish a shared configuration for the example-resilience instance, so we overwrite the shared one:

resilience4j:
  circuitbreaker:
    configs:
      shared:
        register-health-indicator: true
        sliding-window-type: count_based
        sliding-window-size: 5
        failure-rate-threshold: 40
        slow-call-rate-threshold: 40
        permitted-number-of-calls-in-half-open-state: 1
        max-wait-duration-in-half-open-state: 10s
        wait-duration-in-open-state: 10s
        slow-call-duration-threshold: 2s
        writable-stack-trace-enabled: true
        automatic-transition-from-open-to-half-open-enabled: true
    instances:
      example:
        base-config: shared
  timelimiter:
    configs:
      shared:
      timeout-duration: 2s
      cancel-running-future: true
    instances:
      example:
        base-config: shared

Circuit Breaker Listener Creation with Resilience4J

In the next step of our configuration, we are going to add the configuration to register the events produced by our circuit breaker. To do this, we are going to use the class RegistryEventConsumer and define a @Bean of type RegistryEventConsumer<CircuitBreaker>.

  @Bean
  public RegistryEventConsumer<CircuitBreaker> circuitBreakerEventConsumer() {
    return new RegistryEventConsumer<CircuitBreaker>() {

      @Override
      public void onEntryAddedEvent(EntryAddedEvent<CircuitBreaker> entryAddedEvent) {
        entryAddedEvent.getAddedEntry().getEventPublisher()
            .onFailureRateExceeded(event -> log.error("circuit breaker {} failure rate {} on {}",
                event.getCircuitBreakerName(), event.getFailureRate(), event.getCreationTime())
            )
            .onSlowCallRateExceeded(event -> log.error("circuit breaker {} slow call rate {} on {}",
                event.getCircuitBreakerName(), event.getSlowCallRate(), event.getCreationTime())
            )
            .onCallNotPermitted(event -> log.error("circuit breaker {} call not permitted {}",
                event.getCircuitBreakerName(), event.getCreationTime())
            )
            .onError(event -> log.error("circuit breaker {} error with duration {}s",
                event.getCircuitBreakerName(), event.getElapsedDuration().getSeconds())
            )
            .onStateTransition(
                event -> log.warn("circuit breaker {} state transition from {} to {} on {}",
                    event.getCircuitBreakerName(), event.getStateTransition().getFromState(),
                    event.getStateTransition().getToState(), event.getCreationTime())
            );
      }

      @Override
      public void onEntryRemovedEvent(EntryRemovedEvent<CircuitBreaker> entryRemoveEvent) {
        entryRemoveEvent.getRemovedEntry().getEventPublisher()
            .onFailureRateExceeded(event -> log.debug("Circuit breaker event removed {}",
                event.getCircuitBreakerName()));
      }

      @Override
      public void onEntryReplacedEvent(EntryReplacedEvent<CircuitBreaker> entryReplacedEvent) {
        entryReplacedEvent.getNewEntry().getEventPublisher()
            .onFailureRateExceeded(event -> log.debug("Circuit breaker event replaced {}",
                event.getCircuitBreakerName()));
      }
    };
  }

In this method, we can register and analyze the different types of events that will occur with our circuit breaker. Among the different events that we can register are those related to adding, replacing, and removing, for all these events we can take some action.

Creating a Time Limiter Listener with Resilience4J

We added a listener for the circuit breaker, and now we will establish a time limiter, which is a time threshold that our connections must meet.

Next, we add the definition of a @Bean in which we establish different listeners for our TimeLimiter, for this, we create a @Bean of type RegistryEventConsumer<TimeLimiter>.

  @Bean
  public RegistryEventConsumer<TimeLimiter> timeLimiterEventConsumer() {
    return new RegistryEventConsumer<TimeLimiter>() {
      @Override
      public void onEntryAddedEvent(EntryAddedEvent<TimeLimiter> entryAddedEvent) {
        entryAddedEvent.getAddedEntry().getEventPublisher()
            .onTimeout(event -> log.error("time limiter {} timeout {} on {}",
                event.getTimeLimiterName(), event.getEventType(), event.getCreationTime())
            );
      }

      @Override
      public void onEntryRemovedEvent(EntryRemovedEvent<TimeLimiter> entryRemoveEvent) {
        entryRemoveEvent.getRemovedEntry().getEventPublisher()
            .onTimeout(event -> log.error("time limiter removed {}",
                event.getTimeLimiterName())
            );
      }

      @Override
      public void onEntryReplacedEvent(EntryReplacedEvent<TimeLimiter> entryReplacedEvent) {
        entryReplacedEvent.getNewEntry().getEventPublisher()
            .onTimeout(event -> log.error("time limiter replaced {} ",
                event.getTimeLimiterName())
            );
      }
    };
  }

In the previous method, we have defined a listener for events of type add, remove, and replace. Whenever one of these three events occurs, the actions defined in this method will be collected.

Creating a CircuitBreaker in Resilience4J

Next, we are going to create a controller in which we will define a circuit breaker, that is, a circuit will open every time there is a communication problem with a pre-established threshold in the configuration.

This way we can see the logs of our previously defined events:

  @GetMapping(value = "/timeDelay/{delay}", produces = MediaType.APPLICATION_JSON_VALUE)
  @CircuitBreaker(name = RESILIENCE4J_INSTANCE_NAME, fallbackMethod = FALLBACK_METHOD)
  public Mono<Response<Boolean>> timeDelay(@PathVariable int delay) {
    return Mono.just(toOkResponse())
        .delayElement(Duration.ofSeconds(delay));
  }

We are going to force the opening of our circuit breaker, for which we have a configured threshold of 50% and slow calls of 2 seconds.

resilience4j.circuitbreaker.configs.shared.failure-rate-threshold=50
resilience4j.circuitbreaker.configs.shared.slow-call-rate-threshold=50
resilience4j.circuitbreaker.configs.shared.slow-call-duration-threshold=2s

Next, we are going to make several requests to the following endpoint http://localhost:8080/api/timeDelay/10. Once the threshold exceeds 50% with calls that take more than 2 seconds, our circuit breaker will open and we will see something like this:

2021-10-23 17:07:19.176 ERROR 16008 --- [     parallel-6] c.n.c.listener.Resilience4jListener      : circuit breaker example slow call rate 100.0 on 2021-10-23T17:07:19.176497500+02:00[Europe/Madrid]
2021-10-23 17:07:19.183  WARN 16008 --- [     parallel-6] c.n.c.listener.Resilience4jListener      : circuit breaker example state transition from CLOSED to OPEN on 2021-10-23T17:07:19.183016+02:00[Europe/Madrid]

Once the pre-established waiting time of 10 seconds has elapsed, the state of the CircuitBreaker changes from OPEN to HALF_OPEN. At this point, it allows a configurable number of calls to see if the backend is available or not. The state will revert back to CLOSED when the number of failures or slow calls falls below the established threshold.

Creating a TimeLimiter in Resilience4J

Next, we are going to use the timelimiter with Resilience4j, for which we are going to create a lock above the pre-established threshold, thus achieving a timeout and our event will jump.

  @GetMapping(
      value = "/timeout/{timeout}",
      produces = MediaType.APPLICATION_JSON_VALUE
  )
  @TimeLimiter(name = RESILIENCE4J_INSTANCE_NAME, fallbackMethod = FALLBACK_METHOD)
  public Mono<Response<Boolean>> timeout(@PathVariable int timeout) {
    return Mono.just(toOkResponse())
        .delayElement(Duration.ofSeconds(timeout));
  }

Execute the following URL with our example executed to check the timeout, http://localhost:8080/api/timeout/5

When we run the previous URL, we will get the following trace on the console:

2021-10-23 17:03:30.481 ERROR 16008 --- [     parallel-1] c.n.c.listener.Resilience4jListener      : time limiter example timeout TIMEOUT on 2021-10-23T17:03:30.480110200+02:00[Europe/Madrid]
2021-10-23 17:03:30.484  WARN 16008 --- [     parallel-1] c.n.c.web.Resilience4jController         : fallback executed

We have set a 2-second time limit, so when making the request with 5 seconds, our time limiter kicks in.

Conclusion

When creating a microservices architecture, it’s crucial to use both a circuit breaker and a timeout to prevent leaving connections and threads busy. In this example of Circuit Breaker with Resilience4j in Spring Boot, we have seen how to apply both a time limit for our connection and use a circuit breaker with a simple example.

If you want to take a look at the complete example, you can find it on our Github.