Spring Boot - Start/Stop a Kafka Listener Dynamically

Last Updated : 23 Jan, 2025

In a Spring Boot application, Kafka Listeners start automatically once the application launches and they listen for messages from Kafka topics. But there are many scenarios where we might need to dynamically start or stop a Kafka listener based on certain conditions. This can be achieved using KafkaListenerEndpointRegistry, which allows programmatic control over the lifecycle of the Kafka Listener.

In this article, we will learn different ways to control Kafka listeners in the Spring Boot application to optimize message processing and resource management.

What is Kafka Listener?

A Kafka Listener is a Spring Bean that listens to a Kafka topic. It processes incoming messages and provides a convenient way to consume Kafka data asynchronously. Kafka Listeners are configured with the @KafkaListener annotation and are started automatically when the application launches.

Different Approaches to Start/Stop a Kafka Listener

There are different ways to control Kafka listeners dynamically:

Controlling Listeners Based on Message Processing
Using @KafkaListener with autoStartup Property
Manual Control with KafkaListenerEndpointRegistry

1. Controlling Listeners Based on Message Processing

This is a common use case for dynamic listener control is to start listeners only when needed to process Kafka messages. After processing, listeners are stopped to save resources. This is useful when handling messages in bursts, as it prevents listeners from running when there are no messages to process.

Example: In this approach, start the listener when a message is detected, and once the message processing is complete, stop the listener.

Java

@Component public class KafkaListenerControl {      @Autowired        // Injecting Kafka listener registry     private KafkaListenerEndpointRegistry registry;       // Start the listener if it is not already running     public void startListener(String listenerId) {                // Get the listener container by ID         MessageListenerContainer container =            registry.getListenerContainer(listenerId);                // Check if container exists and is not running         if (container != null && !container.isRunning()) {                         // Start the listener container             container.start();                        // Log listener start             logger.info("Listener {} started.", listenerId);          }     }      // Stop the listener if it is currently running     public void stopListener(String listenerId) {                // Get the listener container by ID         MessageListenerContainer container =            registry.getListenerContainer(listenerId);                // Check if container exists and is running         if (container != null && container.isRunning()) {                         // Stop the listener container             container.stop();                         // Log listener stop             logger.info("Listener {} stopped.", listenerId);          }     } }

This approach allows a good control over Kafka listeners and it enable us to start and stop them based on specific conditions or messages being processed.

2. Using @KafkaListener with autoStartup Property

This is one of the simplest ways to control when a Kafka listener starts is to use the autoStartup attribute of the @KafkaListener annotation. By setting autoStartup to false, the listener will not start automatically with the application. It can be started manually based on application logic.

@KafkaListener(id = "dynamicListener", topics = "test-topic", groupId = "test-group", autoStartup = "false")
public void processMessage(Message message) {
logger.info("Processing message: {}", message);
}

In this example, the listener does not start automatically upon application startup. Instead, it can be triggered to start manually using the KafkaListenerEndpointRegistry.

3. Manual Control with KafkaListenerEndpointRegistry

The KafkaListenerEndpointRegistry provides a programmatic interface to start, stop, or manage Kafka listeners manually. This is useful when the listener needs to start or stop based on certain events, such as a user registration or an API call.

@Autowired
KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;

To start Kafka Listener:

public boolean startListener(String listenerId) {
MessageListenerContainer listenerContainer = kafkaListenerEndpointRegistry.getListenerContainer(listenerId);
assert listenerContainer != null : "Listener not found!";
listenerContainer.start();
logger.info("{} Kafka Listener Started", listenerId);

To stop Kafka Listener:

public boolean stopListener(String listenerId) {
MessageListenerContainer listenerContainer = kafkaListenerEndpointRegistry.getListenerContainer(listenerId);
assert listenerContainer != null : "Listener not found!";
listenerContainer.stop();
logger.info("{} Kafka Listener Stopped.", listenerId);

We will be implementing the above syntactical approaches as an example below.

Implementing Start/Stop Kafka Listener Dynamically

We will implement a simple example to start or stop a specific Kafka listener.

Note: To know, integration of Apache Kafka with Spring Boot, refer this article: Spring Boot - Integration with Kafka

1. Define a Message Class

Java

import lombok.AllArgsConstructor; import lombok.Data; import lombok.NoArgsConstructor;  @Data @AllArgsConstructor @NoArgsConstructor public class Message {     private String message; }

2. Configure Kafka Consumer:

Java

@Configuration public class KafkaConsumerConfig {      private String kafkaUrl = "localhost:9092";      @Bean     public ConsumerFactory<String, Message>        messageConsumerFactory() {         JsonDeserializer<Message> deserializer =            new JsonDeserializer<>(Message.class, false);         deserializer.setRemoveTypeHeaders(false);         deserializer.addTrustedPackages("*");          Map<String, Object> config = new HashMap<>();         config.put           (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaUrl);         config.put           (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);         config.put           (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);         config.put           (ConsumerConfig.GROUP_ID_CONFIG, "group-1");         return new DefaultKafkaConsumerFactory<>           (config, new StringDeserializer(), deserializer);     }      @Bean     public ConcurrentKafkaListenerContainerFactory<String, Message>        messageListenerFactory() {         ConcurrentKafkaListenerContainerFactory<String, Message>            containerFactory = new ConcurrentKafkaListenerContainerFactory<>();         containerFactory.setConsumerFactory(messageConsumerFactory());         return containerFactory;     } }

3. Create a Kafka Listener:

Create using below necessary parameters:

id: The container's unique identifier for this listener. If none is specified, an auto-generated ID is used.
groupId: Override the group.id property for the consumer factory with this value for this listener only.
topics: The topics of this listener. The entries can be "topic names," "property placeholder keys," or "expressions." The topic name must be resolved from an expression. This uses group management, and Kafka will assign partitions to group members.
containerFactory: The beans name of the KafkaListenerContainerFactory that will be used to create the message listener container that will serve this endpoint.
autoStartup: Set to true or false to override the container factory's default setting. By default, the value is set to true, and because of this, it'll start consuming messages as soon as our application starts.

Java

import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.context.annotation.Configuration; import org.springframework.kafka.annotation.KafkaListener;  @Configuration public class KafkaMessageListener {      Logger logger = LoggerFactory.getLogger(KafkaMessageListener.class);      @KafkaListener(id = "id-1", groupId = "group-1", topics = "Message-topic",                     containerFactory = "messageListenerFactory", autoStartup = "false")     public void consumeMessage(Message message) {         logger.info("Message received : -> {}", message);     } }

4. Control Kafka Listener (Start/Stop):

The KafkaListenerEndpointRegistry class can be used to get a Kafka Listener Container by listenerId. Here we have used @KafkaListener annotation to stating the bean method as a listener for the Kafka Listener Container. The Kafka Listener can now be started or stopped using this container.

Java

import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.kafka.config.KafkaListenerEndpointRegistry; import org.springframework.kafka.listener.MessageListenerContainer; import org.springframework.stereotype.Component;  @Component public class KafkaListenerAutomation {      private final Logger logger =        LoggerFactory.getLogger(KafkaListenerAutomation.class);      @Autowired     KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;      public boolean startListener(String listenerId) {         MessageListenerContainer listenerContainer =            kafkaListenerEndpointRegistry.getListenerContainer(listenerId);         assert listenerContainer != null : "Listener not found!";         listenerContainer.start();         logger.info("{} Kafka Listener Started", listenerId);         return true;     }      public boolean stopListener(String listenerId) {         MessageListenerContainer listenerContainer            = kafkaListenerEndpointRegistry.getListenerContainer(listenerId);         assert listenerContainer != null : "Listener not found!";         listenerContainer.stop();         logger.info("{} Kafka Listener Stopped.", listenerId);         return true;     } }

5. API Endpoints to Control Kafka Listener:

Using API endpoints, we can start or stop a specific Kafka listener by providing the listenerId.

Java

import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RequestParam; import org.springframework.web.bind.annotation.RestController;  @RestController public class StartOrStopListenerController {      @Autowired     KafkaListenerAutomation kafkaListenerAutomation;      @GetMapping("/start")     public void start(@RequestParam("id") String listenerId) {         kafkaListenerAutomation.startListener(listenerId);     }      @GetMapping("/stop")     public void stop(@RequestParam("id") String listenerId) {         kafkaListenerAutomation.stopListener(listenerId);     } }

Output:

1. Kafka Listener started:

Access the following endpoint to start a listener:

GET /start?id=id-1

When the listener is started, you will see the below log message:

2. Kafka Listener received message:

When the listener processes a message, you should see a log message similar like below:

Message received: -> {message: "Your Message Content"}

3. Kafka Listener stopped:

Access the below endpoint to stop the listener:

GET /stop?id=id-1

When the listener is stopped, you will see the below log message:

Note: To know more how to run Apache Kafka, refer this article: How to Install and Run Apache Kafka on Windows?

Conclusion

In this article, we learned how to dynamically start and stop Kafka listeners in a Spring Boot application. Using KafkaListenerEndpointRegistry gives us programmatic control over the listener lifecycle. We also explored the autoStartup property of @KafkaListener for simpler control. This allows us to optimize Kafka message processing based on conditions.

Spring Boot - Start/Stop a Kafka Listener Dynamically

malavmevada

Improve

Article Tags :

Practice Tags :

Java

Spring Boot - Start/Stop a Kafka Listener Dynamically

What is Kafka Listener?

Different Approaches to Start/Stop a Kafka Listener

1. Controlling Listeners Based on Message Processing

2. Using @KafkaListener with autoStartup Property

3. Manual Control with KafkaListenerEndpointRegistry

Implementing Start/Stop Kafka Listener Dynamically

Conclusion

Similar Reads