Implementing Request Response in Java Apache Kafka

Last Updated : 16 Jul, 2024

Apache Kafka is a very powerful distributed event streaming platform that can be used for building real-time pipelines and streaming applications. It is highly scalable, fault-tolerant, and provides high throughput. One of the common patterns used in the Kafka application is the request-response pattern which is slightly unconventional as Kafka is designed for asynchronous messaging. We can implement the request-response communication between the services using Kafka.

In the request-response pattern, the Consumer sends the request message to the Producer, and the producer can process the request and send it back to the response message. In Kafka, It can achieved using separate topics for the requests and responses along with the correlation IDs to match the responses to corresponding requests.

Prerequisites:

Basic understanding of the Java and Apache Kafka.
JDK and IntelliJ Idea installed in your local system.
Maven for building dependency management.
Required the Kafka Client Library.

Implementing Request Response in Java Apache Kafka

Step 1: Create a Maven Project

Create a new Java maven project using IntelliJ Idea, after that add the following dependencies into the project dependency.

Note: To Install and setup the Apache Kafka refer this link

Required Dependencies:

            <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->             <dependency>                 <groupId>org.apache.kafka</groupId>                 <artifactId>kafka-clients</artifactId>                 <version>3.7.0</version>             </dependency>              <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-api -->             <dependency>                 <groupId>org.slf4j</groupId>                 <artifactId>slf4j-api</artifactId>                 <version>2.0.13</version>             </dependency>              <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->             <dependency>                 <groupId>org.slf4j</groupId>                 <artifactId>slf4j-simple</artifactId>                 <version>2.0.13</version>                 <scope>test</scope>             </dependency>

After creating the project, the folder structure in the IDE will look like the below image:

Step 2: Add the Dependencies into the pom.xml file

Open the pom.xml add the required dependencies into the project.

XML

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0"          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">     <modelVersion>4.0.0</modelVersion>      <groupId>com.gfg</groupId>     <artifactId>kafka-request-response</artifactId>     <version>1.0-SNAPSHOT</version>      <properties>         <maven.compiler.source>17</maven.compiler.source>         <maven.compiler.target>17</maven.compiler.target>         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>     </properties>     <dependencies>              <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->             <dependency>                 <groupId>org.apache.kafka</groupId>                 <artifactId>kafka-clients</artifactId>                 <version>3.7.0</version>             </dependency>              <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-api -->             <dependency>                 <groupId>org.slf4j</groupId>                 <artifactId>slf4j-api</artifactId>                 <version>2.0.13</version>             </dependency>              <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->             <dependency>                 <groupId>org.slf4j</groupId>                 <artifactId>slf4j-simple</artifactId>                 <version>2.0.13</version>                 <scope>test</scope>             </dependency>          </dependencies> </project>

Step 3: Create the KafkaProducerService Class

Java

package com.gfg;  import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.clients.producer.RecordMetadata; import org.apache.kafka.clients.producer.Callback; import org.apache.kafka.common.serialization.StringSerializer;  import java.util.Properties;  /**  * Service class for producing messages to Kafka topics.  */ public class KafkaProducerService {      private final KafkaProducer<String, String> producer;      /**      * Constructor to initialize Kafka producer with provided bootstrap servers.      *      * @param bootstrapServers The list of Kafka bootstrap servers.      */     public KafkaProducerService(String bootstrapServers) {         Properties props = new Properties();         props.put("bootstrap.servers", bootstrapServers);         props.put("key.serializer", StringSerializer.class.getName());         props.put("value.serializer", StringSerializer.class.getName());         this.producer = new KafkaProducer<>(props);     }      /**      * Sends a message to the specified Kafka topic.      *      * @param topic The topic to which the message will be sent.      * @param key   The key for the message.      * @param value The value of the message.      */     public void sendMessage(String topic, String key, String value) {         ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);         producer.send(record, new Callback() {             public void onCompletion(RecordMetadata metadata, Exception exception) {                 if (exception != null) {                     exception.printStackTrace();                 } else {                     System.out.println("Sent message to topic: " + metadata.topic() + " partition: " + metadata.partition() + " offset: " + metadata.offset());                     System.out.println("Received message: Hello, Kafka!");                 }             }         });     }      /**      * Closes the Kafka producer.      */     public void close() {         producer.close();     } }

The KafkaProducerService class manages a Kafka Producer instance configured with bootstrap servers and serializers for keys and values. It sends messages asynchronously to specified topics, handles delivery callbacks, and provides a method to close the producer cleanly.

Step 4: Create the KafkaConsumerService Class

Java

package com.gfg;  import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.ConsumerRecord; import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.common.serialization.StringDeserializer;  import java.time.Duration; import java.util.Collections; import java.util.Properties;  /**  * Service class to manage Kafka consumer instance.  */ public class KafkaConsumerService {      private final KafkaConsumer<String, String> consumer;      /**      * Constructor to initialize Kafka consumer with configuration.      * @param bootstrapServers Kafka server addresses.      * @param groupId Consumer group ID.      * @param topic Topic to subscribe to.      */     public KafkaConsumerService(String bootstrapServers, String groupId, String topic) {         Properties props = new Properties();         props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);         props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);         props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());         props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());         this.consumer = new KafkaConsumer<>(props);         this.consumer.subscribe(Collections.singletonList(topic));     }      /**      * Polls messages from Kafka topic indefinitely.      */     public void pollMessages() {         while (true) {             ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));             for (ConsumerRecord<String, String> record : records) {                 System.out.println("Received message: " + record.value() + " from topic: " + record.topic());             }         }     }      /**      * Closes the Kafka consumer.      */     public void close() {         consumer.close();     } }

KafkaConsumerService initializes a Kafka consumer with specified configuration (bootstrap servers, group ID, topic).
It continuously polls messages from the subscribed Kafka topic using a loop.
The class provides a method to close the Kafka consumer instance when no longer needed.

Step 5: Create the RequestResponseService Class

Java

package com.gfg;  import java.util.UUID;  /**  * Service class for handling request-response pattern using Kafka.  */ public class RequestResponseService {      private final KafkaProducerService producerService;     private final KafkaConsumerService consumerService;     private final String requestTopic;     private final String responseTopic;      /**      * Constructs a RequestResponseService with specified Kafka bootstrap servers and topics.      *      * @param bootstrapServers Kafka bootstrap servers to connect to.      * @param requestTopic     Topic for sending requests.      * @param responseTopic    Topic for receiving responses.      */     public RequestResponseService(String bootstrapServers, String requestTopic, String responseTopic) {         this.producerService = new KafkaProducerService(bootstrapServers);         this.consumerService = new KafkaConsumerService(bootstrapServers, "response-group", responseTopic);         this.requestTopic = requestTopic;         this.responseTopic = responseTopic;     }      /**      * Sends a request message to the Kafka topic specified by requestTopic.      *      * @param message Message to send as a request.      */     public void sendRequest(String message) {         String correlationId = UUID.randomUUID().toString();         producerService.sendMessage(requestTopic, correlationId, message);         System.out.println("Sent request with correlationId: " + correlationId);         // In a real-world scenario, implement a more robust correlation ID handling mechanism.     }      /**      * Starts listening for responses from the Kafka response topic.      */     public void receiveResponse() {         consumerService.pollMessages();     }      /**      * Closes the producer and consumer services.      */     public void close() {         producerService.close();         consumerService.close();     } }

It manages Kafka producer and consumer services for implementing request-response pattern.
Sends requests with unique correlation IDs to a specified topic and prints the ID.
Receives and prints responses from a designated Kafka topic indefinitely.

Step 6: Main Class

Java

package com.gfg;  public class MainApplication {      public static void main(String[] args) {         String bootstrapServers = "localhost:9092";         String requestTopic = "request-topic";         String responseTopic = "response-topic";          RequestResponseService service = new RequestResponseService(bootstrapServers, requestTopic, responseTopic);          // Send a request         service.sendRequest("Hello, Kafka!");           // Start receiving responses         service.receiveResponse();           // Close services on shutdown         Runtime.getRuntime().addShutdownHook(new Thread(() -> {             service.close();         }));     } }

Step 7: Run the application

Output:

This example project demonstrates the basic setup and implementation of the request and response pattern in Apache Kafka.

How to Get Number of Messages in a Topic in Apache Kafka in Java?

maheshdaxt

Improve

Article Tags :

Implementing Request Response in Java Apache Kafka

Prerequisites:

Implementing Request Response in Java Apache Kafka

Step 1: Create a Maven Project

Step 2: Add the Dependencies into the pom.xml file

Step 3: Create the KafkaProducerService Class

Step 4: Create the KafkaConsumerService Class

Step 5: Create the RequestResponseService Class

Step 6: Main Class

Step 7: Run the application

Output:

Similar Reads