Apache Kafka - Create Producer with Keys using Java

Last Updated : 24 Apr, 2025

Apache Kafka is a publish-subscribe messaging system. A messaging system lets you send messages between processes, applications, and servers. Apache Kafka is software where topics (A topic might be a category) can be defined and further processed. Read more on Kafka here: What is Apache Kafka and How Does it Work. Kafka Producers are going to write data to topics and topics are made of partitions. Now the producers in Kafka will automatically know to which broker and partition to write based on your message and in case there is a Kafka broker failure in your cluster the producers will automatically recover from it which makes Kafka resilient and which makes Kafka so good and used today. In this article, we are going to discuss the step-by-step implementation of how to Create an Apache Kafka Producer with Keys using Java.

Step-by-Step Implementation

Step 1: Create a New Apache Kafka Project in IntelliJ

To create a new Apache Kafka Project in IntelliJ using Java and Maven please refer to How to Create an Apache Kafka Project in IntelliJ using Java and Maven.

Step 2: Install and Run Apache Kafka

To Install and Run Apache Kafka in your local system please refer to How to Install and Run Apache Kafka.

Step 3: Create Producer with Keys

First, we have to create Producer Properties. And to create Producer Properties refer to the below code snippet

Create Producer Properties:

Properties properties = new Properties();  properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServer);  properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());  properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

Create the Producer:

KafkaProducer<String, String> producer = new KafkaProducer<>(properties);

Create a Producer Record with Key:

String topic = "gfg_topic";  String value = "hello_geeksforgeeks " + i;  String key = "id_" + i;    ProducerRecord<String, String> record =            new ProducerRecord<>(topic, key, value);

Java Producer with Callback:

producer.send(record, (recordMetadata, e) -> {                  // Executes every time a record successfully sent                  // or an exception is thrown                  if (e == null) {                      logger.info("Received new metadata. \n" +                              "Topic: " + recordMetadata.topic() + "\n" +                              "Partition: " + recordMetadata.partition() + "\n" +                              "Offset: " + recordMetadata.partition() + "\n");                  } else {                      logger.error("Error while producing ", e);                  }              }).get(); // Block the .send() to make it synchronous

Flush and Close the Producer:

producer.flush();  producer.close();

Below is the complete code. Comments are added inside the code to understand the code in more detail.

Java

package org.kafkademo.basics;  import org.apache.kafka.clients.producer.*; import org.apache.kafka.common.serialization.StringSerializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory;  import java.util.Properties; import java.util.concurrent.ExecutionException;  public class KafkaProducerWithKeyDemo {     public static void main(String[] args) throws ExecutionException, InterruptedException {          Logger logger = LoggerFactory.getLogger(KafkaProducerWithKeyDemo.class);          String bootstrapServer = "127.0.0.1:9092";          // Create Producer Properties         Properties properties = new Properties();         properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServer);         properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());         properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());          // Create the Producer         KafkaProducer<String, String> producer = new KafkaProducer<>(properties);          for (int i = 0; i < 10; i++) {              String topic = "gfg_topic";             String value = "hello_geeksforgeeks " + i;             String key = "id_" + i;              // Log the Key             logger.info("Key: " + key);              // Create a Producer Record with Key             ProducerRecord<String, String> record =                     new ProducerRecord<>(topic, key, value);              // Java Producer with Callback             producer.send(record, (recordMetadata, e) -> {                 // Executes every time a record successfully sent                 // or an exception is thrown                 if (e == null) {                     logger.info("Received new metadata. \n" +                             "Topic: " + recordMetadata.topic() + "\n" +                             "Partition: " + recordMetadata.partition() + "\n" +                             "Offset: " + recordMetadata.partition() + "\n");                 } else {                     logger.error("Error while producing ", e);                 }             }).get(); // Block the .send() to make it synchronous         }          // Flush and Close the Producer         producer.flush();         producer.close();      } }

Step 4: Run the Application

Now run the application and below is the output.

"C:\Users\Amiya Rout\.jdks\corretto-11.0.15\bin\java.exe" "-javaagent:  [main] INFO org.apache.kafka.clients.producer.ProducerConfig - ProducerConfig values:       acks = 1      batch.size = 16384      bootstrap.servers = [127.0.0.1:9092]      buffer.memory = 33554432      client.dns.lookup = use_all_dns_ips      client.id = producer-1      compression.type = none      connections.max.idle.ms = 540000      delivery.timeout.ms = 120000      enable.idempotence = false      interceptor.classes = []      internal.auto.downgrade.txn.commit = false      key.serializer = class org.apache.kafka.common.serialization.StringSerializer      linger.ms = 0      max.block.ms = 60000      max.in.flight.requests.per.connection = 5      max.request.size = 1048576      metadata.max.age.ms = 300000      metadata.max.idle.ms = 300000      metric.reporters = []      metrics.num.samples = 2      metrics.recording.level = INFO      metrics.sample.window.ms = 30000      partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner      receive.buffer.bytes = 32768      reconnect.backoff.max.ms = 1000      reconnect.backoff.ms = 50      request.timeout.ms = 30000      retries = 2147483647      retry.backoff.ms = 100      sasl.client.callback.handler.class = null      sasl.jaas.config = null      sasl.kerberos.kinit.cmd = /usr/bin/kinit      sasl.kerberos.min.time.before.relogin = 60000      sasl.kerberos.service.name = null      sasl.kerberos.ticket.renew.jitter = 0.05      sasl.kerberos.ticket.renew.window.factor = 0.8      sasl.login.callback.handler.class = null      sasl.login.class = null      sasl.login.refresh.buffer.seconds = 300      sasl.login.refresh.min.period.seconds = 60      sasl.login.refresh.window.factor = 0.8      sasl.login.refresh.window.jitter = 0.05      sasl.mechanism = GSSAPI      security.protocol = PLAINTEXT      security.providers = null      send.buffer.bytes = 131072      socket.connection.setup.timeout.max.ms = 30000      socket.connection.setup.timeout.ms = 10000      ssl.cipher.suites = null      ssl.enabled.protocols = [TLSv1.2, TLSv1.3]      ssl.endpoint.identification.algorithm = https      ssl.engine.factory.class = null      ssl.key.password = null      ssl.keymanager.algorithm = SunX509      ssl.keystore.certificate.chain = null      ssl.keystore.key = null      ssl.keystore.location = null      ssl.keystore.password = null      ssl.keystore.type = JKS      ssl.protocol = TLSv1.3      ssl.provider = null      ssl.secure.random.implementation = null      ssl.trustmanager.algorithm = PKIX      ssl.truststore.certificates = null      ssl.truststore.location = null      ssl.truststore.password = null      ssl.truststore.type = JKS      transaction.timeout.ms = 60000      transactional.id = null      value.serializer = class org.apache.kafka.common.serialization.StringSerializer    [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.8.0  [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: ebb1d6e21cc92130  [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1674837796602  [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_0  [kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Cluster ID: orhF-HNsR465cORhmU3pTg  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_1  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_2  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_3  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_4  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_5  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_6  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_7  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_8  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Key: id_9  [kafka-producer-network-thread | producer-1] INFO org.kafkademo.basics.KafkaProducerWithKeyDemo - Received new metadata.   Topic: gfg_topic  Partition: 0  Offset: 0    [main] INFO org.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.  [main] INFO org.apache.kafka.common.metrics.Metrics - Metrics scheduler closed  [main] INFO org.apache.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter  [main] INFO org.apache.kafka.common.metrics.Metrics - Metrics reporters closed  [main] INFO org.apache.kafka.common.utils.AppInfoParser - App info kafka.producer for producer-1 unregistered    Process finished with exit code 0

So in the console, you can see, now we have displayed the Key also. Refer to the below image.

Also, you can see the message in the Kafka consumer console. Run this command

kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic gfg_topic --group gfg-group

Output:

Apache Kafka - Create Producer with Callback using Java

AmiyaRanjanRout

Improve

Article Tags :

Practice Tags :

Java

Apache Kafka - Create Producer with Keys using Java

Step-by-Step Implementation

Similar Reads