Spring Boot Apache Kafka Example
Overview
In this article, we begin exploring how we will integrate apache kafka with Spring Boot
- Overview of Apache Kafka
- Install Kafka
- spring boot kafka project
Lets start
Overview of Apache Kafka
Apache Kafka is A high-throughput distributed streaming platform. It's a publish-subscribe messaging rethought as a distributed commit log
A streaming platform has three key capabilities:- Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system
- Store streams of records in a fault-tolerant durable way
- Process streams of records as they occur.

- Building real-time streaming data pipelines that reliably get data between systems or applications
- Building real-time streaming applications that transform or react to the streams of data
Install Kafka
- Download : Go to the download page of Apache Kafka https://kafka.apache.org/downloads.html and download https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.0/kafka_2.12-2.4.0.tgz
- Unzip : it to a particular location - "D:\kafka_2.12-2.4.0"
- Start zookeeper : This Kafka installation comes with a built-in zookeeper. Zookeeper is mainly used to track the status of the nodes present in the Kafka cluster and to keep track of Kafka topics, messages, etc.
D:\kafka_2.12-2.4.0>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
- Start the Apache Kafka : Use below command to start the Apache Kafka
D:\kafka_2.12-2.4.0>.\bin\windows\kafka-server-start.bat .\config\server.properties
Take a look at our suggested posts:
Create Spring Boot Kafka Producer
- Create Spring Boot Maven project
- Define the pom.xml as follows- Add the spring-kafka dependency.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.techgeeknext</groupId> <artifactId>SpringBootKafkaExample</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>1.5.3.RELEASE</version> <relativePath/> </parent> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>
- Define the KafkaProducer class to send message to the kafka topic named as my-topic.
package com.techgeeknext.service; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.stereotype.Service; @Service public class KafkaProducer { @Autowired private KafkaTemplate<String, String> kafkaTemplate; String kafkaTopic = "my-topic"; public void send(String message) { kafkaTemplate.send(kafkaTopic, message); } }
- Define a Controller which will pass the message and trigger the send message to the Kafka Topic using the KafkaProducer class.
package com.techgeeknext.controller; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestParam; import org.springframework.web.bind.annotation.RestController; import com.techgeeknext.service.KafkaProducer; @RestController @RequestMapping(value = "/techgeeknext-kafka/") public class KafkaWebController { @Autowired KafkaProducer kafkaProducer; @GetMapping(value = "/producer") public String producer(@RequestParam("message") String message) { kafkaProducer.send(message); return "Message sent Successfully to the Kafka topic my-topic"; } }
- Next start the Spring Boot Application by running it as a Java Application.
- Start kafka console consumer listening to the my-topic from command prompt
D:\kafka_2.12-2.4.0>.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic my-topic --from-beginning
- Finally hit the url as follows- http://localhost:8080//techgeeknext-kafka/producer?message=test
This will trigger the message to be sent to the my-topic.
- On console consumer running on command prompt, we can see in the consumer started the message is received.
Download Source Code
The full source code for this article can be found on below.Download it here - Spring Boot Kafka Example