Spring Boot Apache Kafka Example

Overview

In this article, we begin exploring how we will integrate apache kafka with Spring Boot

1. Overview of Apache Kafka
2. Install Kafka
3. spring boot kafka project

Lets start

Overview of Apache Kafka

Apache Kafka is A high-throughput distributed streaming platform. It's a publish-subscribe messaging rethought as a distributed commit log

1. Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system
2. Store streams of records in a fault-tolerant durable way
3. Process streams of records as they occur.

1. Building real-time streaming data pipelines that reliably get data between systems or applications
2. Building real-time streaming applications that transform or react to the streams of data

Install Kafka

1. Download : Go to the download page of Apache Kafka https://kafka.apache.org/downloads.html and download https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.0/kafka_2.12-2.4.0.tgz
2. Unzip : it to a particular location - "D:\kafka_2.12-2.4.0"
3. Start zookeeper : This Kafka installation comes with a built-in zookeeper. Zookeeper is mainly used to track the status of the nodes present in the Kafka cluster and to keep track of Kafka topics, messages, etc.

   D:\kafka_2.12-2.4.0>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

4. Start the Apache Kafka : Use below command to start the Apache Kafka

   D:\kafka_2.12-2.4.0>.\bin\windows\kafka-server-start.bat .\config\server.properties

Create Spring Boot Kafka Producer

1. Create Spring Boot Maven project
2. Define the pom.xml as follows- Add the spring-kafka dependency.

   <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.techgeeknext</groupId>
     <artifactId>SpringBootKafkaExample</artifactId>
     <version>0.0.1-SNAPSHOT</version>
   
     <packaging>jar</packaging>
   
   
   	<parent>
   		<groupId>org.springframework.boot</groupId>
   		<artifactId>spring-boot-starter-parent</artifactId>
   		<version>1.5.3.RELEASE</version>
   		<relativePath/>
   	</parent>
   
   	<properties>
   		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
   		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
   		<java.version>1.8</java.version>
   	</properties>
   
   	<dependencies>
   		<dependency>
   			<groupId>org.springframework.kafka</groupId>
   			<artifactId>spring-kafka</artifactId>
   		</dependency>
   		<dependency>
   			<groupId>org.springframework.boot</groupId>
   			<artifactId>spring-boot-starter-web</artifactId>
   		</dependency>
   		<dependency>
   			<groupId>org.springframework.boot</groupId>
   			<artifactId>spring-boot-starter-test</artifactId>
   			<scope>test</scope>
   		</dependency>
   	</dependencies>
   
   	<build>
   		<plugins>
   			<plugin>
   				<groupId>org.springframework.boot</groupId>
   				<artifactId>spring-boot-maven-plugin</artifactId>
   			</plugin>
   		</plugins>
   	</build>
   
   </project>

3. Define the KafkaProducer class to send message to the kafka topic named as my-topic.

   package com.techgeeknext.service;
   
   import org.springframework.beans.factory.annotation.Autowired;
   import org.springframework.kafka.core.KafkaTemplate;
   import org.springframework.stereotype.Service;
   
   @Service
   public class KafkaProducer {
   
   	@Autowired
   	private KafkaTemplate<String, String> kafkaTemplate;
   
   	String kafkaTopic = "my-topic";
   
   	public void send(String message) {
   
   	    kafkaTemplate.send(kafkaTopic, message);
   	}
   }

4. Define a Controller which will pass the message and trigger the send message to the Kafka Topic using the KafkaProducer class.

   package com.techgeeknext.controller;
   
   import org.springframework.beans.factory.annotation.Autowired;
   import org.springframework.web.bind.annotation.GetMapping;
   import org.springframework.web.bind.annotation.RequestMapping;
   import org.springframework.web.bind.annotation.RequestParam;
   import org.springframework.web.bind.annotation.RestController;
   
   import com.techgeeknext.service.KafkaProducer;
   
   @RestController
   @RequestMapping(value = "/techgeeknext-kafka/")
   public class KafkaWebController {
   
   	@Autowired
   	KafkaProducer kafkaProducer;
   
   	@GetMapping(value = "/producer")
   	public String producer(@RequestParam("message") String message) {
   		kafkaProducer.send(message);
   		return "Message sent Successfully to the Kafka topic my-topic";	}
   }
   
   

5. Next start the Spring Boot Application by running it as a Java Application.
6. Start kafka console consumer listening to the my-topic from command prompt

   D:\kafka_2.12-2.4.0>.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic my-topic --from-beginning

7. Finally hit the url as follows- http://localhost:8080//techgeeknext-kafka/producer?message=test
   This will trigger the message to be sent to the my-topic.
   
8. On console consumer running on command prompt, we can see in the consumer started the message is received.

Download Source Code