How to configure and run Kafka with TypeScript

Тo read what is Kafka? Why and in what cases should we use it? Click here

To configure kafka, let's start our journey from the beginning by creating a new project! First, we need to create a folder called 'kafka-typescript' and inside that folder, from the command prompt we need to execute: Kafka - docker-compose up

npm init -y

The command npm init -y quickly creates a new package.json file for a Node.js project with
default values, skipping the interactive setup process.

npm install express

The command npm install express installs the Express.js library, a fast, unopinionated,
minimalist web framework for Node.js, into your project

npm install -g typescript

The command npm install -g typescript installs TypeScript globally on your computer, making the
TypeScript compiler (tsc) available from any command line interface.

npm install typescript --save-dev

The command npm install typescript --save-dev installs TypeScript as a development dependency in your
Node.js project, saving it in your package.json under devDependencies.

npm install ts-node --save-dev

The command npm install ts-node --save-dev installs ts-node, a Node.js package that allows TypeScript
to be executed directly by the Node.js runtime, and saves it as a development dependency in your project's package.json file.

npm install @types/node @types/express --save-dev

The command npm install @types/node @types/express --save-dev installs TypeScript type definitions for Node.js
and Express as development dependencies to provide type checking and autocompletion features during development.

tsc --init

The command tsc --init initializes a new TypeScript project by creating a tsconfig.json file, which configures
various TypeScript compiler options for that project.

npm install kafkajs

The command npm install kafkajs installs the KafkaJS library, which provides a modern, simple, and robust Node.js
client for interacting with Apache Kafka.

npm install @types/kafkajs --save-dev

The command npm install @types/kafkajs --save-dev installs TypeScript type definitions for KafkaJS, allowing
type-safe development, and saves them as a development dependency in your project.

npm install body-parser

The command npm install body-parser installs the body-parser middleware, which is used in Express applications
to parse incoming request bodies before your handlers, available under the req.body property.

npm install --save-dev nodemon

The command npm install --save-dev nodemon installs Nodemon as a development dependency in your Node.js project,
helping automatically restart the server when file changes in the directory are detected.

Check my package.json on github

Next Step: tsconfig.json

Edit tsconfig.json to suit your project, ensuring settings like rootDir and outDir are configured. A simple configuration could look like this:

Check my tsconfig.json on github

Next Step: index.ts

This code sets up a basic Express server that listens on port 3000 and includes routes for sending and receiving messages using Apache Kafka:

Express Setup: It initializes an Express application stored in the variable app.

Body Parsing: It uses bodyParser.json() to parse incoming request bodies in JSON format, which is necessary for handling JSON data in POST requests.

Routes:

A GET route at the root (/) that responds with "Hello World!" when accessed.

A POST route at /api/send which uses the jsonParser middleware to handle JSON payloads and controllers.sendMessageToKafka as the route handler (the handler presumably sends incoming messages to a Kafka topic).

Kafka Consumer: Sets up a Kafka consumer with KafkaConfig, subscribing to messages on 'my-topic' and logging those messages to the console when they are received.

Server Activation: The server starts listening on the specified port (3000) and logs a message to the console indicating it is running and listening for requests.

Overall, this server not only handles basic web requests but is also integrated with Kafka to send messages to and consume messages from a Kafka topic.

Check my index.ts on github

Next Step types.d.ts:

This TypeScript code defines a custom type KafkaRequest that extends the standard Request type (commonly from an Express.js context), but modifies it by replacing the body property with a new structure specific to handling an array of messages. Each message in the array has a text and sender property. The Omit utility type is used to exclude the original body property from the Request type before adding a new body property tailored to include an array of Message objects.

Check my types.d.ts on github

Next Step config.ts:

This TypeScript code defines a class KafkaConfig that encapsulates the functionality for both producing and consuming messages using KafkaJS, a Node.js library for Apache Kafka. The class initializes a Kafka client connected to a broker at 'localhost:9093', and sets up a producer and a consumer. The consumer is part of a group identified by 'test-group'. There are two asynchronous methods in the class: produce for sending messages to a specified topic, and consume for reading messages from a topic and executing a callback function with the message content. Error handling is implemented in both methods to catch and log any issues that occur during the connection or message handling processes.

Check my config.ts on github

Next Step kafkaController.ts:

This TypeScript code defines an Express middleware function sendMessageToKafka that takes a Kafka-specific request, sends a message to a Kafka topic using a Kafka configuration, and handles the response. It imports necessary types from Express, a Kafka configuration module, and a custom type KafkaRequest. The function constructs a payload from the request body, initializes a Kafka configuration, and sends the message to the 'my-topic' Kafka topic. If successful, it sends a JSON response indicating success; if an error occurs, it logs the error and passes it to the next middleware handler for further processing. The function is then exported as part of the controllers object for use elsewhere in the application.

Check my kafkaController.ts on github

Next Step docker-compose.yml:

For this step you have to install Docker on your computer. Then you have to navigate to the folder where the file is located, in my case it is: kafka-typescript/docker. Run `docker-compose up` to start Kafka and Zookeeper.

This Docker Compose configuration sets up a local Kafka environment with ZooKeeper and Kafdrop for management and monitoring (Read more about different Kafka UI) . Here's a brief explanation of each part:

ZooKeeper Service: Runs the latest Bitnami ZooKeeper image, configured to allow anonymous logins, binds port 2181 for client connections, and uses a volume for data persistence.
Kafka Service: Utilizes the latest Bitnami Kafka image, linking to the ZooKeeper instance for coordination. It exposes ports 9092 for internal and 9093 for external Kafka connections, configures plain text listeners for communication, automatically creates and allows deletion of Kafka topics, and stores data persistently. It starts after ZooKeeper is available due to the depends_on setting.
Kafdrop Service: An observational tool that provides a UI to view Kafka topics and configurations, running on port 9000. It connects to Kafka using the internal network and also specifies Java VM options for performance.
Volumes: Specifies persistent storage for ZooKeeper and Kafka data to ensure data is maintained across container restarts.

Оn this picture we show exactly where the command `docker-compose up` should be written. And you have to run the command from the specific directory where the file is located.

how to run the app and docker container

Now everything is ready to run our application and docker containers. In the picture we show where the commands should be entered: `npm run start` and `docker-compose up`, first we have to run `docker-compose up` and after everything is alright, we have to run `npm run start` One more thing, before running `docker-compose up` your desctop docker should be running! Don't forget that step!

Kafka Pic 2

In the given picture you can see how everything works, our application as well as the docker containers. Excellent work!

docker application running with kafka

This picture clearly shows the docker application running on your computer and the result in it of running the command: `docker-compose up`. You can click the 9000:9000 link on the kafdrop line (on the right) to open kafdrop in your browser

main kafdrop page

This is the screen you should see in your browser after clicking on 9000:9000 from the docker container. Now we can enter the first topic from the bottom of the picture, where Topic is, there is a NEW button that we have to click.

create kafka topics

After clicking on the NEW button, we will see a new window where we can enter the topic we want to use. And after entering the topic, press the create button.

view kafka messages screen

On this screen we can clearly see the theme created. Congratulations! Now we have to click on the button VIEW MESSAGES to see the messages for this topic.

view kafka messages screen 2

At the moment we see that we have no published messages on the given photo.

Kafka Pic 9

4 To post a message in Kafka you need to use an additional tool like Postman or as I did in my case a plugin to my Visual Studio called Thunder Client. You have to create your message in the section JSON Content, method should be POST and the url should be: http://localhost:3000/api/send and click the button SEND. On the right you will receive a message with status code 200 and the message will say: 'Message successfully sent!'

show message in kafkadrop

After we have sent a message successfully from Postman or Thunder Client we go back to Kafdrop and we can refresh the page, then we should see the message published in the window as seen in this picture.

Full Code on Github: https://github.com/cholakovit/kafka-typescript