Welcome to this series of posts about microservices applications using NodeJS, NestJS Kafka, Kubernetes, and Amazon Elastic Kubernetes Service (EKS).
In this series, we are going to cover a microservices setup composed of two applications, an API and a worker.
In this post, we are going to create a REST API composed of basic CRUD operations that will produce Kafka events as the outcome.
Why NestJS, Kafka and Kubernetes?
Microservices are small, modular applications that can be easily created, tested, and deployed. They are the perfect solution for organizations that want to break their applications into smaller parts so that they can be scaled more effectively. NestJS is a Node.js framework that makes it easy to create microservices applications. Kafka is a messaging platform that enables programs to stream messages to one or many consumers. Kubernetes is a platform for managing containers. These three tools combined creates one of the best stacks to achieve high scalability and performance.
The end goal is to have a monorepo with three projects, the
microservices, and a
api project’s responsibility is to handle the synchronous side of the architecture. It will have REST endpoints available to perform CRUD operations and as an outcome of those operations, events are going to be fired to a Kafka topic.
microservices project will be responsible for the handling of the asynchronous requests, and we are going to configure a Kafka consumer using the
Lastly, we are going to create a
common project to hold code that should be shared across both projects.
After the project is complete, we are going to setup our applications in the cloud using Kubernetes and Amazon EKS.
Let’s start by creating the api project:
Open http://localhost:3000/ on your browser and you should see the default NestJS
Hello World message.
Open the project in your favorite IDE and look at the gen erated files. Please have a look at the
At this point, the application is up and running and we are going to create our new resource called
services. Imagine services being offered and as a few examples, we can imagine
nest generate resource # Type services # Select REST API # Select default YES
The output should be similar to the following picture:
command-line interface (CLI), NestJS generated the
services resource containing the module, DTOs, controller, and service files. Please have a look at the controller and service classes.
It’s time to edit
create-service.dto.ts to add the fields related to our services entity:
update-service.dto.ts extends the
CreateServiceDto class using nestJS provided
PartialType and also adds an
id field to identify the resource being modified:
For the changes to reflect automatically, it is necessary to restart the web server in development mode, to do so please run the following command:
# Press CTRL+C to stop previous execution npm run start:dev
Let’s test at least the GET and the POST /services endpoints. I have used Postman for this job.
At this point, we have generated an api endpoint containing all four CRUD operations.
For the rest of the tutorial, we are going to setup Kafka for local development, and also configure our application to use ClientKafka to send messages (events).
Kafka Development Setup
To setup the Kafka cluster, please follow the Docker Compose Kafka Setup (Confluent Cloud) tutorial. Once Kafka is running and accessible at http://localhost:9021, we are going to setup the Kafka Client on the
API – Request Flow & KafkaClient Configuration
api request flow should work as follow:
The /services API is going to receive a POST request to create a new service, then it will call the service layer passing the received payload, and lastly, the service layer is going to send a message to a Kafka topic using a ClientKafka Producer.
To configure Kafka, it is necessary to import the client and define a provider to create the Producer. The client and providers configurations are done at the
module level. Open
services.module.ts to configure the
CLIENT_KAFKA and the
KAFKA_PRODUCER provider as follow:
Please pay attention to the
brokers' property, make sure
host.docker.internal is set at
/etc/hosts file pointing to
There are two important parts in the above module configuration, the KafkaClient and the Kafka Producer provider. On the top section, the KafkaClient is configured using the ClientsModule.register method and at the bottom, a provider is specified to generate a Kafka producer using the injected KafkaClient registered above.
After the module configuration is done, it is necessary to inject the KafkaProducer into the service layer. The producer is used to send events to a Kafka topic.
After the code changes are done, it is time to validate the setup. The confluent
control-center web app provides us a nice UI to look for events in real-time.
Make sure to restart the web server again so that it connects to Kafka, open the
control-center (http://localhost:9021/) in your browser. Select the cluster and go to topic -> services -> messages.
With the messages tab open, it is time to send a POST request to the
api server. You can do that using the following command or your favorite client.
You should be able to see the new message in the
control-center, and that validates your setup is good.
NodeJS Microservices with NestJS, Kafka, and Kubernetes – Other Series Posts
- NestJS Kafka and Kubernetes – Part 1
- NodeJS Microservices with NestJS, Kafka, and Kubernetes – Part 2
- NestJS Shared Module Example – Monorepo Setup – Part 3
In the first part of this series, we have learned how to create a NestJS project capable of handling REST requests for CRUD operations, as well as connecting to Kafka and sending events to a topic.
In the second part of this series, we are going to explore the
microservices part of this project. The
microservices project will be responsible for consuming the Kafka events sent by the
If you liked this post, comment and share it with your friends.
All the best,