Express - Elastic Search - Monitoring 2

Next Step

Create a file outside your project "elk-express-ts-monitoring" and call it: "docker-compose.yml"

Check my docker-compose.yml on github

This Docker Compose file outlines the configuration for setting up a monitoring system with Elasticsearch, Kibana, Filebeat, and a custom application (elk-express-ts-monitoring). I hope the users of this content have basic knowladge of DOCKER!

Services: The services section defines four services: elasticsearch, kibana, filebeat, and elk-express-ts-monitoring. Each service corresponds to a container that will be managed by Docker Compose.

Elasticsearch:

• image: Uses Elasticsearch 7.15.0 from Elastic's official Docker repository.
• ports: Maps port 9200 on the host to port 9200 in the container, allowing external access to Elasticsearch.
• environment: Sets up Elasticsearch to run as a single-node cluster, which is suitable for development but not for production.

Kibana:

• image: Specifies the Docker image to use for Kibana, version 7.15.0.
• ports: Maps port 5601 of the host machine to port 5601 of the container, which is the default port for Kibana.
• depends_on: Indicates that the Kibana service depends on the Elasticsearch service, ensuring Elasticsearch starts before Kibana.

Filebeat:

• build: Specifies that the Filebeat image should be built from the Dockerfile located in the ./filebeat directory.
• container_name: Sets the name of the container to filebeat.
• command: Overrides the default command and runs Filebeat with the specified flags (-e for logging to the console and -strict.perms=false to ignore strict permission checks).
• volumes: Mounts a local directory (./elk-express-ts-monitoring-logs) to a directory inside the container (/src/logs).
• depends_on: Indicates that the Filebeat service depends on both Elasticsearch and Kibana services.

elk-express-ts-monitoring:

• build: Specifies that the elk-express-ts-monitoring image should be built from the Dockerfile located in the ./elk-express-ts-monitoring directory.
• container_name: Sets the name of the container to elk-express-ts-monitoring.
• ports: Maps port 3001 of the host machine to port 3001 of the container.
• depends_on: Indicates that this service depends on the Elasticsearch service.
• volumes: Mounts a local directory (./elk-express-ts-monitoring-logs) to a directory inside the container (/src/logs).

Volumes:

This section defines a named volume called elasticsearch_data with the local driver. This volume can be used to persist Elasticsearch data, although it is not explicitly used in the services above.

This Docker Compose file sets up an ELK stack (Elasticsearch, Logstash, and Kibana) with an additional custom service (elk-express-ts-monitoring) and Filebeat for log shipping. The configuration ensures that the services start in the correct order and are appropriately connected.

Check my Dockerfile - filebeat on github

This code is a Dockerfile, which is used to build a Docker image for Filebeat, a lightweight shipper for forwarding and centralizing log data. Let's break down each line to understand what it does:

FROM docker.elastic.co/beats/filebeat-oss:7.15.0:
Specifies the base image for the Docker image being built. It uses the filebeat-oss:7.15.0 image from the Elastic Docker registry as the base. This image contains Filebeat version 7.15.0.

COPY filebeat.yml /usr/share/filebeat/filebeat.yml:
Copies a configuration file into the Docker image. It copies the local filebeat.yml configuration file into the directory /usr/share/filebeat/ inside the Docker image.

USER root:
Sets the user for the subsequent commands to root. The following commands will be executed with root privileges.

RUN chown -R root /usr/share/filebeat/:
Changes the ownership of the specified directory. It changes the ownership of the /usr/share/filebeat/ directory and all its contents to the root user recursively (-R flag).

RUN chmod -R go-w /usr/share/filebeat/:
Modifies the permissions of the specified directory. It removes write permissions (-w) for group (g) and others (o) on the /usr/share/filebeat/ directory and all its contents recursively (-R flag).

The Dockerfile starts with a base image for Filebeat version 7.15.0. It copies a local Filebeat configuration file (filebeat.yml) into the appropriate directory inside the Docker image. It ensures that the directory and its contents are owned by the root user and have restricted write permissions for group and others. This setup is useful for running Filebeat in a containerized environment with a specific configuration and controlled permissions.

Check my Filebeat.yml on github

This configuration code is for setting up Filebeat to monitor log files and send log data to an Elasticsearch instance. Here's a breakdown of each section:

• name: Sets the name of the Filebeat instance to "elk-express-ts-monitoring-filebeat".
• logging.metrics.enabled: Disables logging metrics.
• xpack.security.enabled: Disables X-Pack security features.
• xpack.monitoring.enabled: Disables X-Pack monitoring.
• setup.ilm.enabled: Disables Index Lifecycle Management (ILM) setup.
• setup.template.enabled: Disables template setup.

• filebeat.inputs: Defines the input settings for Filebeat.
• type: log: Specifies that the input type is log files.
• scan_frequency: 1s: Sets the frequency to scan for new log files to every 1 second.
• enabled: true: Enables this input configuration.
• paths: Specifies the paths to the log files to be monitored. In this case, all .log files in the /src/logs/ directory.
• fields: Adds custom fields to the log events.
• service: elk-express-ts-monitoring: Adds a custom field named service with the value elk-express-ts-monitoring.
• fields_under_root: true: Puts custom fields under the root of the event, rather than under a fields sub-dictionary.
• json: Specifies settings for parsing JSON logs.
• keys_under_root: true: Places JSON keys directly under the root of the event.
• overwrite_keys: true: Allows JSON keys to overwrite any existing keys at the root level.
• message_key: 'message': Specifies the key in the JSON object that contains the log message.

• output.elasticsearch: Defines the output settings for sending data to Elasticsearch.
• hosts: Specifies the Elasticsearch hosts to send data to. In this case, it is elasticsearch:9200.
• index: Sets the name of the index where logs will be stored in Elasticsearch to elk-express-ts-monitoring.

This Filebeat configuration monitors log files located in /src/logs/, parses them as JSON, and sends the log data to an Elasticsearch instance running at elasticsearch:9200. The logs are indexed in Elasticsearch under the index name elk-express-ts-monitoring. It disables several X-Pack features and setup options, focusing on basic log shipping functionality.

Check my Dockerfile on github

This Dockerfile is used to create a Docker image for a Node.js application. Here’s a step-by-step explanation of what each line does:

FROM node:20
This line specifies the base image for the Docker image. In this case, it uses the official Node.js image, version 20. This base image includes Node.js and npm (Node Package Manager).

WORKDIR /src
This sets the working directory inside the Docker container to /src. Any subsequent commands (e.g., COPY, RUN) will be executed within this directory.

COPY package.json ./*
This copies the package.json and package-lock.json files (if they exist) from the local filesystem to the current working directory in the container (/src). These files are used to define and lock the dependencies of the Node.js application.

RUN npm install
This runs the npm install command inside the container. It reads the package.json and installs all the dependencies listed there into the container.

COPY . .
This copies all the files from the current directory on the local filesystem to the current directory in the container (/src). Essentially, it transfers the entire application code into the container.

EXPOSE 3001
This informs Docker that the container will listen on port 3001 at runtime. It doesn't actually publish the port; it's more of a documentation step for others who might use the image. To publish the port, you would use the -p flag when running the container (e.g., docker run -p 3001:3001).

CMD ["npm", "start"]
This specifies the command to run when the container starts. In this case, it runs npm start, which typically starts the Node.js application. The actual command for npm start is defined in the scripts section of the package.json file.

To summarize, this Dockerfile sets up a Node.js application inside a Docker container, installs its dependencies, copies the application code into the container, and sets up the container to run the application on port 3001 when it starts.

This is the end of part 2, click here to continue to the part 3