Architecture:

Now we will complete the remaining stack setup using the Helm charts. The main components of this stack are Prometheus, Grafana, and Nodejs Applications in EKS Cluster. We will start with the Prometheus.

Install Prometheus:

Now install the Prometheus using the helm chart.

Add Prometheus helm chart repository.

$ kubectl create ns prometheus 
####Creating a namespace is best practice so that we will create namespace now 
$ helm repo add stable https://charts.helm.sh/stable
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm search repo prometheus-community
$ helm install stable prometheus-community/kube-prometheus-stack --namespace prometheus

Now we will be able to access the resources in the namespace Prometheus, all pods are running now.

By using the below command we will be able to see the resources in the namespace of Prometheus.

kubectl get all -n prometheus

Now we want to access the Prometheus dashboard on the web so that we can be able to scrape the metrics and we can add them as a Data Source in the Grafana.

We need to edit the service file ClusterIP into LoadBalancer so that our Prometheus will be able to run on the web.

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus

Do the changes as ClusterIP to LoadBalancer. So that we will be able to see the targets in the Prometheus UI as shown in the below image. If you apply the command which is shown below:

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus

So that you will be able to see the image which is shown below:

You have to take the load balancer URL with then port 9090. Then you will able to access the Prometheus UI as shown below:

In the above section, there will be targets available, in that specific section we will be able to see the metrics of every service which is append running in the EKS Cluster.

Install Grafana:

Add the Grafana helm chart repository. Later, Update the helm chart repository.

helm repo add grafana https://grafana.github.io/helm-charts 
helm repo update

Now we need to create a Prometheus data source so that Grafana can access the Kubernetes metrics. Create a yaml file prometheus-datasource.yaml and save the following data source configuration into it -

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true

Create a namespace grafana

kubectl create namespace grafana

Install the Grafana

helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='EKS!sAWSome' \
    --values prometheus-datasource.yaml \
    --set service.type=LoadBalancer

Verify the Grafana installation by using the following kubectl command -

kubectl get all -n grafana

Copy External IP address and open it in the browser -

Password you mentioned as EKS!sAWSome while creating Grafana

You can see in the above I have accessed the Grafana using the Loadbalancer and Whatever credentials I have given while installing the Helm charts.

Deploy a Node.js application and monitor it on Grafana.

Now that we have the Docker image, we’ll create a Helm chart so we can deploy the web app to the Kubernetes cluster.

This was the image we pushed into the Dockerhub in the previous Blog.

Part1 blog

Create the necessary structure.

$ helm create nodejsapp
In the main user directory follow this command.

The specific command will create the necessary directories for the deployment of the Nodejs. So that we will be able to scrape the metrics to the Prometheus and then we can visualize in the Grafana.

Go to the sub-directory nodejsapp and edit the values.yaml file first. Look for the image key first and replace it so it looks like this. Make sure you replace the with your Docker Hub username.

image:
  repository: shreedhar4037/nodejs-cci  ###<username>/<repository name>
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

Then look for podAnnotations: {} line and replace it so it looks like this.

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "3000"

My application is running in the 3000 port. So we can give the same ports to the container and the pod as well. If you want you can change the port here.

Look for the service parameter and change it so it looks like this.

service:
  type: LoadBalancer
  port: 80
  targetPort: 3000
  name: nodejsapp-service

Look for the resources key and uncomment the defaults.

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits:
  cpu: 100m
  memory: 128Mi
requests:
  cpu: 100m
  memory: 128Mi

Now we have to get into the directory where the chart folders have located. So we need to edit the templates/deployment.yaml file.

Edit the templates/deployment.yaml file and find these lines.

ports:
  - name: http
    containerPort: {{ .Values.service.port }}

Replace the container port to look like this.

containerPort: 3000

Go back to the root of the nodejsapp folder and check the chart.

helm install nodejsapp --generate-name

Execute the export SERVICE_IP command above and echo the IP. That’s your URL

kubectl get svc

Copy the load balancer URL and paste it into the web you will be able to see the application which we are running inside the pod of the container.

The Nodejs Application is running successfully.

What we care about is the pod. Get the pod running. It’s in the default namespace.

Get the annotations. Replace with your pod name.

kubectl get pod nodejsapp-1689442502-648f7b9587-2g8pr -o jsonpath='{.metadata.annotations}'
{"kubernetes.io/psp":"eks.privileged","prometheus.io/path":"/metrics","prometheus.io/port":"3000","prometheus.io/scrape":"true"}

Import the Grafana dashboard from Grafana Labs.

Now we have set up everything in terms of Prometheus and Grafana. For the custom Grafana Dashboard, we are going to use the open-source Grafana dashboard. For this session, I am going to import a dashboard 6417.

Grafana Dashboard

In the above image, there will be an option for Import. We have to click on that and then we will get an interface as shown below.

Click on the Import option then you will get a Dashboard like this as shown in below.

Refresh the Grafana dashboard to verify the deployment for getting the Nodejs application in the dashboard.

I will show you some of the metrics which is related to the Nodejs Application in the Grafana dashboard.

Nodejs Deployment usage resources in cluster:

Now we will see the EKS Cluster related to Workloads:

Clean Up

In this stage, you're going to clean up and remove all resources which we created during the session. So that it will not be charged to you afterward.

1. Delete the EKS cluster with the following command.

eksctl delete cluster --name <clustername>

1. Delete EC2 Instance.

References:

prometheus.io

grafana.com

kubernetes.io

Follow this blog channel for further interesting topics in the

#10weeksofcloudops

Follow me on Linkedin

Follow me on GitHub

Thank you:

I conclude this exercise will help you to understand the concepts of using Kubernetes metrics monitoring using Prometheus and Grafana dashboards.

Thanks for reading to the end; I hope you gained some knowledge.