Kubernetes Monitoring

Monitor metrics from Kubernetes

This manual is for Kubernetes with minikub on a virtual machine running the Linux Ubuntu operating system.

Quickstart

Step 1 - Preparation

Minikube must be running with Kubernetes version <=v1.23.0 and the Kubernetes API must be exposed.

The recommended way to expose the API is to redirect the API port (default is 8443). This can be done with the following command:

minikube start --kubernetes-version=v1.23.0 --ports=8443:8443

Step 2 - Installation

In Kubernetes, you must install kube-state-metrics and cAdvisor (cAdvisor is integrated into kubelet by default).

Step 3 - kube-state-metrics

For the purposes of the instructions, we used the example provided here.

After downloading the entire folder to the virtual machine with Kubernetes, we type the following command:

kubectl apply -f path_to_folder

Step 4 - Data collector

The next step is to add a data collector to collect data from Kubernetes and send it to the OAP.

Download:

• permissions file

• open-telemetry file

Then set the RevDeBug server address in the open-telemetry file (line 36) - REVDEBUG_SERVER_NAME.

Then in the same file name your cluster k8s (line 57 and line 84) - YOUR_CLUSTER_NAME.

Step 5 - Running

Run the deployment of these files after by:

kubectl apply -f path_to_folder_where_they_are_both_files

Step 6 - Setup

You need to copy the contents of the .kube/config file for Linux the full path is /home/your_user/.kube/config

config

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/ubuntu/.minikube/ca.crt
    extensions:
    - extension:
        last-update: Thu, 16 Mar 2023 12:22:22 UTC
        provider: minikube.sigs.k8s.io
        version: v1.29.0
      name: cluster_info
    server: https://my_k8s_adress:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    extensions:
    - extension:
        last-update: Thu, 16 Mar 2023 12:22:22 UTC
        provider: minikube.sigs.k8s.io
        version: v1.29.0
      name: context_info
    namespace: default
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /home/ubuntu/.minikube/profiles/minikube/client.crt
    client-key: /home/ubuntu/.minikube/profiles/minikube/client.key

Then copy the 3 certificates whose paths are given in the config file:

• /home/ubuntu/.minikube/ca.crt

• /home/ubuntu/.minikube/profiles/minikube/client.crt

• /home/ubuntu/.minikube/profiles/minikube/client.key

In the next step, switch to the server with RevDeBug:

• Go to the /var/revdebug/apm-oap/config/k8s folder - if the folder does not exist then create it.

• Create a new file starting with config, e.g. config-cluster1, config-name.

• Next, create a folder where we will keep the previously downloaded keys, for example: name_your_folder_with_certs.

• In the config-your_name file, we paste the contents of our config from minikub.

• We are adjusting the verifier paths to the RevDeBug container:

certificate-authority: /revdebug/config/k8s/name_of_folder_with_cert/ca.crt
client-certificate: /revdebug/config/k8s/name_of_folder_with_cert/client.crt
client-key: /revdebug/config/k8s/name_of_folder_with_cert/client.key

• We adjust the Kubernetes address - by default, there is an internal API set the address of your Kubernetes/Port API that we exposed at the beginning. Important: always https.

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /revdebug/config/k8s/name_of_folder_with_cert/ca.crt
    extensions:
    - extension:
        last-update: Thu, 16 Mar 2023 12:22:22 UTC
        provider: minikube.sigs.k8s.io
        version: v1.29.0
      name: cluster_info
    server: https://my_k8s_adress:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    extensions:
    - extension:
        last-update: Thu, 16 Mar 2023 12:22:22 UTC
        provider: minikube.sigs.k8s.io
        version: v1.29.0
      name: context_info
    namespace: default
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate:  /revdebug/config/k8s/name_of_folder_with_cert/client.crt
    client-key: /revdebug/config/k8s/name_of_folder_with_cert/client.ke

Step 7 - Environment variables

Go to the folder with docker-compose on the RevDeBug server. Add the following variables to the .env file:

• SW_OTEL_RECEIVER_ENABLED_HANDLERS="oc"

• SW_OTEL_RECEIVER=default

• SW_OTEL_RECEIVER_ENABLED_OC_RULES=k8s-cluster,k8s-node,k8s-service

If you have virtual machine monitoring set up then do not remove SW_OTEL_RECEIVER_ENABLED_OC_RULES:vm,oap just add more rules after the comma.

Step 8 - Down/up RevDeBug compose

docker compose -p revdebug up -d

If you want to enable Kubernetes monitoring option, please contact [e-mail]