Links

Kubernetes Monitoring

Monitor metrics from Kubernetes
This manual is for Kubernetes with minikub on a virtual machine running the Linux Ubuntu operating system.

Quickstart

Step 1 - Preparation
Minikube must be running with Kubernetes version <=v1.23.0 and the Kubernetes API must be exposed.
The recommended way to expose the API is to redirect the API port (default is 8443). This can be done with the following command:
minikube start --kubernetes-version=v1.23.0 --ports=8443:8443
Step 2 - Installation
In Kubernetes, you must install kube-state-metrics and cAdvisor (cAdvisor is integrated into kubelet by default).
Step 3 - kube-state-metrics
For the purposes of the instructions, we used the example provided here.
After downloading the entire folder to the virtual machine with Kubernetes, we type the following command:
kubectl apply -f path_to_folder
Step 4 - Data collector
The next step is to add a data collector to collect data from Kubernetes and send it to the OAP.
Download:
Then set the RevDeBug server address in the open-telemetry file (line 36) - REVDEBUG_SERVER_NAME.
Then in the same file name your cluster k8s (line 57 and line 84) - YOUR_CLUSTER_NAME.
Step 5 - Running
Run the deployment of these files after by:
kubectl apply -f path_to_folder_where_they_are_both_files
Step 6 - Setup
You need to copy the contents of the .kube/config file for Linux the full path is /home/your_user/.kube/config
config
apiVersion: v1
clusters:
- cluster:
certificate-authority: /home/ubuntu/.minikube/ca.crt
extensions:
- extension:
last-update: Thu, 16 Mar 2023 12:22:22 UTC
provider: minikube.sigs.k8s.io
version: v1.29.0
name: cluster_info
server: https://my_k8s_adress:8443
name: minikube
contexts:
- context:
cluster: minikube
extensions:
- extension:
last-update: Thu, 16 Mar 2023 12:22:22 UTC
provider: minikube.sigs.k8s.io
version: v1.29.0
name: context_info
namespace: default
user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
client-certificate: /home/ubuntu/.minikube/profiles/minikube/client.crt
client-key: /home/ubuntu/.minikube/profiles/minikube/client.key
Then copy the 3 certificates whose paths are given in the config file:
  • /home/ubuntu/.minikube/ca.crt
  • /home/ubuntu/.minikube/profiles/minikube/client.crt
  • /home/ubuntu/.minikube/profiles/minikube/client.key
In the next step, switch to the server with RevDeBug:
  • Go to the /var/revdebug/apm-oap/config/k8s folder - if the folder does not exist then create it.
  • Create a new file starting with config, e.g. config-cluster1, config-name.
  • Next, create a folder where we will keep the previously downloaded keys, for example: name_your_folder_with_certs.
  • In the config-your_name file, we paste the contents of our config from minikub.
  • We are adjusting the verifier paths to the RevDeBug container:
certificate-authority: /revdebug/config/k8s/name_of_folder_with_cert/ca.crt
client-certificate: /revdebug/config/k8s/name_of_folder_with_cert/client.crt
client-key: /revdebug/config/k8s/name_of_folder_with_cert/client.key
  • We adjust the Kubernetes address - by default, there is an internal API set the address of your Kubernetes/Port API that we exposed at the beginning. Important: always https.
apiVersion: v1
clusters:
- cluster:
certificate-authority: /revdebug/config/k8s/name_of_folder_with_cert/ca.crt
extensions:
- extension:
last-update: Thu, 16 Mar 2023 12:22:22 UTC
provider: minikube.sigs.k8s.io
version: v1.29.0
name: cluster_info
server: https://my_k8s_adress:8443
name: minikube
contexts:
- context:
cluster: minikube
extensions:
- extension:
last-update: Thu, 16 Mar 2023 12:22:22 UTC
provider: minikube.sigs.k8s.io
version: v1.29.0
name: context_info
namespace: default
user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
client-certificate: /revdebug/config/k8s/name_of_folder_with_cert/client.crt
client-key: /revdebug/config/k8s/name_of_folder_with_cert/client.ke
Step 7 - Environment variables
Go to the folder with docker-compose on the RevDeBug server. Add the following variables to the .env file:
  • SW_OTEL_RECEIVER_ENABLED_HANDLERS="oc"
  • SW_OTEL_RECEIVER=default
  • SW_OTEL_RECEIVER_ENABLED_OC_RULES=k8s-cluster,k8s-node,k8s-service
If you have virtual machine monitoring set up then do not remove SW_OTEL_RECEIVER_ENABLED_OC_RULES:vm,oap just add more rules after the comma.
Step 8 - Down/up RevDeBug compose
docker compose -p revdebug up -d
If you want to enable Kubernetes monitoring option, please contact [email protected]