Aller au contenu principal

Advanced Options and Configuration

This section contains advanced information describing the different ways you can run and manage the cluster.

Certificate Rotation​

By default, certificates expire in 12 months.

If the certificates are expired or have fewer than 90 days remaining before they expire, the certificates are rotated when cluster is restarted.

Certificates can also be rotated manually. To do this, it is best to stop the rke2-server process, rotate the certificates, then start the process up again:

systemctl stop rke2-server
rke2 certificate rotate
systemctl start rke2-server

To renew agent certificates, restart rke2-agent in agent nodes. Agent certificates are renewed every time the agent starts.

systemctl restart rke2-agent

It is also possible to rotate an individual service by passing the --service flag, for example: rke2 certificate rotate --service api-server. See Certificate Management for more details.

Configuring an HTTP proxy​

If you are running the cluster in an environment, which only has external connectivity through an HTTP proxy, you can configure your proxy settings on the RKE2 systemd service. These proxy settings will then be used in RKE2 and passed down to the embedded containerd and kubelet.

Add the necessary HTTP_PROXY, HTTPS_PROXY and NO_PROXY variables to the environment file of your systemd service, usually:

  • /etc/default/rke2-server
  • /etc/default/rke2-agent

RKE2 will automatically add the cluster internal Pod and Service IP ranges and cluster DNS domain to the list of NO_PROXY entries. You should ensure that the IP address ranges used by the Kubernetes nodes themselves (i.e. the public and private IPs of the nodes) are included in the NO_PROXY list, or that the nodes can be reached through the proxy.

HTTP_PROXY=http://your-proxy.example.com:8888
HTTPS_PROXY=http://your-proxy.example.com:8888
NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

If you want to configure the proxy settings for containerd without affecting RKE2 and the Kubelet, you can prefix the variables with CONTAINERD_:

CONTAINERD_HTTP_PROXY=http://your-proxy.example.com:8888
CONTAINERD_HTTPS_PROXY=http://your-proxy.example.com:8888
CONTAINERD_NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

How Agent Node Registration Works

Agent nodes are registered via a websocket connection initiated by the rke2 agent process, and the connection is maintained by a client-side load balancer running as part of the agent process.

Agents register with the server using the cluster secret portion of the join token, along with a randomly generated node-specific password, which is stored on the agent at /etc/rancher/node/password. The server will store the passwords for individual nodes as Kubernetes secrets, and any subsequent attempts must use the same password. Node password secrets are stored in the kube-system namespace with names using the template <host>.node-password.rke2. These secrets are deleted when the corresponding Kubernetes node is deleted.

If the /etc/rancher/node directory of an agent is removed, the password file should be recreated for the agent prior to startup, or the entry removed from the server or Kubernetes cluster (depending on the RKE2 version).

Deploy NVIDIA GPU operator​

The NVIDIA operator allows administrators of Kubernetes clusters to manage GPUs just like CPUs. It includes everything needed for pods to be able to operate GPUs.

Depending on the underlying OS, some steps need to be fulfilled

The NVIDIA operator cannot automatically install kernel drivers on SLES. NVIDIA drivers must be manually installed on all GPU nodes before deploying the operator in the cluster.

The SLES repositories have the nvidia-open packages, which include the open gpu kernel drivers. If your GPU supports the open drivers, you can check that in this list, you can install the drivers by executing:

sudo zypper install -y nvidia-open

If the GPU does not support gpu kernel drivers, you will need to download the proprietary drivers from an NVIDIA repo. In that case, follow these steps:

sudo zypper addrepo --refresh https://developer.download.nvidia.com/compute/cuda/repos/sles15/x86_64/ cuda-sles15
sudo zypper --gpg-auto-import-keys refresh
sudo zypper install -y –-auto-agree-with-licenses nvidia-gl-G06 nvidia-video-G06 nvidia-compute-utils-G06

Then reboot.

If everything worked correctly, after the reboot, you should see the NVRM and GCC version of the driver when executing the command:

cat /proc/driver/nvidia/version

Finally, create the symlink:

sudo ln -s /sbin/ldconfig /sbin/ldconfig.real

Once the OS is ready and RKE2 is running, install the GPU Operator with the following yaml manifest:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    toolkit:
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock
attention

The NVIDIA operator restarts containerd with a hangup call which restarts RKE2

After one minute approximately, you can make the following checks to verify that everything worked as expected:

1 - Check if the operator detected the driver and GPU correctly:

kubectl get node $NODENAME -o jsonpath='{.metadata.labels}' | jq | grep "nvidia.com"

You should see labels specifying driver and GPU (e.g. nvidia.com/gpu.machine or nvidia.com/cuda.driver.major)

2 - Check if the gpu was added (by nvidia-device-plugin-daemonset) as an allocatable resource in the node:

kubectl get node $NODENAME -o jsonpath='{.status.allocatable}' | jq

You should see "nvidia.com/gpu": followed by the number of gpus in the node

3 - Check that the container runtime binary was installed by the operator (in particular by the nvidia-container-toolkit-daemonset):

ls /usr/local/nvidia/toolkit/nvidia-container-runtime

4 - Verify if containerd config was updated to include the nvidia container runtime:

grep nvidia /var/lib/rancher/rke2/agent/etc/containerd/config.toml

5 - Run a pod to verify that the GPU resource can successfully be scheduled on a pod and the pod can detect it

apiVersion: v1
kind: Pod
metadata:
  name: nbody-gpu-benchmark
  namespace: default
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia
  containers:
  - name: cuda-container
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody
    args: ["nbody", "-gpu", "-benchmark"]
    resources:
      limits:
        nvidia.com/gpu: 1
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: all
    - name: NVIDIA_DRIVER_CAPABILITIES
      value: all
Version Gate

Available as of October 2024 releases: v1.28.15+rke2r1, v1.29.10+rke2r1, v1.30.6+rke2r1, v1.31.2+rke2r1.

RKE2 will now use PATH to find alternative container runtimes, in addition to checking the default paths used by the container runtime packages. In order to use this feature, you must modify the RKE2 service's PATH environment variable to add the directories containing the container runtime binaries.

It's recommended that you modify one of this two environment files:

  • /etc/default/rke2-server # or rke2-agent
  • /etc/sysconfig/rke2-server # or rke2-agent

This example will add the PATH in /etc/default/rke2-server:

echo PATH=$PATH >> /etc/default/rke2-server
attention

PATH changes should be done with care to avoid placing untrusted binaries in the path of services that run as root.