Anatomy of a Next Generation Kubernetes Distribution

Architecture Overview

A single binary is installed and configured on all nodes to participate in the Kubernetes cluster. Once started, it is then able to bootstrap and supervise role-appropriate agents per node while sourcing needed content from the network.

A number of Open Source technologies are in use to make this all work:

• K3s
  • Helm Controller
• K8s
  • API Server
  • Controller Manager
  • Kubelet
  • Scheduler
  • Proxy
• etcd
• runc
• containerd/cri
• CNI: Cilium
• CoreDNS
• Ingress NGINX Controller
• Metrics Server
• Helm

All of these, except the NGINX Ingress Controller, are compiled and statically linked with Go+BoringCrypto.

Kubernetes Cluster Process Lifecycle

Prepare Components

kube-apiserver

Pull the kube-apiserver image, if not present already, and spin up a goroutine to wait for etcd and then write the static pod definition in /var/lib/rancher/rke2/agent/pod-manifests/.

kube-controller-manager

Pull the kube-controller-manager image, if not present already, and spin up a goroutine to wait for kube-apiserver and then write the static pod definition in /var/lib/rancher/rke2/agent/pod-manifests/.

kube-scheduler

Pull the kube-scheduler image, if not present already, and spin up a goroutine to wait for kube-apiserver and then write the static pod definition in /var/lib/rancher/rke2/agent/pod-manifests/.

Start Cluster

Spin up an HTTP server in a goroutine to listen for other cluster servers/agents then initialize/join the cluster.

etcd

Pull the etcd image, if not present already, and spin up a goroutine to wait for the kubelet and then write the static pod definition in /var/lib/rancher/rke2/agent/pod-manifests/.

helm-controller

Spin up the goroutine to start the embedded helm-controller after waiting for kube-apiserver to be ready.

Initialize Agent

The agent process entry point. For server processes the embedded K3s engine invokes this directly.

Container Runtime

containerd

Spawn the containerd process and listen for termination. If containerd exits then the rke2 process will also exit.

Node Agent

kubelet

Spawn and supervise the kubelet process. If kubelet exits then rke2 will attempt to restart it. Once the kubelet is running it will start any available static pods. For servers this means that etcd and kube-apiserver will start, in succession, allowing the remaining components started via static pod to connect to the kube-apiserver and begin their processing.

Server Charts

On server nodes, the helm-controller can now apply to the cluster any charts found in /var/lib/rancher/rke2/server/manifests.

• rke2-canal.yaml or rke2-cilium.yaml (daemonset, bootstrap)
• rke2-coredns.yaml (deployment, bootstrap)
• rke2-ingress-nginx.yaml (deployment)
• rke2-kube-proxy.yaml (daemonset, bootstrap)
• rke2-metrics-server.yaml (deployment)

Daemon Process

The RKE2 process will now run indefinitely until it receives a SIGTERM or SIGKILL or if the containerd process exits.