Kubernetes gave us a way to describe a Deployment and let a controller make it true. It never gave us the same thing for the cluster itself. Creating the cluster was still a shell script, a Terraform module, or a cloud console, and adding a node meant running that script again and hoping it was idempotent. Cluster API (CAPI) closes that gap: a cluster becomes a Kubernetes object, nodes become Machines, and scaling a node pool is a one-line patch on a YAML field.
This article builds one from nothing on a laptop. At the end you will have a management cluster, a workload cluster with one control plane and three workers, and you will have upgraded its Kubernetes version by editing a single field. Everything runs as Docker containers through the Cluster API Docker provider (CAPD), so there is no cloud bill and nothing to clean up afterwards except make clean.
Code for every block below is in the cluster-api-explained-with-capd folder.
What you need
- Docker on your machine. I use Colima on an Apple Silicon Mac with a 4 CPU / 8 GB VM; Docker Desktop works the same way, with one sysctl difference covered below.
kind0.33,clusterctl1.14,kubectl,helm. On macOS:brew install kind clusterctl kubectl helm.- About 6 GB of free memory while the upgrade runs. Idle numbers from this run: the management node takes 1.5 GB, a workload control plane 1 GB, each worker about 0.5 GB.
No previous knowledge of Cluster API is assumed. This is standalone.
Why this way
Cluster API is not the only way to manage clusters. Rancher gives you a UI and imports what you already have; Terraform providers create clusters as resources; Omni does it specifically for Talos. The thing Cluster API adds is that the lifecycle lives inside Kubernetes, reconciled by controllers, with the same primitives you already know. A MachineDeployment behaves like a Deployment: change replicas, and a MachineSet creates or removes Machines. Change the version, and machines are replaced in a rolling fashion. That model is what the rest of this series builds on, so it is worth seeing it work before you see it fail (which is the next article).
Why the Docker provider and not a real cloud for the first contact? Because the provider model is the point. Every provider answers the same three questions, and CAPD lets you watch the answers without a credit card:
| Provider family | Question it answers | In this lab |
|---|---|---|
| Infrastructure | "Give me a machine" | Docker: a kindest/node container per node |
| Bootstrap | "How does that machine become a Kubernetes node" | kubeadm |
| Control plane | "How many control-plane nodes, how are they rolled" | kubeadm |
Swap the first row for Hetzner, OpenStack or KubeVirt and nothing else changes. Swap the second and third for Talos and the Docker provider stops working, for reasons the next article shows from source.
Step 1 — A management cluster that can talk to Docker
CAPD creates workload nodes as containers on the same Docker daemon that runs the management cluster. So the management cluster must see the Docker socket. That is the only special thing about this kind config.
# kind/mgmt.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: capi-mgmt
nodes:
- role: control-plane
extraMounts:
- hostPath: /var/run/docker.sock
containerPath: /var/run/docker.sock
kind create cluster --config kind/mgmt.yaml
Twenty seconds on this machine. Now install the providers. clusterctl init pulls the core controller plus one provider of each family. Versions are pinned to the clusterctl minor so the lab is reproducible; CAPD is a test provider and sits behind a feature flag on purpose.
# clusterctl/init.sh
export CLUSTER_TOPOLOGY=true
clusterctl init \
--core "cluster-api:v1.14.2" \
--bootstrap "kubeadm:v1.14.2" \
--control-plane "kubeadm:v1.14.2" \
--infrastructure "docker:v1.14.2"
Installing cert-manager version="v1.21.1"
Installing provider="cluster-api" version="v1.14.2" targetNamespace="capi-system"
Installing provider="bootstrap-kubeadm" version="v1.14.2" targetNamespace="capi-kubeadm-bootstrap-system"
Installing provider="control-plane-kubeadm" version="v1.14.2" targetNamespace="capi-kubeadm-control-plane-system"
Installing provider="infrastructure-docker" version="v1.14.2" targetNamespace="capd-system"
Your management cluster has been initialized successfully!
Twenty-five seconds. Four controller managers, one namespace each, plus cert-manager for the webhooks:
capd-system capd-controller-manager-f5c788445-hdlf8 1/1 Running
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-5fbf8c6fff-k6w4l 1/1 Running
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-579d6964f-dll79 1/1 Running
capi-system capi-controller-manager-68985f747c-rb44p 1/1 Running
Step 2 — Describe a cluster, apply it, watch it appear
clusterctl generate cluster renders a manifest from the provider's template. The Docker provider ships a development flavor, which uses ClusterClass: a reusable class of cluster, and a short Cluster object that references it. I commit the rendered file so the lab does not depend on GitHub being up.
CLUSTER_TOPOLOGY=true clusterctl generate cluster capi-lab \
--infrastructure docker --flavor development \
--kubernetes-version v1.34.0 \
--control-plane-machine-count=1 --worker-machine-count=1 > workload/cluster.yaml
The file is 411 lines. Seven of the eight objects are the class and its templates; the one you will actually edit is the last:
# workload/cluster.yaml (tail)
kind: Cluster
metadata:
name: capi-lab
spec:
clusterNetwork:
pods: { cidrBlocks: [192.168.0.0/16] }
services: { cidrBlocks: [10.128.0.0/12] }
topology:
classRef:
name: quick-start
version: v1.34.0
controlPlane:
replicas: 1
workers:
machineDeployments:
- class: default-worker
name: md-0
replicas: 1
Note the kinds in the class: DevClusterTemplate and DevMachineTemplate, not DockerMachineTemplate. In v1.14 the Docker provider and the in-memory test provider were folded into one "dev" provider with a backend field. Older tutorials use the Docker* names; the objects behave the same.
kubectl apply -f workload/cluster.yaml
What follows, at twenty-second intervals, is the whole Cluster API model in one screen:
--- t=20s ---
NAME CLUSTERCLASS AVAILABLE CP DESIRED CP AVAILABLE W DESIRED W AVAILABLE PHASE
cluster.cluster.x-k8s.io/capi-lab quick-start False 1 0 1 0 Provisioned
NAME CLUSTER NODE NAME READY PHASE AGE
machine.cluster.x-k8s.io/capi-lab-jhh88-v6r77 capi-lab False Provisioning 8s
--- t=60s ---
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/capi-lab-jhh88 AVAILABLE=True INITIALIZED=true
--- t=80s ---
machine.cluster.x-k8s.io/capi-lab-jhh88-v6r77 capi-lab capi-lab-jhh88-v6r77 False Running
machine.cluster.x-k8s.io/capi-lab-md-0-rdj2m-5tstr-m7q59 capi-lab capi-lab-md-0-rdj2m-5tstr-m7q59 False Running
The control plane Machine is created first; the worker Machine sits in Pending until kubeadm on the control plane has produced a join token; then both go Provisioning → Provisioned → Running. On the Docker side, three containers exist: a haproxy load balancer (the DevCluster) and two kindest/node containers (the DevMachines).
capi-lab-lb kindest/haproxy:v20230606-42a2262b 0.0.0.0:32768->6443/tcp
capi-lab-jhh88-v6r77 kindest/node:v1.34.0 127.0.0.1:32770->6443/tcp
capi-lab-md-0-rdj2m-5tstr-m7q59 kindest/node:v1.34.0
Both machines are Running but READY=False. That is not a bug.
Step 3 — Cluster API does not install a CNI
Cluster API's job ends when kubelet joins the cluster. Networking is the operator's decision, so the nodes stay NotReady until you install one. The Machine condition says so verbatim:
$ kubectl get machine capi-lab-jhh88-v6r77 -o jsonpath='{.status.conditions[?(@.type=="NodeReady")].message}'
* Node.Ready: container runtime network not ready: NetworkReady=false
reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
The workload cluster has its own kubeconfig, stored as a Secret in the management cluster. One wrinkle for Colima and Docker Desktop: the kubeconfig points at the load-balancer container's address, which the host cannot reach. The script rewrites the server to the published host port and, for the lab only, skips TLS verification.
# scripts/cni.sh (essentials)
clusterctl get kubeconfig capi-lab > capi-lab.kubeconfig
kubectl --kubeconfig capi-lab.kubeconfig config set-cluster capi-lab \
--server="https://127.0.0.1:$(docker port capi-lab-lb 6443/tcp | cut -d: -f2)" \
--insecure-skip-tls-verify=true
helm --kubeconfig capi-lab.kubeconfig upgrade --install cilium cilium/cilium \
--version 1.18.2 --namespace kube-system --set ipam.mode=kubernetes
NAME STATUS ROLES AGE VERSION
capi-lab-jhh88-v6r77 NotReady control-plane 2m10s v1.34.0
capi-lab-md-0-rdj2m-5tstr-m7q59 NotReady <none> 113s v1.34.0
NAME STATUS ROLES AGE VERSION
capi-lab-jhh88-v6r77 Ready control-plane 2m41s v1.34.0
capi-lab-md-0-rdj2m-5tstr-m7q59 Ready <none> 2m24s v1.34.0
Thirty seconds after the Cilium install, both nodes are Ready, and the management cluster agrees:
$ clusterctl describe cluster capi-lab --show-conditions=false
NAME REPLICAS AVAILABLE READY STATUS REASON
Cluster default/capi-lab, v1.34.0 2/2 2 2 True Available
├─DevCluster default/capi-lab-dg967 True Ready
├─KubeadmControlPlane default/capi-lab-jhh88, v1.34.0 1/1 1 1 True Available
│ └─Machine default/capi-lab-jhh88-v6r77, v1.34.0 1 1 1 True Ready
└─Workers
└─MachineDeployment default/capi-lab-md-0-rdj2m, v1.34.0 1/1 1 1 True Available
└─Machine default/capi-lab-md-0-rdj2m-5tstr-m7q59, v1.34.0 1 1 1 True Ready
That tree is the CRD map. Cluster owns a DevCluster (infrastructure), a KubeadmControlPlane (control plane) and a MachineDeployment (workers). Each Machine owns a KubeadmConfig (bootstrap) and a DevMachine (infrastructure). The full diagram is in docs/crd-map.md.
Step 4 — Scale like a Deployment
With ClusterClass, you do not touch the MachineDeployment. The topology controller owns it and would revert your change. You edit the Cluster:
kubectl patch cluster capi-lab --type merge \
-p '{"spec":{"topology":{"workers":{"machineDeployments":[{"class":"default-worker","name":"md-0","replicas":3}]}}}}'
--- t=15s ---
machinedeployment.cluster.x-k8s.io/capi-lab-md-0-rdj2m DESIRED=3 CURRENT=3 READY=1 PHASE=Running
machine.cluster.x-k8s.io/capi-lab-md-0-rdj2m-5tstr-9rvbc Provisioning 14s
machine.cluster.x-k8s.io/capi-lab-md-0-rdj2m-5tstr-m7q59 Running 3m35s
machine.cluster.x-k8s.io/capi-lab-md-0-rdj2m-5tstr-rrlng Provisioning 14s
Two new Machines, exactly as a ReplicaSet would create two Pods. And then, on this laptop, nothing happened for five minutes.
What went wrong
The two new machines never left Provisioning. docker ps showed only three capi-lab containers instead of five. The DevMachine condition pointed at systemd:
CGroupsReady: Waiting for cgroups ready failed: multi-user target not reached yet
The CAPD controller had created the containers ("Creating worker machine container with image kindest/node:v1.34.0, mode kind 0.20") and they had died one second later with exit code 255. The container log has the actual error:
Welcome to Debian GNU/Linux 12 (bookworm)!
Failed to create control group inotify object: Too many open files
Failed to allocate manager object: Too many open files
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...
Every kind node is a systemd container, and systemd opens a handful of inotify instances per unit. Colima's VM ships with fs.inotify.max_user_instances = 128. The management node, the control plane and the first worker had used them up; the fourth systemd could not allocate one and exited before kubelet ever started. Cluster API did what it should: after five minutes the MachineHealthCheck that comes with the quick-start class deleted the unhealthy machines and the MachineSet created two more, which died the same way.
The fix is one sysctl inside the VM (kind's own documentation lists the same numbers):
# scripts/colima-limits.sh
colima ssh -- sudo sysctl -w fs.inotify.max_user_instances=512 fs.inotify.max_user_watches=1048576
Then delete the stuck machines rather than waiting another five minutes for the health check:
kubectl delete machine capi-lab-md-0-rdj2m-5tstr-b6ttp capi-lab-md-0-rdj2m-5tstr-fhdj8
t=15s ready: 2/4
t=45s ready: 2/4
t=60s ready: 4/4
NAME STATUS ROLES AGE VERSION
capi-lab-jhh88-v6r77 Ready control-plane 10m v1.34.0
capi-lab-md-0-rdj2m-5tstr-jbdzm Ready <none> 37s v1.34.0
capi-lab-md-0-rdj2m-5tstr-m7q59 Ready <none> 9m45s v1.34.0
capi-lab-md-0-rdj2m-5tstr-vzzl5 Ready <none> 37s v1.34.0
Sixty seconds from delete to Ready. Two things worth keeping from this detour. First, the failure was three layers below Cluster API and the Machine object still surfaced a condition that led to it in two commands. Second, a health check that deletes and recreates nodes is the right default in a cloud and a confusing one on a laptop; if you see machines disappearing, look for a MachineHealthCheck before you look for a bug.
Step 5 — Upgrade Kubernetes by changing one field
In Cluster API, a Kubernetes upgrade is not an in-place operation. Machines are immutable: a new Machine with the new version is created, the old one is drained and deleted. With ClusterClass the whole thing is one field:
kubectl patch cluster capi-lab --type merge -p '{"spec":{"topology":{"version":"v1.34.3"}}}'
Sampling kubectl get machines every twenty seconds (name suffix / phase / version / Ready) gives the order of operations:
t=20s 7kg6l Provisioning v1.34.3 v6r77 Running v1.34.0 jbdzm m7q59 vzzl5 Running v1.34.0
t=60s 7kg6l Running v1.34.3 v6r77 Running v1.34.0 ...
t=80s 7kg6l Running v1.34.3 v6r77 Deleting ... ← control plane first, new joins before old leaves
t=120s 7kg6l Running v1.34.3 nrmzq Provisioned v1.34.3 jbdzm m7q59 vzzl5 Running v1.34.0
t=160s 7kg6l Ready nrmzq Ready jbdzm Deleting ← then workers, one at a time
t=180s 955h9 Provisioning
t=240s 955h9 Ready m7q59 Deleting
t=260s shs2x Provisioning
t=320s shs2x Ready vzzl5 Deleting
The KubeadmControlPlane rolls the control plane first: it creates the new machine, waits for etcd and the API server to be healthy, then deletes the old one. Then the MachineDeployment does the same for workers with maxSurge: 1, one new node up before one old node goes. At no point did the cluster have fewer nodes than before. Total wall-clock on this laptop, including the kindest/node:v1.34.3 image pull: about six minutes.
NAME CLUSTERCLASS AVAILABLE CP DESIRED CP AVAILABLE W DESIRED W AVAILABLE PHASE VERSION
capi-lab quick-start True 1 1 3 3 Provisioned v1.34.3
NAME STATUS ROLES AGE VERSION
capi-lab-jhh88-7kg6l Ready control-plane 12m v1.34.3
capi-lab-md-0-rdj2m-chlzm-955h9 Ready <none> 10m v1.34.3
capi-lab-md-0-rdj2m-chlzm-nrmzq Ready <none> 11m v1.34.3
capi-lab-md-0-rdj2m-chlzm-shs2x Ready <none> 9m5s v1.34.3
Every node name changed. That is the model: you never upgrade a node, you replace it. It is also why the memory note at the top matters; during the roll there is always one extra node alive.
Can you jump versions, or do the Kubernetes rules still apply?
They still apply, and Cluster API enforces them for you. Kubernetes' version skew policy says the API server moves one minor version at a time, the control plane goes before the workers, and a kubelet may be at most three minors behind the API server. Patch versions inside the same minor, like the 1.34.0 → 1.34.3 above, have no such limit.
The Cluster webhook encodes exactly that. From core/webhooks/admission/cluster.go in v1.14.2: a version can only increase, it cannot increase by two minors, and it cannot change while a previous upgrade is still rolling. I tried all three on the lab:
$ kubectl patch cluster capi-lab --type merge -p '{"spec":{"topology":{"version":"v1.36.0"}}}'
The Cluster "capi-lab" is invalid: spec.topology.version: Invalid value: "v1.36.0":
version cannot be increased from "1.34.0" to "1.36.0"
$ kubectl patch cluster capi-lab --type merge -p '{"spec":{"topology":{"version":"v1.33.4"}}}'
The Cluster "capi-lab" is invalid: spec.topology.version: Invalid value: "v1.33.4":
version cannot be decreased from "1.34.0" to "1.33.4"
$ kubectl patch cluster capi-lab --type merge -p '{"spec":{"topology":{"version":"v1.35.0"}}}'
cluster.cluster.x-k8s.io/capi-lab patched
$ kubectl patch cluster capi-lab --type merge -p '{"spec":{"topology":{"version":"v1.36.0"}}}'
The Cluster "capi-lab" is invalid: spec.topology.version: Invalid value: "v1.36.0":
version cannot be changed: [control plane is still completing a previous upgrade,
there are still MachineDeployments completing a previous upgrade: [capi-lab-md-0-hmqmq]]
So 1.34 → 1.36 is two patches, and the second one is only accepted once every machine reports 1.35. The ordering (control plane first, then MachineDeployments) is also handled by the topology controller; you do not sequence it yourself. Two things the webhook does not check for you: whether a kindest/node (or, on a real provider, a machine image) exists for the target version, and whether your add-ons support it. The upgrade will start and the new control-plane machine will sit in Provisioning if the image is missing. There is an escape hatch, the unsafe.topology.cluster.x-k8s.io/disable-update-version-check annotation, which turns the checks off; the word "unsafe" in it is there for a reason.
If you want to dig further, these are the pages I keep open when planning an upgrade:
- Kubernetes version skew policy: which component may be how far from which.
- Kubernetes upgrading kubeadm clusters: what kubeadm does on each node; Cluster API runs this for you.
- Cluster API upgrading a workload cluster: the CAPI side, including the ClusterClass flow used here.
- Cluster API version and provider support: which CAPI release supports which Kubernetes versions for management and workload clusters.
Verify
kubectl get cluster capi-lab # AVAILABLE=True, VERSION=v1.34.3
kubectl --kubeconfig capi-lab.kubeconfig get nodes # 4 nodes Ready, all v1.34.3
docker ps --format '{{.Names}}' | grep -c capi-lab # 5: lb + 1 cp + 3 workers
make clean # deletes the Cluster (CAPI removes containers), then kind
make clean deletes the Cluster object first and waits. Cluster API's finalizers remove the DevMachines, so the containers disappear before kind is torn down. If you delete the kind cluster first, the workload containers are orphaned and you clean them by hand.
Wrap-up
What Cluster API gives you is simple to state: the cluster is a Kubernetes object like everything else. Scaling is a replicas change, an upgrade is a version change, and the version rules are enforced by a webhook instead of remembered by you. With the Docker provider you can try all of it on a laptop in an afternoon.
In this series
This is the first article of Bare Metal to Tenant Clusters. Next: Talos Linux, explained by running it — the OS the rest of the series uses, and the reason the Docker provider you just used will not be enough.
Code for this post: https://github.com/miraccan00/blog-wiki/tree/main/cluster-api-explained-with-capd