Every Kubernetes node I have ever debugged over SSH had the same three problems: someone had changed something by hand, nobody knew what, and the node could not be rebuilt from a file. Talos Linux removes the tool that causes all three. There is no SSH, no shell, no package manager, and the whole node is described by one YAML document that you can only change through an authenticated gRPC API.
This article runs Talos on a laptop, as two Docker containers, and does the four things you do to a node in real life: create it, change its configuration, build a custom image, upgrade it. At the end you will have a working cluster, a machine-config patch you applied without a reboot, an Image Factory image with the extensions Longhorn needs, and a Kubernetes upgrade from 1.35 to 1.36 that took about three minutes. You will also know which of those the Docker version of Talos cannot do, and why the newest Talos would not even boot on my machine.
Code for every block below is in the talos-linux-explained folder.
What you need
- Docker on your machine. I use Colima 0.8 on an Apple Silicon Mac, 4 CPU / 8 GB VM. Docker Desktop and Linux work too; two things differ and both are in "What went wrong".
talosctl1.13,kubectl, GNU make. On macOS:brew install talosctl kubectl make(Homebrew installs it asgmake).- Versions in this run: Talos image v1.13.10, talosctl v1.13.5, Kubernetes 1.35.8 upgraded to 1.36.4. About 1 GB of RAM while idle, 2 GB during the upgrade.
No previous knowledge of Talos is assumed. This is standalone; the previous article is not required.
Why this way
A Kubernetes node needs a kernel, a container runtime, a kubelet, and a way to receive its configuration. Ubuntu with kubeadm gives you all of that plus 60,000 packages, a login shell, cloud-init, unattended-upgrades, and every way of drifting that a general-purpose OS has. RKE2 and k3s narrow the Kubernetes part but keep the OS underneath open. Talos takes the opposite position: the OS is the Kubernetes node and nothing else. What you get:
| Property | What it means on a node |
|---|---|
| Immutable | Root filesystem is read-only, squashfs, replaced whole on upgrade. Nothing to apt install. |
| API-driven | Every operation is a gRPC call on port 50000, mTLS, with roles (os:admin, os:operator, os:reader). |
| One machine config | A single YAML document holds kernel args, network, kubelet flags, sysctls, registries, certificates. |
| Minimal | 12 binaries in /usr/bin, most of them nftables and iptables. No shell, no ls, no cat. |
Why run it in Docker for the first contact? Because the property that matters, "the OS is an API", is fully present in container mode: same machined, same apid, same machine config, same talosctl. What is missing is a disk and a kernel of its own, so OS upgrades and disk encryption are out; we will see the API refuse them, which is itself instructive.
Step 1 — Boot a cluster from one command
talosctl cluster create docker generates PKI, writes a machine config for each node, starts one container per node and bootstraps etcd. Nothing touches ~/.talos or ~/.kube; both configs land under ./.lab. The only settings that must exist at generation time are the ones both roles have to agree on and the control plane's own.
# config/lab.patch.yaml — every node
cluster:
allowSchedulingOnControlPlanes: false
machine:
network:
extraHostEntries:
- ip: 10.5.0.2
aliases:
- registry.lab.internal
# config/controlplane.patch.yaml — control plane only
cluster:
apiServer:
extraArgs:
event-ttl: 2h
machine:
nodeLabels:
topology.kubernetes.io/zone: lab-a
# cluster/create.sh (essentials)
talosctl cluster create docker \
--name "$CLUSTER" --state "$LAB/state" \
--image "ghcr.io/siderolabs/talos:${TALOS_VERSION}" \
--kubernetes-version "$K8S_VERSION" \
--workers 1 \
--config-patch @config/lab.patch.yaml \
--config-patch-controlplanes @config/controlplane.patch.yaml \
--talosconfig-destination "$TALOSCONFIG"
$ make up
generating PKI and tokens
creating network talos-lab
creating controlplane nodes
creating worker nodes
waiting for Talos API (to bootstrap the cluster)
bootstrapping cluster
waiting for etcd to be healthy: OK
waiting for apid to be ready: OK
waiting for kubelet to be healthy: OK
waiting for all k8s nodes to report ready: OK
waiting for coredns to report ready: OK
109 seconds from make up to CoreDNS ready on this laptop, with the 236 MB image already pulled. Two containers, one per node; the control plane publishes the Kubernetes API (6443) and the Talos API (50000) on random host ports:
$ docker ps --format '{{.Names}}\t{{.Image}}\t{{.Ports}}'
talos-lab-worker-1 ghcr.io/siderolabs/talos:v1.13.10
talos-lab-controlplane-1 ghcr.io/siderolabs/talos:v1.13.10 0.0.0.0:52967->6443/tcp, 0.0.0.0:52968->50000/tcp
$ talosctl get members
NODE TYPE ID HOSTNAME MACHINE TYPE OS ADDRESSES
10.5.0.2 Member talos-lab-controlplane-1 talos-lab-controlplane-1 controlplane Talos (v1.13.10) ["10.5.0.2"]
10.5.0.2 Member talos-lab-worker-1 talos-lab-worker-1 worker Talos (v1.13.10) ["10.5.0.3"]
$ kubectl get nodes -o wide
NAME STATUS ROLES VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
talos-lab-controlplane-1 Ready control-plane v1.35.8 10.5.0.2 Talos (v1.13.10) 6.8.0-50-generic containerd://2.2.7
talos-lab-worker-1 Ready <none> v1.35.8 10.5.0.3 Talos (v1.13.10) 6.8.0-50-generic containerd://2.2.7
Note the kernel column. In container mode Talos runs on the host's kernel, here Colima's 6.8. That single fact is behind the first entry in "What went wrong".
Two containers, and the reflex is to docker exec into one:
$ docker exec talos-lab-worker-1 sh
OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown
$ docker exec talos-lab-worker-1 ls
OCI runtime exec failed: exec failed: unable to start container process: exec: "ls": executable file not found in $PATH: unknown
That is not a hardened image or a disabled feature. The binaries do not exist. The image's entrypoint is /sbin/init, which is machined, and everything else on the node is either a Go service Talos ships or a container that kubelet started.
Step 2 — No shell, so what do you type
Every operation goes through talosctl, which speaks gRPC to apid on port 50000. Your identity is the client certificate in talosconfig; the role is in the certificate, so "who can do what to this node" is answered by the PKI, not by /etc/sudoers.
$ talosctl config info
Current context: talos-lab
Endpoints: 127.0.0.1:52478
Roles: os:admin
Certificate expires: 1 year from now (2027-09-18)
The full SSH-habit-to-talosctl table is in docs/api-vs-ssh.md; the ones I use daily:
| SSH habit | talosctl |
|---|---|
systemctl status |
talosctl services |
journalctl -u kubelet -f |
talosctl logs kubelet -f |
dmesg |
talosctl dmesg |
cat /etc/hosts |
talosctl read /etc/hosts |
df -h |
talosctl mounts |
ss -tulpn |
talosctl netstat -l |
ps aux |
talosctl processes |
crictl ps |
talosctl containers -k |
top |
talosctl dashboard |
vim /etc/kubernetes/... |
there is no file: talosctl patch mc |
talosctl services is the one to learn first. Talos has seven system services on a control plane and five on a worker, and every one of them reports state and health through the API:
$ talosctl services
NODE SERVICE STATE HEALTH LAST CHANGE LAST EVENT
10.5.0.2 apid Running OK 1m48s ago Health check successful
10.5.0.2 containerd Running OK 1m49s ago Health check successful
10.5.0.2 cri Running OK 1m48s ago Health check successful
10.5.0.2 etcd Running OK 1m36s ago Health check successful
10.5.0.2 kubelet Running OK 1m40s ago Health check successful
10.5.0.2 machined Running OK 1m49s ago Health check successful
10.5.0.2 trustd Running OK 1m48s ago Health check successful
Two containerd instances, by the way: one in the system namespace for Talos's own services, one for the CRI. That is why talosctl containers shows apid and kubelet, and talosctl containers -k shows pods.
And the extraHostEntries from the patch, read back through the API, because there is no other way to read it:
$ talosctl read /etc/hosts --nodes 10.5.0.3
127.0.0.1 localhost
10.5.0.3 talos-lab-worker-1
10.5.0.2 registry.lab.internal
Step 3 — One machine config, patched live
The machine config is the whole node. talosctl get mc -o yaml on the worker returns a 27 KB document: PKI, tokens, kubelet settings, network, registries, sysctls, kernel modules, everything. There is no other source of truth; there is no file on the node you could have edited instead.
Changing it is a patch. The worker patch adds a kubelet flag, two sysctls and a node label, and I always dry-run first, because the API answers the question SSH never could: will this reboot the node?
# config/worker.patch.yaml
machine:
kubelet:
extraArgs:
max-pods: "200"
sysctls:
net.core.somaxconn: "65535"
fs.inotify.max_user_instances: "8192"
nodeLabels:
node.lab/pool: general
# scripts/patch.sh (essentials)
talosctl patch mc --nodes 10.5.0.3 --patch @config/worker.patch.yaml --dry-run
talosctl patch mc --nodes 10.5.0.3 --patch @config/worker.patch.yaml
$ talosctl patch mc --nodes 10.5.0.3 --patch @config/worker.patch.yaml --dry-run
Dry run summary:
Applied configuration without a reboot (skipped in dry-run).
Config diff:
kubelet:
image: ghcr.io/siderolabs/kubelet:v1.35.8
+ extraArgs:
+ max-pods: "200"
...
+ sysctls:
+ fs.inotify.max_user_instances: "8192"
+ net.core.somaxconn: "65535"
...
+ nodeLabels:
+ node.lab/pool: general
"Without a reboot", and the diff against the config the node is actually running. Now for real. Before the patch, kubelet was PID 148 and somaxconn was 4096:
$ talosctl patch mc --nodes 10.5.0.3 --patch @config/worker.patch.yaml
patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.3
Applied configuration without a reboot
$ talosctl get kubeletconfig --nodes 10.5.0.3 -o jsonpath='{.spec.extraArgs}'
{ "max-pods": { "values": [ "200" ] } }
$ talosctl read /proc/sys/net/core/somaxconn --nodes 10.5.0.3
65535
$ kubectl get nodes -l node.lab/pool=general
NAME STATUS ROLES AGE VERSION
talos-lab-worker-1 Ready <none> 33s v1.35.8
$ talosctl services --nodes 10.5.0.3
NODE SERVICE STATE HEALTH LAST CHANGE LAST EVENT
10.5.0.3 kubelet Running ? 0s ago Started task kubelet (PID 1022) for container kubelet
$ talosctl processes --nodes 10.5.0.3 | grep -o 'max-pods=[0-9]*'
max-pods=200
Kubelet was restarted with the new argument, the sysctl is live, and the label is on the Node object. No reboot, no drain, no systemctl daemon-reload. That is what "the config is the node" buys you: the controller that owns each part of the config knows how to apply a change to it.
Which changes need a reboot, then? Talos documents the fields that apply live: .cluster, .machine.network, .machine.kubelet, .machine.kernel, .machine.time, labels, taints, and more. I dry-ran fifteen patches against the worker to see the answer for myself. Kernel modules, sysctls, registry mirrors, kubelet extra mounts, NTP servers, KubeSpan: all "without a reboot". The one that said otherwise was machine.env, the environment for system services, which is read once at boot:
# config/needs-reboot.patch.yaml
machine:
env:
GRPC_GO_LOG_SEVERITY_LEVEL: info
$ talosctl patch mc --nodes 10.5.0.3 --patch @config/needs-reboot.patch.yaml --dry-run
Dry run summary:
Applied configuration with a reboot (skipped in dry-run).
Config diff:
+ env:
+ GRPC_GO_LOG_SEVERITY_LEVEL: info
In the default --mode=auto, that patch would apply and reboot the node. With --mode=no-reboot it is refused; with --mode=staged it is written and applied on the next reboot you schedule. In Talos 1.14 the explicit reboot mode is gone and the reboot is a separate step, which is the right direction: the API tells you what a change costs and you decide when to pay it.
Step 4 — The image is a build artifact
On real hardware the second thing you hit is: my CSI needs iscsid on the host, and there is no package manager. Talos's answer is Image Factory: you describe the image, it builds it, and the description is a five-line YAML you commit next to your Terraform.
# factory/schematic.yaml
customization:
systemExtensions:
officialExtensions:
- siderolabs/iscsi-tools
- siderolabs/util-linux-tools
Those two are what Longhorn, and any iSCSI-based CSI, need on the node: iscsid and blkid/nsenter/fstrim. POST the file, get a deterministic id; the same YAML always yields the same id, so the id is the image's content hash and belongs in your infra repo:
# factory/schematic-id.sh (essentials)
curl -s -X POST https://factory.talos.dev/schematics \
-H 'Content-Type: application/yaml' --data-binary @factory/schematic.yaml
{"id":"613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245", ...}
schematic id : 613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245
installer : factory.talos.dev/metal-installer/613e1592…961245:v1.13.10 # talosctl upgrade --image ...
hcloud : factory.talos.dev/hcloud-installer/613e1592…961245:v1.13.10 # Hetzner Cloud
metal iso : https://factory.talos.dev/image/613e1592…961245/v1.13.10/metal-arm64.iso
No account, no auth, and the installer reference is what you pass to talosctl upgrade. That is the whole story of "installing a package" on Talos: change the schematic, get a new id, upgrade the node to the new image. In the Hetzner article later in this series, this id is a Terraform variable and nothing else about the image exists anywhere.
Step 5 — Two upgrades, two commands
Talos separates upgrading Kubernetes from upgrading the OS, and it is worth keeping them separate in your head too.
talosctl upgrade-k8s rewrites the Kubernetes image versions in every node's machine config and lets the controllers roll the static pods and kubelets, control plane first. No reboot, and it works in Docker:
# scripts/upgrade.sh (essentials)
talosctl upgrade-k8s --nodes 10.5.0.2 --to 1.36.4 --endpoint 127.0.0.1:52477
$ make upgrade
automatically detected the lowest Kubernetes version 1.35.8
discovered controlplane nodes ["10.5.0.2"]
discovered worker nodes ["10.5.0.3"]
> "10.5.0.2": Talos version 1.13.10 is compatible with Kubernetes version 1.36.4
> "10.5.0.3": Talos version 1.13.10 is compatible with Kubernetes version 1.36.4
checking for removed Kubernetes component flags
checking for removed Kubernetes API resource versions
> "10.5.0.2": pre-pulling registry.k8s.io/kube-apiserver:v1.36.4
> "10.5.0.2": pre-pulling ghcr.io/siderolabs/kubelet:v1.36.4
> "10.5.0.3": pre-pulling ghcr.io/siderolabs/kubelet:v1.36.4
updating "kube-apiserver" to version "1.36.4"
> "10.5.0.2": machine configuration patched
> "10.5.0.2": waiting for kube-apiserver pod update
< "10.5.0.2": successfully updated
updating "kube-controller-manager" to version "1.36.4"
< "10.5.0.2": successfully updated
updating "kube-scheduler" to version "1.36.4"
< "10.5.0.2": successfully updated
updating kube-proxy to version "1.36.4"
updating kubelet to version "1.36.4"
> "10.5.0.2": waiting for kubelet restart
< "10.5.0.2": successfully updated
> "10.5.0.3": waiting for kubelet restart
< "10.5.0.3": successfully updated
updating manifests
< configured DaemonSet/kube-system/kube-proxy
waiting for kubernetes objects to be fully reconciled
done
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
talos-lab-controlplane-1 Ready control-plane 22h v1.36.4
talos-lab-worker-1 Ready <none> 22h v1.36.4
Three minutes and eleven seconds, images already pulled. Read the order: it checks compatibility and removed flags first, pre-pulls every image so no node is left half-way waiting on a registry, then rolls the control plane one component at a time and the kubelets one node at a time, and only then touches the manifests. Each step is a machine config patch; talosctl get staticpodstatus shows the static pod versions ticking up. Nothing rebooted. The version skew rules still apply: Talos 1.13 supports Kubernetes 1.31 through 1.36, and upgrade-k8s checks the lowest version in the cluster before it starts. To reach 1.37 I would first upgrade Talos itself to 1.14.
talosctl upgrade is the other one. It pulls an installer image, writes it to the other of the two system partitions, and reboots into it; if the new version does not come up, the previous one is still on disk. In container mode there is no disk, and the API says so, after pulling the image:
$ talosctl upgrade --nodes 10.5.0.2 --image factory.talos.dev/metal-installer/613e1592…961245:v1.13.10
10.5.0.2: pulled image factory.talos.dev/metal-installer/613e1592…961245@sha256:5266bb0d…
error during upgrade: error from node 10.5.0.2: rpc error: code = FailedPrecondition desc = method is not supported in container mode
FailedPrecondition, not a stack trace. reset and disk encryption are refused the same way. The script checks talosctl get platformmetadata (platform: container) and skips the loop over nodes; on metal it runs talosctl upgrade --wait one node at a time.
What went wrong
Five things, and the first one cost the most time.
1. Talos 1.14 does not start on a 6.8 kernel
My first attempt used the current release, v1.14.1. talosctl cluster create printed waiting for Talos API (to bootstrap the cluster) and never moved. The container was up, so the container log was the only place to look:
$ docker logs talos-lab-controlplane-1
[talos] service[machined](Preparing): Creating service runner
[talos] controller runtime goroutine error: fatal controller runtime error: failed to set up /etc overlay:
failed to compose writable /etc overlay: openfs failed: failed to create root filesystem:
FSCONFIG_SET_FD failed: bad file descriptor: key="lowerdir+" fd=4
[talos] service[apid](Waiting): Waiting for api certificates, config to be ready
machined died before the config was loaded, so apid waited forever and so did talosctl. The cause is in Talos PR #13558, merged in June 2026 for 1.14: /etc became a writable overlay whose lower layers are passed to the kernel as file descriptors, fsconfig(FSCONFIG_SET_FD, "lowerdir+"). The kernel's overlayfs documentation has that feature "since kernel v6.13". Colima 0.8's Ubuntu VM runs 6.8, which knows lowerdir+ as a path but not as a descriptor, and returns EBADF. On metal this cannot happen, Talos ships its own 6.18 kernel; in Docker, Talos runs on your kernel. The lab pins v1.13.10 for that reason; on a recent Docker Desktop or a Linux host with 6.13+, TALOS_VERSION=v1.14.1 make up works.
2. talosctl ignores docker context
failed to connect to the docker API at unix:///var/run/docker.sock: dial unix /var/run/docker.sock: connect: no such file or directory
The docker CLI finds Colima through a context; talosctl dials the socket path directly. The Talos docs suggest a symlink; I prefer not to touch /var/run. The Makefile reads the socket from the context and exports it:
# Makefile
export DOCKER_HOST ?= $(shell docker context inspect -f '{{.Endpoints.docker.Host}}' 2>/dev/null)
3. Flannel needs br_netfilter, and Talos cannot load it for you
With 1.13.10 the cluster bootstrapped, but make up stalled at waiting for coredns to report ready. Both CoreDNS pods were ContainerCreating and both kube-flannel pods were in CrashLoopBackOff:
$ talosctl logs kubelet --nodes 10.5.0.3 | tail -1
failed to setup network for sandbox: plugin type="flannel" failed (add):
failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory
$ kubectl logs -n kube-system -l k8s-app=flannel --tail=1
E0918 14:44:09.535929 main.go:289] Failed to check br_netfilter: stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory
On a real Talos node the kernel module is there. In Docker, Talos is a privileged container on a kernel it did not ship, and it will not modprobe the host. The fix is on the VM, and once it is loaded flannel recovers on its own:
colima ssh -- sudo modprobe br_netfilter
Sixty seconds later CoreDNS was Running and make up finished. The Talos Docker page mentions this one; I had not read that far.
4. The kubeconfig points at an address your Mac cannot reach
$ kubectl get nodes
Unable to connect to the server: dial tcp 10.5.0.2:6443: i/o timeout
talosctl kubeconfig writes the cluster's control-plane endpoint, https://10.5.0.2:6443, which is the container's address on the Docker network inside the Colima VM. Docker published that port on 127.0.0.1:52477. On Linux both work; on macOS only the published one does. The same applies to talosctl upgrade-k8s, which talks to the API server itself and has an --endpoint flag for exactly this. The create script rewrites the server, and because talosctl cluster create puts 127.0.0.1 in the API server's certificate, no --insecure-skip-tls-verify is needed:
# cluster/create.sh (tail)
talosctl kubeconfig "$KUBECONFIG" --force
PORT=$(docker port "${CLUSTER}-controlplane-1" 6443/tcp | head -1 | cut -d: -f2)
kubectl config set-cluster "$CLUSTER" --server="https://127.0.0.1:${PORT}"
5. "successfully updated" is about the config, not the pod
I sampled kubectl get pods -n kube-system every ten seconds during the upgrade. The API server was unreachable for about thirty seconds while its static pod was replaced, which is expected with one control-plane node. Less expected: right after upgrade-k8s printed kube-scheduler: successfully updated, the scheduler restarted three times and was not ready for close to a minute, and the controller-manager restarted again during the kubelet roll. The upgrade never noticed, because what it waits for is the config-version annotation on the static pod, not readiness. Sidero closed #14227, which describes exactly this, as "not planned", and #14152 explains why the API server restarts a second time during the scheduler step: the scheduler's config version is part of the API server's annotation too.
The first time I ran this upgrade the same scheduler stayed in CrashLoopBackOff for seven minutes, every attempt getting Retry-After from the API server through KubePrism and giving up after client-go's ten retries, before recovering on its own. I could not reproduce it on the clean run above, and I had been running diagnostics against the node in parallel, so I will not blame Talos for it. The lesson stands either way: after upgrade-k8s, run talosctl get staticpodstatus and talosctl health before you call it done, and do not apply other config patches while it runs, since every cluster.* change restarts a static pod.
Verify
talosctl get members # 2 members, Talos (v1.13.10)
talosctl services --nodes 10.5.0.3 # 5 services Running / OK
kubectl get nodes # 2 Ready, both v1.36.4
kubectl get nodes -l node.lab/pool=general # the worker, labelled by the live patch
talosctl read /proc/sys/net/core/somaxconn --nodes 10.5.0.3 # 65535
docker exec talos-lab-worker-1 sh # executable file not found
make down # containers, network and ./.lab gone
NAME STATUS ROLES AGE VERSION
talos-lab-controlplane-1 Ready control-plane 10m v1.36.4
talos-lab-worker-1 Ready <none> 10m v1.36.4
$ talosctl get kubeletconfig --nodes 10.5.0.3 -o jsonpath='{.spec.image}'
ghcr.io/siderolabs/kubelet:v1.36.4
make down runs talosctl cluster destroy, which removes the two containers and the network, then deletes ./.lab. Nothing else on the machine was touched, which is easy to say for a lab that never had a shell.
When Talos, and when not
Talos is the right default when the node exists only to run Kubernetes and you want the node to be a function of a file: bare metal, Hetzner, KubeVirt VMs, anything you rebuild rather than repair. It is the wrong choice when the node must also run something that is not a pod: an agent your security team installs with apt, a vendor tool that expects /usr/local/bin, an NFS server, a developer who needs to tcpdump on the box (there is talosctl pcap, but they will not thank you). Ubuntu with kubeadm or RKE2 stays the better fit for those, and for teams whose runbooks are written in bash over SSH and who cannot rewrite them this quarter. The honest test is: if a node broke right now, would you fix it or replace it? Talos is for the second answer.
In this series
Previous: Cluster API, Explained by Building One: kind, the Docker Provider and a Workload Cluster in 15 Minutes · Next: Why Cluster API's Docker Provider Can't Bootstrap Talos. You have now seen both halves of that title: CAPD expects a node with systemd, a shell and cloud-init to run scripts in; Talos has none of them, on purpose. Later in the series the same Image Factory id from Step 4 boots 17 Hetzner nodes from Terraform with zero SSH.
Code for this post: https://github.com/miraccan00/blog-wiki/tree/main/talos-linux-explained