Most teams end up with more than one Kubernetes cluster, and then the shared services arrive: auth, a registry cache, a feature flag service. Each cluster runs its own copy. When one copy goes down, every app in that cluster loses the service, while the same service sits idle in the cluster next door. In this post I look at what Cilium Cluster Mesh is, what it solves and what you gain, and then I show the lab where I scaled a service to 0 in one cluster and its clients kept getting answers from the other. It broke in a few places; those are at the end, with the real messages.
What is Cilium Cluster Mesh?
Cilium Cluster Mesh connects the networks of several Kubernetes clusters that all run Cilium as their network plugin (CNI).
Cilium already knows every pod, every service and every identity in its own cluster, because it is the network. Cluster Mesh shares that knowledge between clusters. After that:
- A pod in one cluster can reach a pod in another cluster directly, node to node.
- A Service with the same name in several clusters can become one global service with backends from all of them.
- A network policy can allow or deny traffic by cluster name.
There is no central control plane. Each cluster runs a small clustermesh-apiserver, and the Cilium agents of the other clusters read from it.
What problem does it solve?
One copy per cluster, and no way to borrow. With four clusters there are four copies of every shared service. When one dies, that cluster has nothing, and the other three copies can't help, because a cluster only sees its own pods.
Extra parts just to connect clusters. The usual answer is a VPN between the clusters, a global load balancer and a DNS record per service. That is three more components, all outside Kubernetes, to configure for every new cluster.
Policies stop at the cluster border. If traffic crosses from one cluster to another, a normal network policy can't say "allow this, but only from that cluster".
What you gain with Cluster Mesh
| Gain | What it means in practice | In the lab |
|---|---|---|
| Clusters see each other | One command connects both sides; agents add the other cluster's nodes | All 2 nodes are connected to all clusters, both directions |
| Global services | One annotation; same name and namespace in each cluster | 4 backends in the service table, 2 per cluster |
| Failover without DNS | affinity=local: stay local, cross over only when local pods are gone |
Scaled to 0: 3 of 3 answers from the other cluster, 0 failed |
| Policy by cluster | io.cilium.k8s.policy.cluster in a CiliumNetworkPolicy |
x-wing reached rebel-base only in mesh-2 |
| No new datapath | No gateway, no proxy, no NAT between clusters | Packets go straight from node to node |
Before deciding, know what you don't get:
| Need | Cluster Mesh |
|---|---|
| Clusters without Cilium | They can't join. Every cluster must run Cilium. |
| Overlapping pod networks | Not allowed. Pod and service CIDRs must be different in every cluster. |
| A cluster id you can change later | cluster.id (1–255) is set at install; changing it means restarting every workload. |
| Network connectivity | You bring it: nodes in all clusters must reach each other (same network, VPN or routed private network). |
How it works
- Sharing: each cluster's
clustermesh-apiserverholds a small etcd with that cluster's nodes, identities, endpoints and global services. The other cluster's agents connect to it over mTLS, so both clusters must share one certificate authority (CA). - Datapath: pod traffic already flows in a tunnel (VXLAN) between nodes. The other cluster's nodes are added to the list of tunnel endpoints, so a packet for a pod over there goes straight to its node.
- Services and policy: a Service annotated
service.cilium.io/global: "true"gets the other cluster's pods as backends. Identities carry the cluster name, which a policy can select.
The lab: what I built
Everything runs on kind on a MacBook (Colima, 4 CPU / 8 GB): two clusters, mesh-1 and mesh-2, two nodes each, Kubernetes 1.37 and Cilium 1.20.2. The pod networks don't overlap (10.244.0.0/16 and 10.245.0.0/16), and the cilium-ca secret of mesh-1 is copied to mesh-2 before Cilium is installed there, so both share one CA. The demo is Cilium's own: rebel-base answers with the name of its cluster, x-wing in mesh-1 calls it. The whole lab took about 8 minutes, most of it creating the two clusters.
Step 1: Connect the two clusters
One command, cilium clustermesh connect, configured both sides. It took 110 seconds, almost all of it waiting for the clustermesh-apiserver pods to start (more on that in the errors). After that, each cluster saw the other's two nodes.
What changed at the agent level: an agent in mesh-1 now lists mesh-2's nodes with Source: clustermesh. A packet for a 10.245.x pod goes straight to 172.20.0.5 or .6. That is the whole datapath side of the mesh.
Step 2: One service, two clusters
The same rebel-base Service goes to both clusters with the annotation service.cilium.io/global: "true". The rule is simple: same name, same namespace, the annotation in both. x-wing in mesh-1 called the name rebel-base six times: 4 answers came from Cluster-1 and 2 from Cluster-2. DNS didn't change, the client didn't change. The service table of the agent simply had four backends, two from each cluster.
Step 3: Failover with affinity=local
A random split is usually not what you want. Traffic should stay in its own cluster and cross over only when the local pods are gone. That is a second annotation, service.cilium.io/affinity: local. Then I scaled rebel-base in mesh-1 to 0.
3 of 3 answers from the other cluster while the local copy was gone, 3 of 3 back home when it returned, 0 failed requests, 11 seconds for the whole round trip. This is the fix for "one copy dies, the others sit idle", with no DNS change.
Step 4: A policy that names the other cluster
Network policies still work per cluster: a CiliumNetworkPolicy that selects name: rebel-base only selects the pods in its own cluster. To reach the other cluster you have to name it with the label io.cilium.k8s.policy.cluster. I wrote a policy that lets x-wing reach rebel-base only in mesh-2, and removed the affinity, so the Service again spread requests over all four backends.
5 of 8 requests timed out. The Service still picked any of the four backends, and the policy blocked the two local ones. Hubble showed why: the SYN from x-wing to a local rebel-base was EGRESS DENIED. A policy selects pods, not services. With operator: Exists on the cluster label ("rebel-base in any cluster"), all 8 answered, from both clusters, in 3 to 20 ms.
Hubble also shows which cluster each flow went to, because flows carry the cluster name on both sides.
Final check
Where it broke: 3 real errors
1. clustermesh-apiserver stuck at Init:0/1: configmap "clustermesh-remote-users" not found
Right after installing Cilium, the clustermesh-apiserver pod never started, and cilium status --wait gave up after 5 minutes (361 seconds in my first run). The event says it is waiting for a ConfigMap called clustermesh-remote-users. Nothing in the Helm chart creates it: cilium clustermesh connect does. So the order is: wait for the Cilium agents only, run connect, and then the apiserver starts on its own. The lab's install script waits for the agents and leaves the apiserver to connect.
The FailedScheduling: untolerated taint line above it is harmless: for the first minute the nodes are tainted not-ready until Cilium is up.
2. Right after connect: two apiserver pods and "0 of 2 replicas"
For about a minute after connect, each cluster shows two clustermesh-apiserver pods and connect keeps printing only 0 of 2 replicas are available. connect runs a Helm upgrade that changes the pod, so a new pod starts next to the old stuck one. The new one becomes 3/3, the old one is removed. If you look at cilium status in that minute you see errors and nothing is actually wrong. Wait for Deployment clustermesh-apiserver is ready.
3. Failover got stuck on policy
This is Step 4 again, but it is the error people hit in production. You set up the global service and failover, test it without policies, and it works. Then you apply your usual default-deny policies (see the network policy post) and failover breaks, because the rule that allows rebel-base only selects the local rebel-base. The fix is to select the cluster label with Exists, so a new cluster doesn't need a policy change.
What I did not hit, because the lab files prevent it: overlapping pod CIDRs (both kind configs set podSubnet), a missing cluster.id (it's in each cluster's values file), and a CA mismatch (the script copies cilium-ca before the second install and checks same CA). The documented symptoms are cluster ID must be set to a value between 1 and 255 and x509: certificate signed by unknown authority.
When to use it, and when not
| Situation | Choice |
|---|---|
| All your clusters run Cilium, and you want shared services with failover between them | Cluster Mesh. No new components in the datapath. |
| Some clusters run another CNI | Submariner (CNI-independent, gateway nodes) or a service mesh with multi-cluster support. |
| You already run Istio and need mTLS and retries across clusters | Istio multi-cluster. Cluster Mesh doesn't add application-level mTLS. |
| Clusters in different clouds or data centers with no private link | Build the network first (VPN or routed link). Cluster Mesh needs node-to-node reachability. |
| A cluster should use shared services but not offer its own | Cluster Mesh with service.cilium.io/shared: "false" on that cluster's Service. |
Summary
- Cilium Cluster Mesh connects the networks of clusters that run Cilium, without a gateway or a central control plane.
- One
connectcommand configured both sides; each agent then listed the other cluster's nodes. - One annotation made a Service global, with backends from both clusters.
- With
affinity=local, a service scaled to 0 in one cluster kept answering from the other: 3 of 3, 0 failed. - Policies still select pods per cluster: 5 of 8 requests timed out until the policy allowed any cluster with
Exists.
Sources: Cilium docs Setting up Cluster Mesh, Load-balancing & Service Discovery (global services, affinity), Multi-cluster network policy; Cilium 1.20 Helm reference (clustermesh.*); the Cilium rebel-base / x-wing cluster mesh example.
Next in the series
This is the sixth post in the Cilium End to End series. Next: What Is Tetragon? What It Solves, What You Gain, and a Kubernetes Lab That Kills a Process Before It Reads a Secret. Previous post: What Is a Sidecar-less Service Mesh? What It Solves, What You Gain, and a Cilium Lab with WireGuard and Mutual Auth.
The code for this lab (two kind configs, the common and per-cluster values, the global service demo, the two policies, the connect/failover/policy/Hubble scripts and the Makefile) is shared with Lab Access members on Buy Me a Coffee; the repo invite comes by email after you join: https://buymeacoffee.com/miraccanyilmaz/membership