When a pod in Kubernetes calls a Service, most teams don't know what sends the packet to the right pod, and they don't need to: kube-proxy writes iptables rules on every node and it works. Until the cluster grows. In this lab I put 100 more Services in front of the same two nginx pods: on the kube-proxy cluster, the iptables rules on a node went from 117 lines to 1217. On the Cilium cluster they stayed at 93, with 111 Services.
This is the first post of the Cilium series. I build a kind cluster without kube-proxy, install Cilium, and look at three questions: what replaced kube-proxy, how Cilium recognizes pods, and how we see what happens on the network. Every step with the lab's real output and screenshots.
The problem, and why it matters
Networking in Kubernetes has two basic jobs: give pods IPs and connect them, and when someone says "this Service", turn the request into one of the pods behind it. In a classic setup kube-proxy does the second job, and it does it by writing rules on every node, for every Service. More Services, more rules, and the rules are rewritten on every change.
The second problem is security. Answering "who may talk to whom" with IP addresses is hard in Kubernetes, because pods restart all the time and their IPs change. The payment service that was 10.244.2.241 yesterday has another address today.
The third is visibility. "Why didn't this request arrive" usually means logging into a node and capturing packets; there is no standard way to ask the whole cluster "which pod called which service, and was it allowed".
Cilium solves all three in one component: a fast lookup inside the kernel instead of a rule table, an identity derived from labels instead of IPs, and Hubble, which shows every flow.
What is Cilium: eBPF, identity and Hubble
eBPF. Small, verified programs that run inside the kernel, attached to specific points (a packet arriving on an interface, a connection being opened), without recompiling the kernel. Cilium attaches them to every pod's network interface. When a packet arrives, the program looks at an eBPF map: which pods does this Service IP go to, may this source talk to this destination. The map is a hash table; with five Services or five thousand, the lookup takes the same time.
Identity. Cilium gives every pod a numeric identity derived from its labels. Pods with the same labels share an identity; a pod that restarts and gets a new IP keeps it. Policies are written against identities, not IPs. In this post we see identities; the series' third post writes the rules.
Hubble. Cilium's datapath already sees every flow; Hubble exposes it through an API and a UI. Instead of capturing packets you ask the whole cluster "who went where, and what was the verdict".
Why Cilium: compared with kube-proxy
| kube-proxy (iptables mode) | Cilium (kube-proxy replacement) | |
|---|---|---|
| Service resolution | One iptables chain per Service, on every node | eBPF load-balancer map, a hash lookup |
| As Services grow | Rules grow linearly (~11 lines per Service in the lab) | iptables doesn't change (93 lines in the lab) |
| Recognizing a pod | IP address | Identity derived from labels |
| Visibility | None; packet capture on a node | Hubble: CLI and UI, cluster-wide |
| Cost | Simple, everyone knows it | Learning curve; debugging needs cilium-dbg |
Calico has its own eBPF dataplane and is a serious alternative. I picked Cilium for this series because what the next posts need (LoadBalancer IPs, Gateway API, encryption, multi-cluster) comes with the same agent: one thing to learn instead of separate components to install and run.
Why kind, and why without kube-proxy? Reading "Cilium replaces kube-proxy" is one thing; watching Services work on a cluster that has no kube-proxy pod at all is another. kind gives you that cluster in minutes, and deleting it is one command.
The lab: what I built
On an Apple Silicon Mac, on Colima (4 CPU / 8 GB, kernel 6.8):
- kind v0.33 with 1 control plane + 2 workers, Kubernetes v1.37.0. The default network plugin (CNI) and kube-proxy are off.
- Cilium 1.20.2 with Helm: kube-proxy replacement on, VXLAN tunnel, Hubble Relay and Hubble UI on.
- Cilium's own Star Wars demo: a two-replica
deathstarService,tiefighter(Empire) andxwing(Alliance). The same three apps run through the whole series.
Every step is in a Makefile; the whole install from scratch takes about 2.5 minutes (cluster 28 s, Cilium 121 s).
Step 1 — kind without a CNI and without kube-proxy: nodes NotReady
When the cluster first comes up, the nodes are NotReady and the CoreDNS pods are Pending. That's expected: there's no network plugin to give pods an IP. kube-proxy was never installed.
Step 2 — Installing Cilium, and cilium status
Cilium went in with Helm; cilium status showed every component OK and the nodes turned Ready. Each node runs a Cilium agent (DaemonSet) and an Envoy (for L7 traffic, in later posts), plus one operator, Hubble Relay and Hubble UI.
Step 3 — What replaced kube-proxy: the eBPF service map
cilium-dbg status inside the agent says two things. First, KubeProxyReplacement: True: Service resolution now happens in eBPF. Second, something I didn't expect: Masquerading: IPTables and Host: Legacy. So even on a cluster without kube-proxy, Cilium still uses iptables for address translation of outgoing traffic. Moving that to eBPF is a separate setting I didn't turn on here; the Service side is fully eBPF.
With kube-proxy, a packet's path goes through one chain per Service. With Cilium, an eBPF program on the pod's interface does one lookup in one table:
To measure the difference I built two clusters: kind's default (with kube-proxy) and this lab. On both I put 100 Services in front of the same two nginx pods and counted the iptables lines on a node:
On a small cluster the difference is small: 117 versus 93 with 3 Services. It opens up with the number of Services: kube-proxy adds about 11 lines and one chain per Service, while on Cilium iptables doesn't change at all; the Services live in the eBPF load-balancer table (459 entries). When I repeated the measurement, the starting numbers were slightly different (105 for kube-proxy, 89 for Cilium; other rules on the node change over time), but the result was the same: 1217 with kube-proxy, no growth at all with Cilium.
Step 4 — Cilium identity: labels, not IPs
Once the demo app was installed, every pod became a CiliumEndpoint and got an identity. The two deathstar replicas are on different nodes with different IPs, but share one identity (8469), because their labels are the same.
To see that the identity isn't tied to an IP, I deleted a deathstar pod. Its replacement got a new IP (10.244.2.35) and the same identity, 8469:
This is the foundation for the rest of the series: policies will be written against these identities, and the rules stay the same however often pods move. Right now there's no policy, so every identity can talk to every identity: both tiefighter and xwing can land on the Death Star ("Ship landed"). The second one being allowed is the problem of the series' third post.
Step 5 — Seeing flows with Hubble: CLI and Hubble UI
Hubble Relay collects flows from the agents on all nodes. I looked at the flows going to deathstar with hubble observe. Each line has the source pod and its identity, the destination pod and its identity, the direction and the verdict:
Right now the only verdict is FORWARDED. With policies DROPPED shows up too; with L7 policies the HTTP method and path join these lines.
Hubble UI shows the same thing as a service map. In the default namespace it draws tiefighter and xwing going to port 80 of deathstar, with the flow table below:
Step 6 — cilium connectivity test
Cilium's own connectivity test starts client and server pods in its own namespace and runs scenarios. Narrowed to pod-to-pod, pod-to-service and L7 tests, it took 574 seconds on Colima: 51 tests (381 actions) successful, 84 tests skipped because of the narrowing.
What went wrong: 4 real failures
1. Establishing connection to apiserver ... localhost:6443: k8sServiceHost set to localhost
On a cluster without kube-proxy, Cilium has to know on its own how to reach the API server; Service IPs don't work yet, because Cilium is what makes them work. So we give the API server's address explicitly in Helm. Once I wrote localhost, because that's what the kubeconfig shows.
The result was sneaky: no error line at all. The agents on the worker nodes stayed at Init:0/6; their init container quietly wrote "Establishing connection to apiserver ... localhost:6443" every five seconds and waited. The agent on the control-plane node worked, because on that node localhost:6443 really is the API server.
The right value is the control-plane container's name on the Docker network (cilium-lab-control-plane). In production it's the API server's load balancer address.
2. dial tcp 127.0.0.1:4245: connect: connection refused: Hubble CLI without a port-forward
The Hubble CLI connects to Hubble Relay inside the cluster. Run it without a port-forward and you get this error. Opening cilium hubble port-forward fixed it.
3. hubble observe inside an agent came back empty
To skip the port-forward, I ran hubble observe inside a Cilium agent; there was demo traffic, yet the output was empty. Hubble inside an agent only sees that agent's node, and the demo pods were on other nodes. To see the whole cluster you look through the Relay (with the port-forward from #2).
4. No kube-proxy, yet Masquerading still says "IPTables"
After a "kube-proxy-free" install I assumed iptables was gone entirely. cilium-dbg status disagrees: address translation of outgoing traffic (masquerading) and host routing use iptables and the legacy path in the default install. Services are in eBPF (the measurement in Step 3 shows it), but it isn't "zero iptables": the node has 93 lines. eBPF masquerading is a separate Helm setting and needs kernel support; I didn't turn it on here.
Verify
These show the lab is set up right:
- There's no
kube-proxyDaemonSet (NotFound), yet Services work. cilium status: Cilium, Operator, Envoy and Hubble Relay OK; nodes Ready.cilium-dbg status:KubeProxyReplacement: True.- The iptables line count on a node doesn't change when Services are added; zero
KUBE-SVCchains. kubectl get cep: every pod has an identity, pods with the same labels share it.- Hubble UI shows the
defaultnamespace's service map andforwardedflows. cilium connectivity test: 51/51 successful.
Sources: Cilium docs Kubernetes Without kube-proxy, Identity-Relevant Labels, Setting up Hubble Observability and Connectivity Test; kind Configuration (disableDefaultCNI, kubeProxyMode); Kubernetes Virtual IPs and Service Proxies.
Next in the series
This is the first post of the Cilium End to End series. Next: Cilium IPAM: Pod CIDR Modes, LB-IPAM and L2 Announcements for a Bare-Metal LoadBalancer. Previous post: Kyverno Admission Webhooks.
The code for this lab (kind configuration, Cilium values, scripts and Makefile) is shared with Buy Me a Coffee supporters; access comes by email after you support: https://buymeacoffee.com/miraccanyilmaz