In most Kubernetes clusters, traffic between pods on different nodes is not encrypted. I ran a packet capture on a node of my lab cluster and read the HTTP requests between two pods in plain text. The usual answer is a service mesh, and a service mesh usually means one extra proxy container in every pod. In this post I look at what a sidecar-less service mesh is, what it solves and what you gain, and then I show the lab where Cilium did the mesh jobs without a single sidecar: encryption, identity, HTTP visibility and a traffic split. It broke in a few places; those are at the end, with the real messages.
What is a sidecar-less service mesh?
A service mesh is a layer that handles four things for the services in a cluster, so the applications don't have to:
- Encryption: traffic between services can't be read on the network.
- Identity: a service only accepts requests from services it knows.
- Visibility: you can see which service called which, with method, path and status code.
- Traffic management: splitting traffic between versions, routing by header.
The classic way (Istio, Linkerd) puts a proxy sidecar container next to every application container. All traffic goes through it.
A sidecar-less mesh does the same jobs once per node instead of once per pod. In Cilium, encryption happens in the Linux kernel with WireGuard, HTTP-level work goes through one Envoy per node (cilium-envoy), and identity comes from SPIFFE through SPIRE. The pods stay as they are.
What problem does it solve?
Traffic in the clear. By default, nothing encrypts pod-to-pod traffic between nodes. Anyone who can see the network (a compromised node, a mirrored switch port, a cloud network tap) can read it. In the lab, one capture on one node showed GET / HTTP/1.1, the Host header and the 200 OK.
The cost of sidecars. A sidecar is one more container in every pod: it is injected on every deploy, it needs memory and CPU in every pod, it must be restarted when the mesh is upgraded, and every kubectl exec asks "which container?". For many teams that cost is the reason they never turn the mesh on.
"Who called whom?" Without a mesh, the network only sees IPs and ports. When something returns 500, you can't see which caller, which path, how long it took.
What you gain with Cilium's sidecar-less mesh
| Gain | What it means in practice | In the lab |
|---|---|---|
| Encryption with no change to the apps | WireGuard between nodes, in the kernel; pods don't know | Readable HTTP lines in the capture: 6 before, 0 after |
| Small switch | Two Helm values and an agent restart | 31 s on three nodes |
| Identity checks between services | A policy with authentication.mode: required; agents do a SPIFFE handshake |
The intruder pod timed out; the client got 200 after a ~1 s handshake |
| HTTP visibility | An L7 rule sends traffic through the node's Envoy; Hubble shows method, path, status | GET /headers → 200 2ms per request |
| Standard traffic split | GAMMA: a Gateway API HTTPRoute whose parent is a Service |
50 requests at 90/10: 46 / 4 |
| No extra component per pod | Envoy and SPIRE agent run per node | 0 sidecars |
Before deciding, know what you don't get:
| Need | Cilium 1.20 |
|---|---|
| Retries, timeouts, circuit breaking | Only through raw Envoy config (CiliumEnvoyConfig), little validation |
| A certificate inside the pod (application-level mTLS) | No. Encryption happens at the node; the pod never sees a certificate |
| Identity across clusters | No. Mutual auth does not work with Cluster Mesh in 1.20 |
| Long-term mutual auth | Deprecated in 1.20, removed in 1.21. The upgrade notes point to ztunnel transparent encryption instead |
I still show mutual auth, because the policy model (authentication.mode: required) is what you will meet in the replacement too.
How it works
- Encryption: every Cilium agent creates a WireGuard interface (
cilium_wg0) and shares its public key through Kubernetes. Pod traffic to another node goes through that interface, encrypted, on UDP port 51871. - Identity: Cilium already gives every group of pods a numeric identity from their labels. With mutual auth on, the operator registers those identities in SPIRE, and before the first packet between two identities passes, the agents on both sides check each other's SPIFFE certificate.
- Visibility and traffic split: an L7 rule or an
HTTPRoutesends the traffic throughcilium-envoyon the node, which reports every request to Hubble and applies weights and header matches.
The lab: what I built
Everything runs on kind on a MacBook (Colima, 4 CPU / 8 GB, kernel 6.8 with the WireGuard module): Kubernetes 1.37, Cilium 1.20.2, SPIRE 1.15.3 from the Cilium chart, Gateway API v1.6.1. Two podinfo versions run on one worker node; a client pod and an intruder pod run on the other, so their traffic has to cross nodes. Pods on the same node never go through WireGuard. The whole lab took about 4 minutes in the last run.
Step 1: Read the traffic, then encrypt it
First the capture without encryption. A small hostNetwork pod with tcpdump on the podinfo node watched eth0 while the client sent three requests. Inside 80 packets there were 6 readable HTTP lines: the request line, the Host and User-Agent headers and the response status.
Then two Helm values (encryption.enabled: true, encryption.type: wireguard) and a restart of the Cilium agents. 31 seconds later every agent reported WireGuard with 2 peers, one for each other node.
The same capture after that: only UDP packets on port 51871, and 0 readable HTTP lines in 80 packets. The apps did not change, and no pod restarted except the Cilium agents.
Step 2: No identity, no way in
SPIRE comes with the Cilium chart. The server was healthy, all three node agents were attested, and the operator had registered podinfo's Cilium identity as a SPIFFE ID.
Then a policy on podinfo: accept the client, and require authentication. The intruder, which the policy does not allow, timed out. The client got 200, but the first request took about one second: that is the handshake between the two agents. Hubble showed the dropped SYN packets with Authentication required, and the forwarded flows carried auth_type: SPIRE.
Two requests, two one-second waits: podinfo v1 and v2 carry different labels, so they are two identities, and each one pays its own first handshake. Ten replicas of one Deployment would be one identity and one handshake.
Step 3: See HTTP without a sidecar
An L7 rule that allows all HTTP on port 9898 is enough to send that traffic through the node's Envoy. After that Hubble showed every request with method, URL, status and time.
Step 4: Split traffic inside the cluster with GAMMA
GAMMA is the part of Gateway API for traffic inside the cluster: an HTTPRoute whose parent is a Service instead of a Gateway. With weights 90 and 10, 46 of 50 requests went to v1 and 4 to v2 (44/6 and 45/5 in earlier runs). With a second rule, requests with the header x-canary: true went to v2.
Final check
Where it broke: 3 real errors
1. SPIRE server stuck at 1/2: Stopping Server APIs right after Starting Server APIs
With the chart defaults, the SPIRE server never became ready and the node agents waited in Init:Error. The log had no real error. The cause: the server runs as uid 1000, but its socket directory is a host path created as root, so binding the socket fails silently (cilium/cilium#40533). The fix in the lab's values is to run the server as root (securityContext.runAsUser: 0), the way the chart behaved before SPIRE 1.9.
2. The setting says on, the agents say Encryption: Disabled
helm upgrade with encryption on returned STATUS: deployed, and the Cilium ConfigMap said enable-wireguard: true. All three agents still said Disabled. Helm only rewrites the configuration; the agents read it when they start. After a restart of the cilium DaemonSet (27 s) all three were on WireGuard. Nothing warns you about this. If you don't check the agents, you think traffic is encrypted when it isn't.
3. failed to get certificate for local identity: the auth test ran too early
The first time, I applied the auth policy right after SPIRE came up, and the client timed out too. The operator registers identities in SPIRE on its own schedule, and at that moment podinfo's entry did not exist yet (Found 0 entries). Three minutes later the same test passed. Check the SPIRE entry before blaming the policy.
One practical note: the kind node image has no tcpdump. The lab runs the capture from a nicolaka/netshoot pod with hostNetwork, pinned to the node, which sees the same eth0.
When to use it, and when not
| Situation | Choice |
|---|---|
| Cilium is already your network plugin and you want encryption between nodes, HTTP visibility and simple splits | Cilium's sidecar-less mesh. No new components per pod. |
| You need retries, timeouts, fault injection or outlier detection every day | Istio ambient (also sidecar-less: ztunnel per node, waypoint proxy for L7). |
| Regulation requires a certificate inside the application (end-to-end mTLS) | A mesh with application-level mTLS; Cilium encrypts at the node. |
| You need service identity long term | Plan for ztunnel; Cilium's SPIFFE mutual auth is deprecated in 1.20 and removed in 1.21. |
Summary
- A sidecar-less service mesh does encryption, identity, visibility and traffic management once per node, not once per pod.
- With Cilium, WireGuard took readable HTTP on the wire from 6 lines to 0, in 31 seconds, with no change to the apps.
- SPIFFE mutual auth blocked a pod without permission, at a cost of about one second for the first request per identity, and it is deprecated in 1.20.
- An L7 rule gave HTTP visibility in Hubble, and GAMMA split in-cluster traffic 46/4 at 90/10.
- Always check the agents after
helm upgrade: the setting can say on while encryption is off.
Sources: Cilium docs Transparent Encryption (WireGuard), Mutual Authentication (Beta), Layer 7 Protocol Visibility and Gateway API GAMMA support; Cilium 1.20 upgrade notes; cilium/cilium#40533 and #47132; Gateway API GAMMA initiative.
Next in the series
This is the fifth post in the Cilium End to End series. Next: Cilium Cluster Mesh: Two kind Clusters, Global Services and Cross-Cluster Policy. Previous post: What Is Cilium Gateway API? What It Solves, What You Gain, and a Lab Without a Separate Ingress Controller.
The code for this lab (kind config, the mesh and encryption values with the SPIRE fix, the podinfo/client/intruder manifest, the two policies, the two GAMMA routes, the capture and auth scripts and the Makefile) is shared with Lab Access members on Buy Me a Coffee; the repo invite comes by email after you join: https://buymeacoffee.com/miraccanyilmaz/membership