What Is Cilium Gateway API? Benefits and a Hands-On Lab

Most Kubernetes clusters that run Cilium already have an Envoy proxy on every node. And most of them still install a separate ingress controller next to it, usually ingress-nginx. That is two proxies doing the same job. In this post I look at what Cilium Gateway API is, what problem it solves and what you gain, and then I show the lab where I removed the second proxy: one shared Gateway, a 90/10 traffic split, a header canary, HTTPS from cert-manager and a redirect. It broke in four places; those are at the end, with the real error messages.

What is Cilium Gateway API?

Gateway API is the Kubernetes standard for bringing traffic into a cluster. It is the successor of the Ingress object. Instead of one object with controller-specific annotations, it splits the job into a few clear resources:

GatewayClass: which implementation runs the gateway (here: Cilium).
Gateway: the entry point itself. Which ports, which hostnames, which TLS certificate, which namespaces may attach routes.
HTTPRoute: the routing rules. Host, path, headers, and which backend gets what share of the traffic.

Cilium Gateway API is Cilium's implementation of that standard. The controller runs inside cilium-operator, and the traffic goes through the cilium-envoy DaemonSet that Cilium already runs for L7 policies and Hubble. You turn it on with one Helm value (gatewayAPI.enabled); there is no extra controller to install.

What problem does it solve?

Two proxies for one job. If Cilium is your network plugin, Envoy is already on every node. An ingress controller next to it is a second proxy: a second thing to upgrade, a second list of security fixes, a second place to look when a request is slow.

Canary and routing rules hidden in annotations. Ingress only understands host and path. Everything else (traffic weights, header matching, redirects) is an annotation, and every controller has its own. Moving from one controller to another means rewriting all of them, and a typo in an annotation is often silently ignored.

No clear owner. With Ingress, the same object holds the TLS certificate, the hostnames and the application's routes. The platform team and the application team edit the same file. Gateway API separates them: the platform team owns the Gateway, application teams own their HTTPRoutes.

What you gain with Gateway API on Cilium

Gain	What it means in practice	In the lab
One proxy instead of two	Gateway traffic goes through the Envoy Cilium already runs	Extra components installed: only cert-manager
Weights and headers are fields, not annotations	A canary is `weight: 90` / `weight: 10` in the route	200 requests: 177 to v1, 23 to v2
Clear ownership	Platform team owns the Gateway, apps attach routes from their own namespace	A route from an unlabelled namespace was rejected
Portable	The same `HTTPRoute` works on Envoy Gateway, Istio and others; only the GatewayClass changes	Routes are plain Gateway API v1
HTTPS without manual certificates	cert-manager reads one annotation on the Gateway and creates the Secret	Certificate Ready before the wait even started

How it works

Diagram: before, Cilium's Envoy plus a separate ingress-nginx with canary rules as annotations; after, one cilium-envoy DaemonSet with Gateway and HTTPRoute and a 90/10 weight field. Below, one request: client, Gateway IP from the LB-IPAM pool, Cilium agent redirecting to the node's Envoy, HTTPRoute rules, podinfo v1 and v2

When the operator sees a Gateway, it creates a LoadBalancer Service for it. On kind there is no cloud load balancer, so the IP comes from Cilium's LB-IPAM pool and is announced on the network with L2 announcements, as in the IPAM post. When a request reaches that IP, the Cilium agent redirects it to the Envoy on the node, and Envoy applies the rules from the HTTPRoutes. No separate controller pod, no separate proxy.

The lab: what I built

Everything runs on kind on a MacBook (Colima, 4 CPU / 8 GB): Kubernetes 1.37, Cilium 1.20.2 without kube-proxy, Gateway API v1.6.1, cert-manager v1.21.2. The application is podinfo in two versions, v1 and v2; the response says which version answered, which is enough to count a split. The whole lab, from an empty machine to the final check, took 3 minutes 38 seconds in the last run.

Step 1: Install the Gateway API CRDs, then Cilium

Order matters here. The Cilium operator checks at startup that all seven Gateway API resource types exist. If one is missing, the Gateway controller never starts (that is the first error below). With the CRDs in place, the GatewayClass cilium was accepted, and the shared Gateway in the platform namespace got an IP from the pool and became Programmed in under a second.

make up: GatewayClass cilium ACCEPTED True; the shared Gateway in namespace platform is PROGRAMMED True with address 172.20.250.1; its LoadBalancer Service has the same external IP

Accepted only means "I understood the object". Programmed means "the configuration is in Envoy". When debugging, look at Programmed.

Step 2: Routes, a 90/10 split and a header canary

The application team writes HTTPRoutes in its own namespace (demo). I applied three versions of the same route, one after another, the way a real canary rollout would change one file in Git:

Host podinfo.lab.local goes to v1.
Weights 90 and 10 between v1 and v2. Out of 200 requests, 177 went to v1 and 23 to v2. Earlier runs gave 178/22 and 186/14. The split is per request and statistical, so it gets close to 90/10 without matching exactly.
Requests with the header x-canary: always go to v2, everything else to v1. This is how a tester reaches the new version on purpose.

Three routes: the first request answered by podinfo-v1; 200 requests at 90/10 counted as 177 v1 and 23 v2; with the x-canary header the answer is v2, without it v1

None of this needed an annotation. Weight and header match are normal fields of the route.

Step 3: HTTPS with cert-manager, then a redirect

For HTTPS I added a second listener on port 443 to the Gateway and one annotation that points to a ClusterIssuer. cert-manager saw the annotation and created the certificate Secret by itself; it was Ready before my wait command even started. For the lab I used a self-signed root CA, and the test checks the certificate against that CA (no -k). Last, a small route on port 80 sends a 301 to HTTPS.

The certificate podinfo-lab-local-tls is Ready and its Secret exists; HTTPS returns HTTP/2 200 from v1; plain HTTP on port 80 returns 301 Moved Permanently with location https://podinfo.lab.local:443/

Final check

make verify: GatewayClass True, shared-gw Programmed, two routes, one LoadBalancer Service with ports 80 and 443, cilium-envoy 3/3 ready; extra namespaces are only cert-manager and cilium-secrets

The only extra namespaces are cert-manager and cilium-secrets. The second one belongs to Cilium; the operator copies the listener's TLS Secret there for Envoy.

Where it broke: 4 real errors

1. GatewayClass stuck at `ACCEPTED Unknown`: one of the 7 CRDs was missing

In my first attempt I installed six of the seven Gateway API CRDs; I thought BackendTLSPolicy was not needed. The GatewayClass existed but never became True, and the operator log named the missing resource. Applying the CRD alone did not fix it, because the check only runs when the operator starts. A restart of cilium-operator and the cilium DaemonSet did.

First attempt: GatewayClass ACCEPTED Unknown; the operator log says Required GatewayAPI resources are not found, backendtlspolicies.gateway.networking.k8s.io not found; fix: apply the CRD and restart the operator

2. `AddressNotAssigned: Gateway waiting for address`: no IP pool

A Gateway gets its IP from whatever gives LoadBalancer Services an IP; on Cilium without a cloud, that is the LB-IPAM pool. I deleted the pool and created a second Gateway. Its Service stayed <pending>, and the Gateway said Programmed=False AddressNotAssigned. The surprise was the other Gateway: deleting the pool also took the IP away from shared-gw, which had been working. When I put the pool back, both were Programmed within 0.2 seconds, but in a different order: shared-gw moved from .1 to .2. Do not delete an IP pool on a live cluster.

Pool deleted: gw-b and shared-gw both PROGRAMMED False, both Services EXTERNAL-IP pending; gw-b condition AddressNotAssigned; pool back, gw-b programmed after 0.2 s, now gw-b has 172.20.250.1 and shared-gw 172.20.250.2

3. `NotAllowedByListeners`: a route from a namespace without the label

The Gateway only accepts routes from namespaces with the label gateway-access=true. I made a namespace without it and put a copy of the route there. The Gateway itself showed nothing wrong; its count of attached routes simply did not go up. The error was on the route's own status. Look there first when a route does nothing.

Route copy in namespace demo-b: the Gateway still has one attached route per listener; the route says Accepted=False NotAllowedByListeners, not allowed to attach due to namespace restrictions, and BackendNotFound for podinfo-v1

The second line is a bonus: the backend Service lives in demo, not in demo-b, and the route reports that too.

4. `HTTP/1.1` instead of `HTTP/2` on the HTTPS listener

My HTTPS test answered HTTP/1.1 200 although the client offered HTTP/2. Cilium does not advertise HTTP/2 on Gateway listeners by default (ALPN is off). With gatewayAPI.enableAlpn: true and an operator restart, the same test returned HTTP/2 200. Nothing warns you about this; you only notice it if you check.

With enableAlpn false the HTTPS test returns HTTP/1.1 200 OK; after setting it to true and restarting the operator it returns HTTP/2 200

One more practical note: on macOS the kind network is inside the Colima VM, so the Gateway IP is not reachable from the Mac. Every request in the lab was sent from a small curl pod inside the cluster.

When to use it, and when not

Situation	Choice
Cilium is already your network plugin, you need host/path routing, splits, header canaries, TLS termination	Cilium Gateway API. No second proxy.
You need rate limiting, JWT validation, external auth or WASM filters at the edge	Envoy Gateway (or another full gateway). Your `HTTPRoute`s move as they are; only the GatewayClass changes.
You don't run Cilium	Any Gateway API implementation; the routes in this post are standard.
You still have many Ingress objects	Migrate route by route; Gateway and Ingress objects can live in the same cluster while you move.

Cilium also has its own Ingress controller (ingressController.enabled). I did not use it: an Ingress object still cannot express weights or headers without annotations, and Cilium's docs recommend Gateway API for new setups.

Summary

Cilium Gateway API lets the Envoy that Cilium already runs act as your ingress. No separate controller.
Weights, header matches and redirects are fields in a standard HTTPRoute, so the same routes work on other implementations.
In the lab: one shared Gateway, a 90/10 split that landed at 177/23, a header canary, HTTPS with HTTP/2 and a redirect. The only extra install was cert-manager.
Four things to watch: all 7 CRDs before Cilium, an IP pool before the Gateway (and never delete it on a live cluster), the namespace label for routes, and enableAlpn for HTTP/2.

Sources: Cilium docs Gateway API Support, LoadBalancer IP Address Management (LB IPAM) and L2 Announcements; Kubernetes Gateway API docs (v1.6.1, standard channel); cert-manager docs Annotated Gateway resource.

Next in the series

This is the fourth post in the Cilium End to End series. Next: What Is a Sidecar-less Service Mesh? What It Solves, What You Gain, and a Cilium Lab with WireGuard and Mutual Auth. Previous post: The Road to Default-Deny with CiliumNetworkPolicy: L3/L4, DNS and L7 HTTP Rules.

The code for this lab (kind config, Cilium values, the Gateway and route files, cert-manager setup, scripts and the Makefile) is shared with Lab Access members on Buy Me a Coffee; the repo invite comes by email after you join: https://buymeacoffee.com/miraccanyilmaz/membership