The Road to Default-Deny with CiliumNetworkPolicy: L3/L4, DNS and L7 HTTP Rules

In a Kubernetes cluster without network policies, every pod can reach every other pod and the whole internet. The first thing security usually asks for is "turn on default-deny": everything closed, only what we allow is open. In this post I did that step by step with Cilium on kind, and it broke in places I didn't expect. The most surprising one: a cluster-wide policy that only allows things, and denies nothing, cut the internet access of another namespace I never touched.

This is the third post in the Cilium series. We write rules on top of the identities from the first post: first L3/L4, then HTTP method and path, then DNS names, and finally default-deny for a whole namespace. Every step comes with real output from the lab and the verdicts Hubble shows.

The problem, and why it matters

Kubernetes defaults to "everything open". When a pod is compromised or misconfigured, it can reach every service in the cluster and every address outside. Network policy is how you narrow that down.

Kubernetes' own NetworkPolicy works at the IP and port level. That is often not enough:

• You can't write "the payment service only accepts POST /charge"; port 80 is either open or closed.
• You can't say "this pod may only reach api.github.com"; the IPs of external services keep changing.
• There is no standard way to see what a rule is blocking.

The hard part isn't writing the rule, though, it's turning it on. The moment you enable default-deny on a live cluster, dependencies nobody listed (DNS, the API server, metrics scraping) break at once.

Cilium's CiliumNetworkPolicy covers all three: L7 HTTP rules, egress by DNS name, and every decision (verdict) visible in Hubble. There is also audit mode, to see "what would I block" without enforcing anything.

What a Cilium policy looks at: identity, direction and layer

In the first post of the series we saw that each pod gets an identity derived from its labels. Cilium policies are written against that identity, not IPs; the rule stays the same when a pod restarts with a new IP.

Once a policy selects a pod, the pod goes into default-deny for that direction (ingress or egress): anything the policy doesn't allow is dropped. A pod that no policy selects is wide open. So turning on default-deny really means "write a policy that selects every pod in a namespace, in both directions".

A request needs permission at both ends: the client's egress and the server's ingress. That turned out to be the most important lesson of the lab:

The lab: what I built

On an Apple Silicon Mac, on Colima (4 CPU / 8 GB, kernel 6.8), the same cluster as in the first post:

• kind v0.33 with 1 control plane + 2 workers, Kubernetes v1.37.0, default CNI and kube-proxy disabled.
• Cilium 1.20.2, kube-proxy replacement, VXLAN, Hubble Relay and UI on.
• The demo is Cilium's Star Wars app again: a two-replica deathstar service, tiefighter (Empire, org=empire) and xwing (Alliance, org=alliance). Plus a pod that goes outside: mediabot.
• Six policy files, applied in order; after each step the same three requests: tiefighter landing, xwing landing, and tiefighter's PUT /v1/exhaust-port (the request that blows up the Death Star).

Cluster and Cilium from scratch take 79 seconds, all policy steps 72 seconds, the audit mode round 73 seconds.

Step 1 — L3/L4 policy: who talks to whom, on which port

The first policy selects deathstar and only allows traffic from pods labeled org=empire, only on 80/TCP. Then a second policy adds an L7 rule:

With no policy all three requests went through; tiefighter's PUT "exploded" the deathstar. After 01, xwing's request got no answer at all (5-second timeout), because the packet is dropped before it reaches the deathstar. But tiefighter can still blow it up: the L3/L4 rule says "port 80 is open", it doesn't look at what goes through it.

Step 2 — L7 HTTP policy: rules on method and path

02 adds an HTTP section to the same rule: only POST /v1/request-landing. With this rule in place, Cilium sends traffic for that port through the Envoy proxy on the node; the proxy reads the request and turns away anything that doesn't match with Access denied. tiefighter still lands, but can no longer reach the exhaust port.

Hubble shows the two layers separately. L3/L4 decisions are policy-verdict lines, L7 decisions are http-request lines:

The lines also carry the source and destination identities (ID:14317 xwing, ID:25112 deathstar). To see which rule stopped whom and why, you don't need to log into a node and capture packets.

Step 3 — DNS-based egress (toFQDNs): GitHub only

03 restricts outbound traffic by DNS name: mediabot may only reach *.github.com and github.com, only on 443. We don't write IPs; Cilium sends the pod's DNS queries through its own DNS proxy, learns the IPs in the answers and applies the rule to those IPs.

Requests to GitHub returned 200, docs.cilium.io timed out. In Hubble the destination shows as (world) with the verdict EGRESS DENIED. The agent's FQDN cache holds the IPs Cilium learned from the DNS answers.

This policy has a second rule: port 53 to kube-dns. Without it the pod can't even send a DNS query, and toFQDNs never learns an IP. That comes back later as a bigger problem.

Step 4 — Namespace-wide default-deny and the egress trap

Now the real step. 04 selects every pod in the default namespace and closes both directions, keeping only DNS to kube-dns open. 05 is a cluster-wide baseline: DNS and API server access for every namespace except kube-system. I applied both and sent the same three requests:

All three requests failed, tiefighter's too. The deathstar's ingress rule still allowed tiefighter; the problem was at the other end. Hubble said it plainly: EGRESS DENIED. Default-deny had closed tiefighter's outbound side too, and no rule told it "you may go to the deathstar".

06 adds that egress rule: tiefighter to the deathstar's port 80. With it applied, things were back to normal: tiefighter lands, xwing is dropped, the exhaust port is refused. Lesson: after default-deny, every connection needs two rules, the server's ingress and the client's egress. Kubernetes' own NetworkPolicy works the same way.

Step 5 — Audit mode: see it before you enforce it

On a live cluster, applying 04 and then 06 means an outage in between. That's what Cilium's audit mode is for: policies are evaluated and Hubble logs "I would have dropped this", but nothing is dropped.

To try it, I deleted 06 and turned on audit mode with Helm:

Audit mode didn't turn on. The ConfigMap says true, but the agents keep dropping traffic, and the log says why: Mismatch found ... policy-audit-mode actual=false. It's the same thing I hit with L2 announcements in the previous post: helm upgrade writes the setting, and the running agent only reads it at startup. Here the result is more dangerous: you think "I'm in audit, nothing will be cut" while you are actually enforcing.

After restarting the agents, audit mode really was on:

Now even xwing lands, and Hubble has AUDITED lines: tiefighter's egress (because 06 is missing) and both directions for xwing. That list is the list of rules to write before you switch to enforce.

One thing stood out: the exhaust-port request still got Access denied in audit mode. Audit mode relaxes L3/L4 decisions, but L7 HTTP rules keep being enforced in Envoy. If you add an L7 rule trusting audit mode, that rule will block even in audit.

Final state: which policy selects which pod

With audit off and 06 back in place, the final state:

Every endpoint is under policy in both directions; four policies in the namespace, one baseline for the cluster. Each agent only knows the pods on its own node, so I used a small script that collects this table from the agent on every node.

Where it broke: 4 real errors

1. policy-verdict:none EGRESS DENIED (UDP): default-deny without a DNS rule

The plainest default-deny is an empty rule for both directions: nothing allowed. I applied that (without the baseline) and sent a request from tiefighter to the deathstar by service name:

curl didn't say "could not resolve the name"; it looked like a plain timeout. The real cause is in Hubble: UDP 53 to kube-dns was dropped. In a real application this shows up as every service saying "can't connect" at the same time, and DNS is not the first thing you think of. That's why 04 and 05 both carry a DNS rule.

2. An allow-only CCNP locked another namespace

The baseline (05) is a CiliumClusterwideNetworkPolicy that only contains allows: DNS and the API server. It's easy to think "it only allows things, it can't break anything". To test that, I created a separate namespace with no policies of its own (team-b) and sent a request from it to GitHub:

Applied without the enableDefaultDeny field, the baseline cut the internet access of the pod in team-b. The reason is the rule from above: the moment a policy selects a pod in the egress direction, that pod goes into egress default-deny. The baseline selects every pod outside kube-system, which means every namespace in the cluster. With enableDefaultDeny: {ingress: false, egress: false} the policy only adds its allows and puts no one into default-deny. Don't write a cluster-wide policy without that field.

3. Mismatch found ... key=policy-audit-mode actual=false: audit mode didn't turn on

The case from Step 5. After helm upgrade the ConfigMap is right, and the agents run with the old setting. The lab's audit-on and audit-off targets now restart the agents.

4. The Makefile demo never ran the tests in between

My first demo target called the same test target after every policy step. The output had only the first test: make runs a target only once per invocation and silently skips repeated prerequisites. What I thought was "test after every step" was a single test. The demo is now a script and really tests every step.

The draft had two more items that the lab didn't confirm, so I removed them: on kind the kube-apiserver entity matched correctly (traffic to the API server shows as (kube-apiserver) in Hubble and is dropped once the baseline is removed), and the toFQDNs rule worked from the very first request.

An order for turning on default-deny in a live cluster

The order that came out of the lab, as a proposal for a large cluster:

1. Cluster-wide baseline: DNS and the API server, with enableDefaultDeny: false.
2. Turn on audit mode, restart the agents, check that no Mismatch found warning is left.
3. Default-deny for one namespace. Use --verdict AUDIT in Hubble to see what would be cut.
4. For every AUDITED flow, write one policy per intent, at both ends (the server's ingress, the client's egress).
5. The next namespaces, a few at a time.
6. Once all are clean, turn audit off, again restarting the agents.

Because L7 rules are enforced even in audit mode, it's safer to add them after the audit round.

Verification

• kubectl get cnp,ccnp -A: deathstar-l3l4, default-deny, empire-to-deathstar, mediabot-fqdn and the cluster-wide baseline-allow, all VALID True.
• Every endpoint in the default namespace is under policy in both directions.
• tiefighter lands, xwing times out, the exhaust port returns Access denied.
• mediabot gets 200 from GitHub and can't reach docs.cilium.io.
• hubble observe --type policy-verdict shows every decision with source and destination identity.
• With audit mode on, L3/L4 decisions are AUDITED and the agent log has no Mismatch found.

Sources: Cilium docs Layer 3 Examples, Layer 4 Examples, Layer 7 Examples, DNS based policy (toFQDNs), Policy Enforcement Modes, Policy Audit Mode and CiliumClusterwideNetworkPolicy (enableDefaultDeny); Kubernetes Network Policies.

Next in the series

This is the third post in the Cilium End to End series. Next: Cilium Gateway API: Built-in Envoy, HTTP, TLS and Traffic Splits Without a Separate Ingress Controller. Previous post: Cilium IPAM: Pod CIDR Modes, LB-IPAM and L2 Announcements for a Bare-Metal LoadBalancer.

The code for this lab (kind config, the six policies, two trap scenarios, the audit round, scripts and the Makefile) is shared with Lab Access members on Buy Me a Coffee; the repo invite comes by email after you join: https://buymeacoffee.com/miraccanyilmaz/membership