Kyverno Admission Webhooks: Default-Deny for Every Namespace, 3 Core Rules, and the Gap When Kyverno Goes Down

In most Kubernetes clusters "we have rules" means a wiki page and good intentions: don't use latest, don't deploy to default, put a memory limit on every container, give every namespace a default-deny. As long as the rules live in people's heads, someone forgets, and nothing breaks on the day they do. The problem shows up later, when a node runs out of memory or an unexpected connection gets out of a namespace that never had a policy.

This post moves those rules to the cluster's front door, the Kubernetes admission layer. Kyverno writes the Cilium default-deny for every new namespace itself, refuses a pod that breaks a rule at the API server, and the same rules run in CI on a pull request without touching a cluster. I added three rules from Kyverno's official library to the lab; on the first pass they found 12 violations. Two of them were the lab's own "good example" manifests.

All the code is in the kyverno-cilium-policy-chain folder.

The problem, and why it matters

Kubernetes runs whatever it's given. An image tagged latest, a container with no memory limit, a test pod that lands in default: all valid YAML, all accepted by the API server. Code review catches some of it, but everyone who runs kubectl apply without a review, every Helm chart and every operator is outside that door.

On the network side the same gap is quieter. Cilium can give every namespace a closed default (default-deny), but someone has to write it by hand. If someone forgets to write it for a new namespace, every pod in it can talk anywhere, and no error ever points that out, because everything works.

The rules have to be enforced somewhere. The cheapest place is the moment an object enters the cluster: the admission layer, which the API server asks before it stores any request. A pod refused there never runs, and the fix happens in the author's terminal, right then.

What is a Kubernetes admission webhook: who's at the door, and what they ask

Every kubectl apply is a request to the API server. After authentication and authorization, and before writing to etcd, the API server asks two groups in order: first mutating admission (may change the object, for example add a missing label), then schema validation, then validating admission (only says yes or no). Some of these are built into the API server; the ones added from outside are admission webhooks: the API server sends the request over HTTPS to a service and continues or refuses based on the answer. Kyverno is that service.

The Kubernetes pages worth reading:

• Dynamic Admission Control: how webhooks are called. Especially failure policy, timeouts, namespaceSelector and best practices and warnings.
• Admission Control in Kubernetes: the admission controllers built into the API server.
• Validating Admission Policy and Mutating Admission Policy: their webhook-free counterparts, CEL running inside the API server.

Kyverno's webhooks in the lab, straight from the cluster:

gpol.validate.kyverno.svc-ignore-finegrained-generate-default-deny   Ignore   10s
vpol.validate.kyverno.svc-fail-01f0a879                              Fail     10s
vpol.validate.kyverno.svc-fail-86e66184                              Fail     10s

What the three settings mean:

• failurePolicy: Fail. If the webhook can't answer, the request is refused. I made Kyverno's service unreachable and tried to start a pod in team-a:

Error from server (InternalError): Internal error occurred: failed calling webhook "vpol.validate.kyverno.svc-fail-01f0a879": failed to call webhook: Post "https://kyverno-svc.kyverno.svc:443/vpol/disallow-host-network/disallow-latest-tag/disallow-privileged/require-app-label/require-requests-limits?timeout=10s": dial tcp 10.96.26.41:443: connect: no route to host

With Ignore, the pod would have gone in without any policy. The generate webhook is Ignore: creating a namespace doesn't depend on Kyverno (the cost of that is in What went wrong #3). - timeoutSeconds: 10. The API server waits at most this long. A slow webhook slows every deploy. - namespaceSelector. The API server never sends requests from some namespaces to the webhook. kube-system and kyverno are out; otherwise a dead Kyverno couldn't even recreate its own pod.

What you need

• Docker (Colima on a Mac, 4 CPU / 8 GB). make up builds a kind cluster with no CNI and no kube-proxy and puts Cilium and Hubble on it; nothing else to prepare.
• helm, kubectl, kind 0.33, the cilium CLI and the kyverno CLI (brew install kyverno cilium-cli).
• Versions in this run: Kubernetes v1.37.0 (kind node), Cilium 1.20.2, Kyverno v1.19.1 (chart 3.9.1), Gateway API v1.6.1 CRDs, kyverno CLI 1.19.1. make up takes 128 seconds.

Why Kyverno: compared with Gatekeeper and ValidatingAdmissionPolicy

There are three options for admission policy, and all three run in production:

The deciding row is generate. Neither VAP nor Gatekeeper can "create a CiliumNetworkPolicy when a namespace appears". VAP's advantage is having no component; since Kyverno 1.19's new API also uses CEL, rules move between the two.

The second decision is which Kyverno API. For years everyone wrote kyverno.io/v1 ClusterPolicy. Installing 1.19, the Helm notes say it plainly:

⚠️  WARNING: The legacy kyverno.io policy types are deprecated and will be removed in a future release. Migrate to their policies.kyverno.io replacements:
    - ClusterPolicy / Policy → ValidatingPolicy, MutatingPolicy, GeneratingPolicy, ImageValidatingPolicy (and their namespaced variants)

All nine policies in this post use the new API (policies.kyverno.io/v1). The ClusterPolicy examples you'll find online have different field names; don't mix them. Kyverno's policy library has moved to the new API too (the *-vpol, *-gpol, *-mpol folders).

The third decision: Audit first, then Deny. In Audit a violation isn't refused, it's written to a PolicyReport. Had I started with Deny, the lab's own "good" manifests and one of kind's system components would have been refused; Step 5 has the real list. The one exception is the Gateway rule: today there's no Gateway outside platform, so it's Deny from day one.

Step 1 — Gateway CRDs, Cilium, Kyverno, in that order

make up   # cluster → gateway-crds → cilium → kyverno

The order bit me once (What went wrong #1): Kyverno reads the cluster's list of resource types when it starts. If the Gateway API CRDs are installed after it, the Gateway rule refuses every Gateway with resource gateways not found. CRDs first.

Kyverno's values keep the system namespaces out in two separate places:

# kyverno/values.yaml (core)
config:
  resourceFilters:
    - '[*/*,kube-system,*]'
    - '[*/*,kube-public,*]'
    - '[*/*,kube-node-lease,*]'
    - '[*/*,kyverno,*]'
    - '[*/*,local-path-storage,*]'
  webhooks:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: [kube-system, kyverno]

resourceFilters works inside Kyverno: even if a request arrives, no policy is evaluated for these namespaces. webhooks.namespaceSelector works on the API server side: requests from these namespaces never reach Kyverno. When Kyverno is dead, the second one saves you.

NAME                                             READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-769b8f7647-7w9wq    1/1     Running   0          43s
kyverno-background-controller-86d8df7447-fkww8   1/1     Running   0          43s
kyverno-cleanup-controller-86c886ffff-8fs62      1/1     Running   0          43s
kyverno-reports-controller-57c7978d69-qx2wd      1/1     Running   0          43s

Four controllers, four jobs: admission answers the webhook, background runs generate, reports writes PolicyReports, cleanup deletes resources with a TTL. In the lab each has one replica, and the Helm notes warn: Setting the admission controller replica count below 2 means Kyverno is not running in high availability mode. In production the admission controller runs 3.

Step 2 — Kyverno GeneratingPolicy: a Cilium default-deny for every new namespace

Kyverno's default roles don't know the cilium.io group. Kyverno uses aggregated ClusterRoles for this: any ClusterRole with the right label is added to that controller's role automatically. Generate is done by the background controller, so that's who gets the permission:

# rbac/kyverno-cilium-clusterrole.yaml (first of three)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno:background-controller:cilium
  labels:
    rbac.kyverno.io/aggregate-to-background-controller: "true"
rules:
  - apiGroups: ["cilium.io"]
    resources: ["ciliumnetworkpolicies"]
    verbs: ["create", "update", "patch", "delete", "get", "list", "watch"]

The same file has two more roles: cilium.io read for the admission controller, Gateway read for the reports controller (#2 explains why).

The policy. The trigger is CREATE on a Namespace; what it produces is a CiliumNetworkPolicy that denies everything except DNS, as an object inside CEL:

# policies/generate-default-deny.yaml
apiVersion: policies.kyverno.io/v1
kind: GeneratingPolicy
metadata:
  name: generate-default-deny
spec:
  evaluation:
    synchronize:
      enabled: true          # policy changes propagate; a deleted CNP comes back
    generateExisting:
      enabled: true          # one-time pass over namespaces that already exist
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["namespaces"]
  matchConditions:
    - name: not-a-system-namespace
      expression: >-
        !(object.metadata.name in ['kube-system', 'kube-public', 'kube-node-lease',
          'kyverno', 'cilium-secrets', 'local-path-storage'])
    - name: not-opted-out
      expression: >-
        !(has(object.metadata.labels) &&
          'network.platform/default-deny' in object.metadata.labels &&
          object.metadata.labels['network.platform/default-deny'] == 'skip')
  variables:
    - name: nsName
      expression: "object.metadata.name"
    - name: downstream
      expression: >-
        [{
          "apiVersion": dyn("cilium.io/v2"),
          "kind": dyn("CiliumNetworkPolicy"),
          "metadata": dyn({
            "name": "default-deny",
            "namespace": string(variables.nsName),
            "labels": {"network.platform/managed-by": "kyverno"}
          }),
          "spec": dyn({
            "endpointSelector": {},
            "ingress": [],
            "egress": [
              {
                "toEndpoints": [
                  {"matchLabels": {"k8s:io.kubernetes.pod.namespace": "kube-system",
                                   "k8s:k8s-app": "kube-dns"}}
                ],
                "toPorts": [
                  {"ports": [{"port": "53", "protocol": "UDP"}, {"port": "53", "protocol": "TCP"}],
                   "rules": {"dns": [{"matchPattern": "*"}]}}
                ]
              }
            ]
          })
        }]
  generate:
    - expression: generator.Apply(variables.nsName, variables.downstream)

• generateExisting: true. A one-time pass over the namespaces that exist when the policy is created. That's how default got its default-deny in the lab.
• synchronize: true. When the policy changes, the generated CNPs follow; if someone deletes a CNP, it comes back. The CNP I deleted in the lab was back in 5 seconds. A CNP edited by hand was not reverted (#6).
• The DNS exception is inside the CNP. A default-deny that blocks everything blocks DNS too, and pods can't resolve names. The exception is part of every generated policy, so nobody has to remember it.

make policies && make demo

namespace/team-a created
--- generated CiliumNetworkPolicy:
spec:
  description: 'Generated by Kyverno: deny all, allow DNS to kube-dns'
  endpointSelector: {}
  ingress: []
status:
  conditions:
  - message: Policy validation succeeded
    status: "True"
    type: Valid

$ kubectl get cnp -A
NAMESPACE   NAME           AGE     VALID
default     default-deny   4m10s   True
team-a      default-deny   3m55s   True

Step 3 — Refuse privileged, hostNetwork and unlabeled pods at admission

Default-deny's holes are in the pod spec. A pod with hostNetwork: true uses the node's network; the namespace policy doesn't apply to it. A privileged: true container can write to /sys/fs/bpf, which means it can unload Tetragon's programs. A pod without an app label can't be selected by any CNP. All three are caught at admission:

# policies/disallow-privileged.yaml (core)
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: disallow-privileged
spec:
  validationActions: [Audit]
  matchConstraints:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: [kube-system, kube-public, kube-node-lease, kyverno, cilium-secrets, local-path-storage, tetragon]
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  evaluation:
    background:
      enabled: true
  autogen:
    podControllers:
      controllers: [deployments, statefulsets, daemonsets, jobs, cronjobs]
  variables:
    - name: allContainers
      expression: >-
        object.spec.containers + (has(object.spec.initContainers) ? object.spec.initContainers : [])
  validations:
    - expression: >-
        variables.allContainers.all(c,
          !has(c.securityContext) || !has(c.securityContext.privileged) || c.securityContext.privileged == false)
      message: "privileged containers are not allowed: they can disable the eBPF runtime layer (Tetragon)"

• autogen.podControllers. The rule is written for Pods but applied to Deployments and the rest too; the error shows up when the Deployment is applied, not later where a ReplicaSet silently fails to create pods. In the lab it refused a Deployment with a latest image at the Deployment level.
• evaluation.background: true. The rule runs not only on new requests but on pods already in the cluster, and writes the result to a PolicyReport. That's what the Audit phase is.
• The exclusion list is the same in every policy. Not a matter of taste; #5 shows why.

The hostNetwork rule is the same skeleton with one expression: !has(object.spec.hostNetwork) || object.spec.hostNetwork == false. So is the app label rule: has(object.metadata.labels) && 'app' in object.metadata.labels.

The Gateway rule looks at another API group and is Deny from day one. Tenants write HTTPRoutes; the Gateway belongs to the platform:

# policies/restrict-gateway-namespace.yaml (core)
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
      - apiGroups: ["gateway.networking.k8s.io"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["gateways"]
  validations:
    - expression: "object.metadata.namespace == 'platform'"

Error from server: error when creating "demo/bad-gateway.yaml": admission webhook "vpol.validate.kyverno.svc-fail-86e66184" denied the request: Policy restrict-gateway-namespace failed: Gateway objects live in the platform namespace only (got team-a); use an HTTPRoute

Step 4 — Kyverno MutatingPolicy: add a missing label instead of refusing

Making the app label mandatory means telling teams with old Helm charts to change them. A Deployment already has a name; we derive the label from it. Mutating admission runs before validating: the label is added, and validate sees the result.

# policies/mutate-name-label.yaml (core)
apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: mutate-app-label-from-name
spec:
  matchConditions:
    - name: template-has-no-app-label
      expression: >-
        !(has(object.spec.template.metadata) && has(object.spec.template.metadata.labels) &&
          'app' in object.spec.template.metadata.labels)
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: |
          Object{
            spec: Object.spec{
              template: Object.spec.template{
                metadata: Object.spec.template.metadata{
                  labels: Object.spec.template.metadata.labels{
                    app: object.metadata.name
                  }
                }
              }
            }
          }

--- unlabelled Deployment, labels after mutation:
deployment.apps/echo created
{"app":"echo"}
{"app":"echo","role":"echo"}

role is the team's own label, app is the one Kyverno added. Thanks to matchConditions, a Deployment that already has the label isn't touched.

Step 5 — 3 core rules from the Kyverno library: latest tag, default namespace, requests/limits

The rules so far are specific to this chain. Then there are the rules almost every cluster wants first. The best-practices-vpol folder of Kyverno's policy library has 15 rules; I took three, because all three are "works today, breaks one day" mistakes and each is fixed with one line. I found no data that measures which rule is more "popular"; the choice rests on these reasons.

I took the CEL expressions from the library unchanged; I only added the same system-namespace exclusion as the other rules, and autogen. The latest rule:

# policies/disallow-latest-tag.yaml (validations)
  validations:
    - expression: "object.spec.containers.all(container, container.image.contains(':'))"
      message: "An image tag is required."
    - expression: "object.spec.containers.all(container, !container.image.endsWith(':latest'))"
      message: "Using a mutable image tag e.g. 'latest' is not allowed."

The default namespace rule looks not at the pod but at the namespace it's in (namespaceObject):

# policies/disallow-default-namespace.yaml (validations)
  validations:
    - expression: "namespaceObject.metadata.name != 'default'"
      message: "Using 'default' namespace is not allowed."

The library keeps autogen off for this one: a Deployment carries its namespace in its own metadata, not in the template. The requests/limits rule checks containers, initContainers and ephemeralContainers; the file is in the folder.

All three were installed in Audit. The first PolicyReport:

default              disallow-default-namespace   Pod/quick-test                       Using 'default' namespace is not allowed.
local-path-storage   require-requests-limits      Pod/local-path-provisioner-75f7fc7dc5-twhg7 CPU and memory resource requests and memory limits are required for containers.
local-path-storage   require-requests-limits      Deployment/local-path-provisioner    CPU and memory resource requests and memory limits are required for containers.
team-a               disallow-latest-tag          Pod/careless                         Using a mutable image tag e.g. 'latest' is not allowed.
team-a               require-requests-limits      Pod/careless                         CPU and memory resource requests and memory limits are required for containers.
team-a               require-requests-limits      Pod/good                             CPU and memory resource requests and memory limits are required for containers.
team-a               require-requests-limits      Deployment/echo                      CPU and memory resource requests and memory limits are required for containers.
...

12 violations. quick-test, careless and bad are demo pods written badly on purpose. But good and echo were the lab's "good example" manifests, and neither had resources. local-path-provisioner is kind's own component; these two rules had left local-path-storage out of the system namespace list. With Deny on day one, both would have been refused. This is exactly what Audit is for: first the list, then the fix, then Deny.

The fix is two lines: resources on the good manifests, local-path-storage in the exclusion list. The next report showed only the three demo pods.

Step 6 — From Audit to Deny, and kyverno apply in CI

The switch to Deny happens in the policy files, not with kubectl patch (#4 explains why):

# scripts/enforce.sh (core)
for p in require-app-label disallow-host-network disallow-privileged \
         disallow-latest-tag require-requests-limits disallow-default-namespace; do
  f=$(grep -l "name: $p\$" policies/*.yaml)
  sed -i.bak 's/validationActions: \[Audit\]/validationActions: [Deny]/' "$f" && rm -f "$f.bak"
done
kubectl apply -f policies/

Applying the same bad pod again:

Error from server: error when creating "demo/bad-pod.yaml": admission webhook "vpol.validate.kyverno.svc-fail-01f0a879" denied the request: Policy disallow-host-network failed: hostNetwork is not allowed: it bypasses CiliumNetworkPolicy and the namespace default-deny;
  Policy disallow-privileged failed: privileged containers are not allowed: they can disable the eBPF runtime layer (Tetragon);
  Policy require-app-label failed: pod bad has no 'app' label; Cilium policies select on it;
  Policy require-requests-limits failed: CPU and memory resource requests and memory limits are required for containers.

Four violations, one message. A Deployment with a latest image was refused at the Deployment level too; the good pod keeps running.

The last piece: run the same rules on a pull request, without a cluster. kyverno apply applies the policies to a directory of manifests:

# ci/kyverno-apply.sh (core)
kyverno apply policies/ --resource "$MANIFESTS" --values-file ci/values.yaml --crd-paths "$CRDS"

policy restrict-gateway-namespace -> resource team-a/Gateway/rogue failed:
policy disallow-host-network -> resource team-a/Pod/bad failed:
policy disallow-privileged -> resource team-a/Pod/bad failed:
policy require-app-label -> resource team-a/Pod/bad failed:
policy require-requests-limits -> resource team-a/Pod/bad failed:
policy disallow-latest-tag -> resource team-a/Pod/careless failed:
policy require-requests-limits -> resource team-a/Pod/careless failed:
policy disallow-default-namespace -> resource default/Pod/quick-test failed:
pass: 25, fail: 8, warn: 0, error: 0, skip: 0

All 8 violations from the cluster, a non-zero exit code, a red PR. Without --values-file and --crd-paths this number was different; see #7.

The chain is complete: Kyverno keeps a rule-breaking pod out, and whatever gets in is under Cilium's default-deny. The next layer is watching what processes do inside (a runtime tool like Tetragon); that's outside this post.

What went wrong: 9 real failures and their fixes

1. resource gateways not found: Gateway CRDs installed after Kyverno

The first make up didn't install the CRDs at all; the Gateway policy stayed READY false. I installed the CRDs afterwards, and now Gateway requests were refused for the wrong reason:

admission webhook "vpol.validate.kyverno.svc-fail-86e66184" denied the request: resource gateways not found in group gateway.networking.k8s.io/v1

Kyverno reads the list of resource types when it starts. Restarting the Kyverno controllers fixed it. make up now installs the CRDs before Cilium and Kyverno.

2. Policy is not ready for reporting: the reports controller couldn't read Gateways

Even after the restart the status was:

WebhookConfigured=True; RBACPermissionsGranted=False Policy is not ready for reporting, missing permissions: get gateway.networking.k8s.io/v1, Resource=gateways

Refusing works (the webhook is set up), but the reports controller can't read Gateways, so it can't write PolicyReports. Giving the reports controller get/list/watch gateways through an aggregated role made it READY true. The role is needed even when the CRDs are there from the start; I removed it and checked.

3. While Kyverno was down: a pod got in, new namespaces got no default-deny

I tried two kinds of outage:

• Scaling the admission controller to 0. When Kyverno shuts down cleanly it deletes its own webhooks (webhooksCleanup.enabled: true in the chart): seven webhook configurations went down to two. With the policies in Deny, the careless pod (a latest image, no resources) was admitted. When Kyverno came back the pod kept running; the background scan reported it with 2 violations, but didn't stop it. During a planned shutdown, scale-down or upgrade, the door is open.
• Kyverno up, its service unreachable. The webhooks stayed, and the pod was refused with failed calling webhook ... no route to host (Fail). A pod in kube-system could still start (selector), and a new namespace could still be created (the generate webhook is Ignore).

The namespaces created during both outages (team-b, team-c) still had no default-deny 2 minutes after Kyverno came back. The "forgotten namespace" problem this post solves comes back with a Kyverno outage. generateExisting is a one-time pass that runs when the policy is created; adding an annotation to the policy didn't trigger it. Changing the spec did:

kubectl patch generatingpolicy generate-default-deny --type merge -p '{"spec":{"evaluation":{"generateExisting":{"enabled":false}}}}'
kubectl patch generatingpolicy generate-default-deny --type merge -p '{"spec":{"evaluation":{"generateExisting":{"enabled":true}}}}'

Within 10 seconds both namespaces had their default-deny. Those two lines go into the runbook for after a Kyverno outage. Running the admission controller with 3 replicas and a PodDisruptionBudget in production also narrows the window.

4. kubectl apply quietly put Deny back to Audit

The first enforce.sh switched the policies to Deny with kubectl patch. Then I fixed the exclusion list and ran kubectl apply -f policies/; the files said Audit, so all seven policies went back to Audit and the bad pod was admitted. No error, no warning. An Argo CD sync would have done the same. The switch to Deny now happens in the files; in a real repo that's a PR.

5. Four violations, one in the error: a separate webhook per namespaceSelector

On the first Deny attempt the bad pod had 4 violations in the report, but the error only mentioned require-app-label. Kyverno groups policies by their namespaceSelector and opens a separate webhook for each distinct exclusion list; my five pod rules had three different lists, and the two rules without a list (default namespace and Gateway) formed a fourth group: four vpol webhooks. The API server returns the answer of the first webhook that refuses. With one shared list the webhook count went down to two and all four violations came in one message. For a developer that's the difference between one apply and four.

6. synchronize: true doesn't revert a hand-edited CNP

I expected synchronize: true to undo manual changes. I changed the description of team-a's default-deny with kubectl patch; 45 seconds later the change was still there. When I deleted the CNP, it was back in 5 seconds. So if someone edits the default-deny to loosen it, Kyverno doesn't notice. The way to close that is a ValidatingPolicy that refuses UPDATE on CNPs labeled network.platform/managed-by=kyverno; I didn't try it in the lab.

7. error: 5 in CI: namespaceObject is empty, Gateways never evaluated

The first kyverno apply gave 6 fails and 5 errors. All five errors came from disallow-default-namespace: without a cluster, namespaceObject is empty. The fix, as in Kyverno's own CLI tests, is to provide the namespaces in a values file (ci/values.yaml, a namespaces: list). There was also a silent gap: the Gateway rule didn't run in CI at all, the result was 0 with no error. The CLI didn't know the Gateway kind; with the Gateway CRD passed through --crd-paths, the rogue Gateway failed too. Despite its name, --crd-paths doesn't take a directory: CRD file must have .yaml or .yml extension.

8. An excluded namespace stayed in the PolicyReport

After adding local-path-storage to the exclusion list, the old violation rows stayed in its PolicyReport. Deleting the reports cleared them, and they didn't come back. When you change an exclusion list, don't trust the old report rows.

9. permission denied ... ciliumnetworkpolicies: Kyverno doesn't retry generate

I removed the background controller's cilium.io role and created a new namespace. The namespace was created, no CNP came; the error was only in the background controller's log:

failed to generate resources: permission denied: ciliumnetworkpolicies.cilium.io "default-deny" is forbidden: User "system:serviceaccount:kyverno:kyverno-background-controller" cannot get resource "ciliumnetworkpolicies" in API group "cilium.io" in the namespace "team-d"

I restored the role; for 2 minutes nothing happened, Kyverno doesn't retry a failed generate. The generateExisting toggle from #3 worked here too. That's why the install order matters: rbac/ first, then policies/.

Verify

make verify

$ kubectl -n team-a get cnp
NAME           AGE    VALID
default-deny   6m2s   True

$ kubectl get validatingpolicies -o custom-columns='NAME:.metadata.name,ACTIONS:.spec.validationActions'
NAME                         ACTIONS
disallow-default-namespace   [Deny]
disallow-host-network        [Deny]
disallow-latest-tag          [Deny]
disallow-privileged          [Deny]
require-app-label            [Deny]
require-requests-limits      [Deny]
restrict-gateway-namespace   [Deny]

$ kubectl apply -f demo/bad-pod.yaml
Error from server: error when creating "demo/bad-pod.yaml": admission webhook "vpol.validate.kyverno.svc-fail-01f0a879" denied the request: Policy disallow-host-network failed: ...; Policy disallow-privileged failed: ...; Policy require-app-label failed: ...; Policy require-requests-limits failed: ...

• Create a new namespace and run kubectl -n <ns> get cnp: default-deny should be there within seconds.
• kubectl get validatingpolicies -o custom-columns=NAME:.metadata.name,READY:.status.conditionStatus.ready: all seven true.
• make ci: pass: 25, fail: 8, warn: 0, error: 0. If error isn't 0, the values or the CRD file is missing.
• make audit puts the files back to the starting point, make down deletes the cluster.

In the UI: make reporter installs Policy Reporter, which shows Kyverno's PolicyReports. Its namespace gets a default-deny from Kyverno too; so it can reach the API server, I put a second CNP next to the generated one without touching it (Cilium merges the two). make ui-reports → http://localhost:8082: with the demo pods in Audit the Dashboard showed 71 pass, 7 fail. make ui-hubble → http://localhost:12000, namespace team-a: requests from demo/curl-pod.yaml to https://example.com are dropped, DNS works.

Sources: Kubernetes docs Dynamic Admission Control, Admission Control, Validating Admission Policy, Mutating Admission Policy; Kyverno's policy library, the Kyverno 1.19.1 Helm notes and the migration to CEL guide, the security vs operability section; Kyverno's CLI tests (test/cli/test-validating-policy/with-namespaceObject-2).

Previous post

Moving Secrets into Vault: From base64 in Git to Vault and ESO Without Downtime

The code for this post: https://github.com/miraccan00/blog-wiki/tree/main/kyverno-cilium-policy-chain