Moving Secrets into Vault: From base64 in Git to Vault and ESO Without Downtime

In the previous post I put ZITADEL's Postgres password and masterkey into Git as base64, and said I did it on purpose. This post moves them into Vault. First, though: deleting the file from Git fixes nothing. The value stays in the history, and git show blog-05:zitadel/db/secrets.yaml brings it back any time. The only thing that actually helps is changing the value.

By the end of this post ZITADEL's three Secrets live in Vault, not in Git. External Secrets Operator (ESO) writes them into Kubernetes, and Git keeps only where each value lives. The Postgres password is changed. While it changed, I sent ZITADEL a request every half second; all 180 came back 200. The old password from Git history now gets FATAL: password authentication failed. The one secret from 05 that was never in Git, Argo CD's client secret in ZITADEL, goes the same way and rotates in 0.7 seconds.

I also owe a correction. At the end of 05 I wrote "change every one of them on the way". For the masterkey that isn't possible. ZITADEL encrypts its data with that key; a new key can't read the old data. The masterkey moves into Vault unchanged, and Step 3 says so plainly.

The lab is in vault-eso-secret-migration. The platform side is branch blog-05b of platform-gitops. Its three commits are the three steps of the migration, and I go through each one below. blog-05 is untouched; the previous post's lab still runs from it.

What you need

• The previous post's lab: Argo CD SSO Integration: OIDC and RBAC with ZITADEL. If you have a running 05 cluster, make from-05 moves it to blog-05b. If not, this lab's Makefile builds everything from scratch.
• 4 CPU / 8 GB Docker (Colima, Apple Silicon). Vault and ESO add about 180 MiB on top of 05. Measured working set: vault-0 70 MiB, ESO controller 33, webhook 26, cert-controller 51.
• kind, kubectl, helm 3.15+, jq, curl, openssl.
• Versions in this run: hashicorp/vault chart 0.34.1 (Vault 2.0.4), external-secrets chart 2.11.0 (ESO v2.11.0, API external-secrets.io/v1), argo/argo-cd chart 10.9.2 (Argo CD v3.5.3), zitadel/zitadel chart 10.1.0 (ZITADEL v4.19).

Why this way

Change the value, not just the file. Once a secret has been in Git, treat everyone who could read the repo as having it. You can rewrite history with git filter-repo, but that doesn't reach clones, forks or CI caches. So the real step of this migration isn't the move, it's the rotation.

ESO, not the Vault Agent injector. ESO turns the value in Vault into an ordinary Kubernetes Secret. The Secret names stay the same, so I didn't touch ZITADEL's chart or the Postgres StatefulSet; both still read zitadel-db and zitadel-postgres. The injector adds a sidecar to every pod and writes the value to a file, which means changing how ZITADEL starts. The cost: the value still sits in etcd as a Secret. You still need etcd encryption and RBAC; ESO replaces neither.

Not SOPS or Sealed Secrets. Both keep the encrypted value in Git. Rotation is still a commit, and if the key leaks, every value in the history opens. In Vault every value has a version history, a rollback is one command, and who read what is recorded in one place.

Vault inside the cluster, one node. For the lab. In production Vault runs outside the cluster on 3 VMs; the key is split into 5 shares and any 3 open it (Shamir). Init and unseal are the same steps either way, and the lab uses 5/3 too. A licensing note: Vault has been under BSL 1.1 since 1.15; the open source fork under the Linux Foundation is OpenBao. I didn't try OpenBao in this lab.

One difference from production. Our production runs ESO 0.10.4 with external-secrets.io/v1beta1. This lab runs 2.11 with v1. I didn't try these manifests against v1beta1; when applying them to an older ESO, check the fields with kubectl explain externalsecret.spec.

Step 1 — Argo CD installs Vault and ESO

The first commit on blog-05b (3011bfd) adds three Applications to apps/:

root (Application)
├── platform, products (AppProject, wave -2)
├── argocd (wave -1)
├── applicationsets, external-secrets (wave 0)   → the ESO operator and its CRDs
├── zitadel-db, vault, external-secrets-store (wave 1)
└── zitadel (wave 2)

external-secrets-store is its own Application because the ClusterSecretStore CRD comes with external-secrets. In one Application, Argo CD would try to apply the store before the CRD exists. The ESO Application has ServerSideApply=true: its CRDs are larger than the 256 KiB last-applied annotation that kubectl apply allows.

Vault's values file is short:

# platform-gitops/vault/values.yaml
injector:
  enabled: false         # secrets reach pods through ESO, not the sidecar injector
server:
  ha:
    enabled: true
    replicas: 1
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable     = 1        # lab: port-forward only; production terminates TLS on Vault itself
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
        }
        storage "raft" {
          path = "/vault/data"
        }
        service_registration "kubernetes" {}

ha.enabled: true with replicas: 1 looks contradictory. In this chart, raft storage lives under HA mode; one replica means a raft cluster of one node. Going to production changes only the replica count.

Argo CD installs Vault but doesn't open it. On first start Vault is sealed: it holds no encryption key, answers nothing, and its pod is 0/1 Ready. Init and unseal are a human's job:

$ make vault-init
initialized, keys in .lab/vault-init.json
Sealed                  false
Version                 2.0.4
Storage Type            raft
HA Mode                 active

.lab/vault-init.json holds the 5 key shares and the root token. In the lab it sits in .lab/, is gitignored, and make down deletes it. In production the 5 shares go to 5 different people and the root token is revoked after setup.

Then, how ESO gets into Vault:

$ make vault-config
Success! Enabled the kv-v2 secrets engine at: secret/
Success! Enabled kubernetes auth method at: kubernetes/
Success! Data written to: auth/kubernetes/config
Success! Uploaded policy: eso-read
Success! Data written to: auth/kubernetes/role/eso

# scripts/vault-config.sh — ESO only reads
path "secret/data/platform/*"     { capabilities = ["read"] }
path "secret/metadata/platform/*" { capabilities = ["read", "list"] }

ESO logs in to Vault as its own ServiceAccount: Kubernetes auth checks the ServiceAccount token with the Kubernetes API. No Vault token is stored anywhere. The policy on the role only reads. Writing to Vault is a human's job, not the cluster's.

# platform-gitops/external-secrets/store/cluster-secret-store.yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: http://vault-active.vault.svc:8200
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: eso
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
            audiences: [vault]

$ kubectl get clustersecretstore vault
NAME    AGE   STATUS   CAPABILITIES   READY
vault   8s    Valid    ReadWrite      True

ReadWrite is the store's capability list: what the provider supports, not what ESO is allowed to do. The real permission is the eso-read policy in Vault.

In the UI: make vault-ui → http://localhost:8200, token jq -r .root_token .lab/vault-init.json. Under secret/platform/ there are two folders: ZITADEL's values and Argo CD's OIDC client.

Step 2 — Side by side: same values, different names

In the first round the values went into Vault unchanged. They're read from the blog-05 branch, which is where they leak from anyway:

# scripts/seed-from-git.sh (core)
RAW=https://raw.githubusercontent.com/miraccan00/platform-gitops/blog-05/zitadel/db/secrets.yaml
v kv put secret/platform/zitadel/db POSTGRES_USER=... POSTGRES_PASSWORD=... POSTGRES_DB=... ZITADEL_DATABASE_POSTGRES_DSN=...
v kv put secret/platform/zitadel/masterkey masterkey=...

Then one ExternalSecret per Secret, but with a different target name: zitadel-db-eso instead of zitadel-db. Nothing reads these copies. They exist only to prove that Vault → ESO → Secret produces exactly what's in Git. To compare without printing values, I looked at the first 12 characters of each key's SHA-256:

$ bash scripts/compare.sh zitadel zitadel-postgres zitadel-postgres-eso
== zitadel-postgres
  POSTGRES_DB                      713824a0429e
  POSTGRES_PASSWORD                f05fca35a4c1
  POSTGRES_USER                    713824a0429e
== zitadel-postgres-eso
  POSTGRES_DB                      713824a0429e
  POSTGRES_PASSWORD                f05fca35a4c1
  POSTGRES_USER                    713824a0429e

zitadel-db's DSN (db9451511e8d) and the masterkey (c4579c585107) matched too. POSTGRES_DB and POSTGRES_USER share a hash because both are zitadel.

What this step buys you: if something is wrong (a wrong path, a missing key, ESO unable to log in to Vault) you see it here, with nothing broken.

Step 3 — The cut: Secrets leave Git, names stay

The second commit (e872343) does two things: it deletes zitadel/db/secrets.yaml and points the ExternalSecrets at the real names. The values are still the same, so from ZITADEL's side nothing should change.

# platform-gitops/zitadel/db/externalsecrets.yaml (one of three)
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: zitadel-db
  namespace: zitadel
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  refreshInterval: 1h
  secretStoreRef: {kind: ClusterSecretStore, name: vault}
  target: {name: zitadel-db}
  data:
    - {secretKey: ZITADEL_DATABASE_POSTGRES_DSN, remoteRef: {key: platform/zitadel/db, property: ZITADEL_DATABASE_POSTGRES_DSN}}

SkipDryRunOnMissingResource is for a clean cluster. zitadel-db is in wave 1; if ESO's CRD doesn't exist yet at that moment, Argo CD's dry run would stop the whole sync with "no such resource type". The Application got a retry as well.

The masterkey moves to an ExternalSecret the same way, with its value unchanged. The comment in the file says it: ZITADEL encrypts its data with this key, a new one can't read the old data. So the masterkey in Git history stays valid. Someone who could read the repo and also gets a copy of the database could decrypt ZITADEL's encrypted fields. The defence is closing access to Postgres and changing its password; both are in this post.

Argo CD's sync result after the push:

Secret zitadel-masterkey Pruned pruned
Secret zitadel-postgres Pruned pruned
Secret zitadel-db Pruned pruned
ExternalSecret zitadel-postgres-eso Pruned pruned
ExternalSecret zitadel-masterkey-eso Pruned pruned
ExternalSecret zitadel-db-eso Pruned pruned
ExternalSecret zitadel-db Synced externalsecret.external-secrets.io/zitadel-db created
ExternalSecret zitadel-masterkey Synced externalsecret.external-secrets.io/zitadel-masterkey created
ExternalSecret zitadel-postgres Synced externalsecret.external-secrets.io/zitadel-postgres created

Argo CD deleted the three Secrets that left Git. ESO created new ones with the same names in the same second (17:58:57): new UIDs, owned by the ExternalSecrets now. The ZITADEL pod didn't restart, because it had read its environment at start. But there was a brief moment with no Secret at all. A pod restarting right then would have hit CreateContainerConfigError. What went wrong #1 is how I'd avoid that.

In the UI: in the zitadel-db Application's tree each ExternalSecret has the Secret it produces under it. Argo CD reads that relation from the Secret's ownerReference:

Step 4 — Rotation: the Postgres password, no downtime

The value is in Vault now. Next, change it. Order matters, because two sides know the password: Postgres and ZITADEL's DSN.

# scripts/rotate-db-password.sh (core)
NEW=$(openssl rand -hex 24)

# 1. Postgres. Inside the pod psql connects over the local socket without a password.
kubectl -n zitadel exec zitadel-postgres-0 -- psql -U zitadel -d zitadel -qc "ALTER USER zitadel WITH PASSWORD '$NEW'"

# 2. Vault. patch, not put: put replaces the whole secret and drops every key you didn't repeat.
v kv patch secret/platform/zitadel/db POSTGRES_PASSWORD="$NEW" \
  ZITADEL_DATABASE_POSTGRES_DSN="postgresql://zitadel:[email protected]:5432/zitadel?sslmode=disable"

# 3. ESO. Don't wait for refreshInterval (1h): any change to this annotation triggers a sync now.
kubectl -n zitadel annotate es zitadel-db zitadel-postgres force-sync="$(date +%s)" --overwrite

# 4. ZITADEL reads the DSN from env only at start.
kubectl -n zitadel rollout restart deploy/zitadel

Why Postgres first: Postgres checks the password only when a new connection opens. After ALTER USER, ZITADEL's open connections keep working, and the old pod serves until the pod with the new DSN is Ready. The other way round, starting ZITADEL with the new DSN first, the new pod would log in with a password Postgres doesn't know yet.

refreshInterval: 1h is a choice. ESO checks the value in Vault every hour. In production we use 1 hour on 68 ExternalSecrets and 1 minute on the 2 that change often. If you don't want to wait an hour during a rotation, the force-sync annotation runs ESO now.

During the rotation two probes hit ZITADEL every half second: one on /debug/ready, one on /oauth/v2/keys, which goes to the database:

$ make rotate-db
deployment.apps/zitadel restarted
Waiting for deployment "zitadel" rollout to finish: 1 old replicas are pending termination...
deployment "zitadel" successfully rolled out
rotated; Vault version: 2
bash scripts/rotate-db-password.sh  0.28s user 0.15s system 13% cpu 3.296 total

== probe: 0 non-200 / 110
== probe-keys: 0 non-200 / 70

The new pod opened 10 connections with the new password at 18:02:09 (pg_stat_activity). None of the 180 requests failed.

On publication day I rebuilt the lab on a clean cluster and ran the rotation four more times. One of 422 requests, to /debug/ready, didn't come back 200. I didn't capture its code; the probe pod was already gone. Not an outage, but "zero failures" isn't guaranteed on every run either. ZITADEL runs one replica here, and a request that arrives while the old pod shuts down can fall through. I haven't measured it with two replicas. Then the real question: does the password in Git history still open anything?

$ OLD=$(git show origin/blog-05:zitadel/db/secrets.yaml | awk '/POSTGRES_PASSWORD:/{print $2}' | base64 -d)
$ kubectl -n zitadel exec zitadel-postgres-0 -- env PGPASSWORD="$OLD" \
    psql -h zitadel-postgres.zitadel.svc -U zitadel -d zitadel -c 'select 1'
psql: error: connection to server at "zitadel-postgres.zitadel.svc" (10.96.70.87), port 5432 failed: FATAL:  password authentication failed for user "zitadel"

It doesn't.

A side note: ESO updated the zitadel-postgres Secret too, and the Postgres StatefulSet reads POSTGRES_PASSWORD from it. But the postgres image uses that variable only on the first initdb, when the data directory is empty. So the Postgres pod didn't need a restart. ALTER USER is what actually changes the password.

In the UI: Vault masks the values, and every change is a new version:

Version 1 is the value from Git. vault kv rollback -version=1 brings it back if needed. I saw that work in #2, undoing my own mistake.

Step 5 — The one secret that was never in Git

In 05, ZITADEL created an OIDC app for Argo CD and generated its client secret. zitadel-setup.sh wrote that straight into the argocd/argocd-oidc-zitadel Secret. It wasn't in Git, but it wasn't recorded anywhere either: when the cluster went, the secret went with it, and rotating it lived in one script.

The third commit (f2e4cd4) wires it to ESO too:

# platform-gitops/argocd/secrets/oidc-zitadel.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: argocd-oidc-zitadel
  namespace: argocd
spec:
  refreshInterval: 1h
  secretStoreRef: {kind: ClusterSecretStore, name: vault}
  target:
    name: argocd-oidc-zitadel
    template:
      metadata:
        labels:
          app.kubernetes.io/part-of: argocd
  dataFrom:
    - extract: {key: platform/argocd/oidc-zitadel}

template.metadata.labels is required. Argo CD resolves $argocd-oidc-zitadel:clientSecret in oidc.config only from Secrets labeled app.kubernetes.io/part-of: argocd, the same label whose deletion took Argo CD down in 04. template.metadata is what puts it on the Secret.

dataFrom.extract takes every key at the Vault path (clientID, clientSecret, cliClientID). Convenient, with a cost: if a key disappears from the path, it disappears from the Secret silently. #2 is exactly that.

I noticed an interesting difference here. This Secret had been created by a script, not by Argo CD. ESO didn't delete and recreate it; it took it over in place: same UID (676ed02b…), event Updated: secret updated. The deletion in Step 3 came from Argo CD's prune, not from ESO.

The rotation runs the other way round, because ZITADEL generates the new secret:

# scripts/rotate-oidc-secret.sh (core)
secret=$(api POST "/management/v1/projects/$project/apps/$app/oidc_config/_generate_client_secret" '{}' | jq -r .clientSecret)
v kv patch secret/platform/argocd/oidc-zitadel clientSecret="$secret"
kubectl -n argocd annotate es argocd-oidc-zitadel force-sync="$(date +%s)" --overwrite

$ make rotate-oidc
rotated; Vault version: 2

The SHA-256 prefix of clientSecret in the Secret went from 934c2ab6e5fb to 4c8dca6d00b0.

ZITADEL invalidates the old secret the moment it generates the new one. The script took 0.7 seconds; a login started in that window would have failed. Existing sessions aren't affected, because Argo CD uses the client secret only at login. argocd-server wasn't restarted; Argo CD picked up the change in the Secret on its own, and alice logged in right away:

{"userinfo":{"loggedIn":true,"username":"[email protected]","iss":"http://zitadel.127.0.0.1.nip.io:8081","groups":["platform-admins"]},"canSync":{"value":"yes"}, ...}

In the UI: the ExternalSecret and its Secret in the argocd-secrets Application; three versions of the same path in Vault. On a clean install those are setup (1), cliClientID (2) and the rotation (3):

What went wrong

1. There's a gap between Argo CD's prune and ESO's create. In Step 3 the Secrets didn't exist for a moment, because Argo CD deleted them. As far as I could measure it was within the same second, but it isn't zero. ZITADEL had one replica and it didn't restart in that moment; I was lucky.

Step 5 showed why: ESO takes over a Secret that Argo CD doesn't track in place. Next time I'd do it in two commits. The first adds argocd.argoproj.io/sync-options: Prune=false to the Secrets in Git, the second removes them from the file. Argo CD then leaves the Secret alone and ESO takes it over in place. I didn't try this order in the lab; if you're migrating a running 05 cluster, try it once and check that the UID doesn't change.

2. kv put deleted a key and broke the Argo CD login. In production I once used vault kv put to add a key to an existing path. put replaces everything at the path; it deleted the repo access token stored there and Argo CD lost access to the repo. I repeated it on purpose in the lab:

$ vault kv put secret/platform/argocd/oidc-zitadel clientSecret="$CS"
version            3
$ kubectl -n argocd get secret argocd-oidc-zitadel -o json | jq -c '.data|keys'
["clientSecret"]

clientID and cliClientID were gone. Argo CD didn't fail; it logged a warning and sent the reference as is as the client ID:

level=warning msg="secret key does not exist in secret"
level=info msg="Performing authorization_code flow login: http://zitadel.127.0.0.1.nip.io:8081/oauth/v2/authorize?client_id=%24argocd-oidc-zitadel%3AclientID&..."

And ZITADEL found no app by that name:

HTTP/1.1 400 Bad Request
{"error":"invalid_request","error_description":"Errors.App.NotFound"}

The fix is one command, because Vault keeps the old version:

$ vault kv rollback -version=2 secret/platform/argocd/oidc-zitadel
version            4
$ kubectl -n argocd annotate es argocd-oidc-zitadel force-sync="$(date +%s)" --overwrite

Rollback doesn't delete anything; it writes the old version as a new one (4) and the history stays. The rule: put when you create a path, patch every time after that. The scripts follow it.

3. On a clean install one Application stays Degraded and the wait script never ends. make vault waited for 13 Applications to turn green, and after 10 minutes it was still waiting. argocd-secrets was Degraded, because the OIDC client is created in ZITADEL by make setup and the Vault path doesn't exist until then. ESO's status: SecretSyncedError. In production I had a worse version of this: one missing key in Vault left an ExternalSecret in an early sync wave Degraded, and none of the later waves were applied. In the lab, wait-apps.sh got a SKIP= and make vault doesn't wait for argocd-secrets. The production lesson: before an install, list the keys that have to be in Vault.

4. vault status returns an error code while sealed. The first version of the init script died on its first line:

command terminated with exit code 2

vault status exits with 2 for a sealed Vault. Under set -e the script took that as a failure and stopped, while "sealed" is the very answer it was asking for. The script now reads the status as JSON and ignores the exit code.

5. The role had no audience. vault write auth/kubernetes/role/eso succeeded, but with a warning:

WARNING! The following warnings were returned from Vault:
  * Role eso does not have an audience configured.

Without an audience, Vault accepts a ServiceAccount token issued for any purpose in the same cluster. I added audience=vault to the role and audiences: [vault] to the store; ESO now logs in only with a token issued for Vault.

6. Back-to-back rotations, the second one failed. make rotate-db right before make rotate-oidc:

make: *** [rotate-oidc] Error 52

curl exit code 52 means an empty reply. kubectl port-forward doesn't connect to a Service but to one pod behind it. When rotate-db restarted ZITADEL, the pod the port-forward was attached to was gone, and the make zitadel-ui loop takes a second or two to reconnect. rotate-oidc-secret.sh now waits for /debug/ready first. It's another face of the port-forward trouble in 05 (#5).

Verify

make status                                       # 13 Applications Synced/Healthy, Sealed false, 4 ExternalSecrets SecretSynced
kubectl -n zitadel get secret zitadel-db -o jsonpath='{.metadata.ownerReferences[0].kind}'   # ExternalSecret
git show origin/blog-05b:zitadel/db/ 2>/dev/null | grep secrets.yaml || echo "no secrets.yaml on blog-05b"
make rotate-db                                    # "rotated; Vault version: N", ZITADEL stays up
make rotate-oidc                                  # then log in to Argo CD through ZITADEL

To see the old password fail, the psql command from Step 4: FATAL: password authentication failed.

In the UI:

• Argo CD: 13 Applications, all Synced/Healthy on blog-05b.
• In the zitadel-db and argocd-secrets trees, a Secret under each ExternalSecret.
• Vault: every rotation adds a row to Version History.
• Log in as alice / dave / bob: the permissions from 05 hold (sync product-helloapi-dev: 200 / 200 / 403).

Sources: hashicorp/vault chart 0.34.1 values (server.ha.raft, injector), Vault docs Kubernetes auth method and KV v2 (kv patch, kv rollback), External Secrets docs HashiCorp Vault provider, ExternalSecret (refreshInterval, target.template, dataFrom.extract) and the force-sync annotation, Argo CD docs Sync Options (Prune=false, SkipDryRunOnMissingResource, ServerSideApply) and User Management (the $secret:key reference and the part-of label), the postgres Docker image's note on POSTGRES_PASSWORD.

What changed after this post

In this series

Previous: Argo CD SSO Integration: OIDC and RBAC with ZITADEL · Next: Talos on Hetzner with Terraform: 17 Nodes, Zero SSH

The code for this post: https://github.com/miraccan00/blog-wiki/tree/main/vault-eso-secret-migration