A Kubernetes Secret you create from a manifest had its value stored somewhere beforehand, be it in a Git repository, in a CI variable or on the machine that rendered the manifest. The External Secrets Operator (ESO) reverses the direction. The repository holds only a reference, and a controller in the cluster reads the value from an external store and creates an ordinary Kubernetes Secret from it. That base64 is not encryption was the topic of the article „Kubernetes-Secrets: warum base64 kein Schutz ist“ (German) in this series in June. This article covers the follow-up question, where the value comes from and who rotates it.

What ESO actually does

ESO extends Kubernetes with custom resources that describe where secrets live and how they are synchronized. Three resources carry the model. A SecretStore (bound to one namespace) holds access and authentication details for the backend. A ClusterSecretStore does the same cluster-wide. An ExternalSecret (also namespaced) declares which data to fetch and points to one of the two stores.

The controller locates the store, builds a client with its credentials, fetches the values, creates the Kubernetes Secret and keeps it in sync afterwards. The flow is a pull. The cluster asks the store, and the store writes nothing into the cluster.

Installation

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

The examples below use external-secrets.io/v1, which is what the current documentation shows.

Setting up access to Vault

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: my-namespace
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"        # KV mount
      version: "v2"         # KV engine v1 or v2
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "my-app"
          serviceAccountRef:
            name: "my-sa"

The Vault provider supports several authentication methods, including token, AppRole, Kubernetes, LDAP/UserPass, JWT/OIDC, AWS IAM and TLS certificates. With the token variant, a static token sits in the cluster as a Kubernetes Secret. That is a secret in its own right, and someone has to rotate it. Kubernetes authentication via the ServiceAccount avoids this. The role my-app has to exist on the Vault side for that. Instead of Vault, the same setup works with AWS Secrets Manager (service: SecretsManager).

Requesting the secret

You store the value in Vault, for example with vault kv put secret/my-app password=CHANGE_ME. In the cluster you only describe the reference.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-app-credentials
  namespace: my-namespace
spec:
  refreshInterval: "1h0m0s"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: my-app-credentials
    creationPolicy: Owner
  data:
    - secretKey: db-password
      remoteRef:
        key: my-app
        property: password

The manifest contains no value, only key and property. The key is relative to the path of the store. A secret at secret/my-app appears as my-app in the remoteRef. The status of the resource shows whether the sync worked:

kubectl get externalsecret my-app-credentials -n my-namespace \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}'
# on success: SecretSynced

Rotation and deletion

refreshInterval sets how often the controller reads the value again. Without a setting, the default is 1h0m0s. With 0s the Secret is created once and never updated afterwards. When the value changes in Vault, it arrives in the Kubernetes Secret at the next sync. Whether your application sees it depends on how the Secret is consumed. As an environment variable, the container reads the new value only after a restart. As a volume mount, the kubelet updates the file automatically, with a delay. With subPath mounts, nothing arrives at all.

Two fields govern deletion, and they mean different things. creationPolicy decides who owns the Secret. The default is Owner. If the ExternalSecret resource is deleted, the Secret disappears with it. Orphan creates the Secret without setting an owner and leaves it in place on deletion. Merge merges fields into an existing Secret, None creates nothing.

deletionPolicy, by contrast, concerns the store. The default is Retain. If the values are deleted in Vault, the Kubernetes Secret stays. With Delete, ESO removes the Secret once all data fields are gone at the provider, with Merge only the affected keys. If you clean up in Vault, you have not removed the value from the cluster under the default settings.

One store for several namespaces

A SecretStore cannot reference resources across namespaces. A ClusterSecretStore can be used by ExternalSecret resources in any namespace, which then carry kind: ClusterSecretStore in their secretStoreRef. With conditions (namespace list, selector or regex) you limit who may use it. Without that limit, an ExternalSecret in any namespace is enough to use the store’s access rights. For ServiceAccount or secret-reference authentication, the documentation additionally requires a namespace entry for the ClusterSecretStore.

What ESO does not solve

After the sync, an ordinary Kubernetes Secret exists, and according to Kubernetes it is stored unencrypted in etcd by default. ESO therefore replaces neither encryption at rest (EncryptionConfiguration, optionally with a KMS provider) nor RBAC. Anyone who may read Secrets in the namespace also reads the synchronized value. What shifts is the source of truth, rotation and repository contents. The value is created and changed in one place, and no copy remains in Git or in the CI configuration.

There is also a new trust boundary. The controller needs access to the store, and anyone allowed to use a broadly authorized ClusterSecretStore inherits that access. The Vault role and conditions belong under least privilege just like any other access.

Putting it in context

ESO moves the problem from the manifest into the store and gives you rotation and a central source there. The cost is an additional controller and an additional operational dependency on the store. Protecting the finished Secret inside the cluster remains the job of encryption at rest and RBAC.

Questions or comments on this setup: mm@mhm-dl.de.

Use the newsletter sign-up form to enter your email address, tick the consent box and confirm the sign-up afterwards by email. By signing up you consent to the newsletter covering, among other things, the following topics: training on Docker, Kubernetes, CI/CD, Git workflows and DevSecOps tools, as well as the requirements of NIS2, CRA and DORA and their technical implementation.