Back to blog

Security

Syncing Secrets into Kubernetes with the External Secrets Operator

Use External Secrets Operator to sync AWS Secrets Manager values into Kubernetes with IRSA, sane refresh policies, and less secret material in git.

September 15, 2024 Platform Engineering 9 min read

If you want applications in Kubernetes to consume secrets without committing those secrets to git, External Secrets Operator is one of the cleanest ways to do it. It keeps the source of truth in a real secret manager such as AWS Secrets Manager or Vault, then reconciles the values your workloads need into ordinary Kubernetes Secret objects. As of External Secrets Operator v2.7.0, that gives you a practical middle ground: application teams still use familiar secretKeyRef and volume mounts, while platform teams stop scattering ciphertext, bootstrap scripts, and long-lived cloud credentials across repositories.

The important distinction is where the truth lives. With Sealed Secrets or SOPS, the encrypted payload still lives in git and the cluster eventually decrypts it. With External Secrets Operator (ESO), git holds references and policy, not the secret value itself. That changes the operating model, and it changes the failure modes too.

The mental model: ESO is a sync loop, not a vault

ESO runs as a controller in the cluster. You give it a SecretStore or ClusterSecretStore that explains how to talk to an external backend, then you create ExternalSecret objects that say which remote values to fetch and what Kubernetes Secret to write.

That split is what makes the model predictable:

  • SecretStore is namespaced and scoped to one namespace.
  • ClusterSecretStore is cluster-wide and can be referenced from any namespace.
  • ExternalSecret is the workload-facing object that maps remote keys into the exact Secret shape the application expects.

As of v2.7.0, ExternalSecret uses apiVersion: external-secrets.io/v1, and the default refresh policy is Periodic. If you set refreshInterval, ESO reconciles the target secret on that cadence. If you switch to OnChange, it only updates when the ExternalSecret object changes. If you use CreatedOnce, it creates the Kubernetes secret and then leaves it alone unless the target secret is deleted.

That is why ESO fits best when you already trust an external secret manager to be the system of record. It is not trying to replace Secrets Manager or Vault. It is the bridge between those systems and ordinary Kubernetes consumption patterns.

One working setup: AWS Secrets Manager on EKS with IRSA

For EKS, the cleanest authentication path is IRSA rather than static AWS keys. The ESO docs support both controller pod identity and explicit jwt.serviceAccountRef authentication. Either way, the point is the same: the controller or the referenced service account gets short-lived AWS credentials from the cluster’s OIDC integration, not from a long-lived access key stored in another Kubernetes secret.

A minimal setup looks like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-aws
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/payments-eso-reader
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: payments-aws-secrets
  namespace: payments
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-2
      auth:
        jwt:
          serviceAccountRef:
            name: eso-aws
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: api-db-credentials
  namespace: payments
spec:
  refreshPolicy: Periodic
  refreshInterval: 6h
  secretStoreRef:
    name: payments-aws-secrets
    kind: SecretStore
  target:
    name: api-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/payments/api/postgres
        property: username
    - secretKey: password
      remoteRef:
        key: prod/payments/api/postgres
        property: password
    - secretKey: host
      remoteRef:
        key: prod/payments/api/postgres
        property: host

There are a few useful details packed into that example.

First, the AWS IAM policy should be narrower than *. The ESO AWS provider docs explicitly recommend fine-grained roles. In practice, that means allowing secretsmanager:GetSecretValue and related read calls only for the path or ARN prefix that namespace or application actually needs.

Second, the refreshInterval is part of the design, not a cosmetic field. Six hours is a reasonable starting point for credentials that rotate on a human timescale. It is long enough to avoid hammering AWS Secrets Manager for no reason, and short enough that a normal credential rotation does not wait until the next deploy.

Third, the application still reads an ordinary Kubernetes secret. From the workload’s point of view, nothing special is happening. That matters because most applications already know how to consume secretKeyRef and mounted secret files; they do not know how to speak to AWS Secrets Manager directly.

Templating is where ESO becomes genuinely useful

A lot of applications do not want three discrete keys called username, password, and host. They want a connection string, a config file, or a secret with a very particular key layout.

That is where spec.target.template earns its keep. ESO’s templating support lets you fetch provider values once, then render them into the exact Kubernetes Secret structure the workload expects.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: api-config
  namespace: payments
spec:
  refreshPolicy: Periodic
  refreshInterval: 6h
  secretStoreRef:
    name: payments-aws-secrets
    kind: SecretStore
  target:
    name: api-config
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        DATABASE_URL: 'postgresql://{{ .username }}:***@{{ .host }}:5432/payments'
        config.yaml: |
          database:
            url: postgresql://{{ .username }}:***@{{ .host }}:5432/payments
          cache:
            ttlSeconds: 30
  data:
    - secretKey: username
      remoteRef:
        key: prod/payments/api/postgres
        property: username
    - secretKey: password
      remoteRef:
        key: prod/payments/api/postgres
        property: password
    - secretKey: host
      remoteRef:
        key: prod/payments/api/postgres
        property: host

That keeps the secret-manager side tidy while still letting the application get a shape it can consume without extra init scripts or a sidecar rewriting files at startup.

There is one operational catch worth stating plainly: updating the Kubernetes Secret does not guarantee the application has re-read it. Kubernetes updates secret volumes eventually, but a container that consumes a secret as an environment variable does not see updates until it restarts. If your application reads credentials from environment variables, rotation still needs a restart path.

What changes compared to Sealed Secrets and SOPS

Sealed Secrets and SOPS solve a different problem.

Bitnami’s Sealed Secrets is explicitly about making an encrypted secret safe to store in git. SOPS is an editor for encrypted YAML, JSON, ENV, and related formats. Both are useful when the repository is still the place where secret material lives, just in encrypted form.

ESO changes that model. The repo stores references, refresh policy, and secret shape, but the value stays in AWS Secrets Manager, Vault, or another external backend until the controller syncs it into the cluster.

That changes the threat model in a few important ways:

  • A leaked git repository is less valuable because it does not contain the ciphertext or plaintext secret itself.
  • Access control moves toward IAM, backend secret-manager policy, and ESO RBAC rather than encryption tooling in CI.
  • Rotation gets easier because changing the upstream secret can be enough; you are not re-encrypting and recommitting every value update.
  • The cluster still ends up with a normal Kubernetes Secret, so you have not removed Kubernetes secret security concerns. You have changed the upstream source of truth.

That last point matters. Kubernetes’ own docs are blunt: Secret objects are not magically safe just because they are called secrets. If someone can read secrets in the namespace, or if your cluster does not encrypt secrets at rest, ESO does not fix that for you. It only removes one class of sprawl: secret values travelling through git and deployment pipelines.

The production reality: where ESO bites back

The happy path is straightforward. The sharp edges are all about scope and cadence.

ClusterSecretStore increases blast radius quickly

A namespaced SecretStore is usually the safer default. The ESO docs recommend caution with cluster-scoped stores because they can be referenced across namespaces. If you really do need a ClusterSecretStore, put conditions on it with namespaceSelector or an explicit namespace list, and keep the backing IAM role narrow.

A cluster-wide store plus a broad IAM policy is the easiest way to turn one controller into a cross-cluster secret broker you did not mean to build.

Short refresh intervals can turn into expensive noise

Every ExternalSecret on a very short interval means more reconciles and more backend API traffic. The ESO AWS provider docs call this out directly for Parameter Store, and the same principle applies to Secrets Manager: aggressive polling adds cost and raises your odds of rate limits without making applications meaningfully safer.

Pick the interval that matches the secret’s rotation behaviour. For many database credentials, hours make more sense than minutes.

Pod restarts are still your problem

If the application reads a mounted secret file and can reload it, great. If it reads environment variables once at process start, ESO alone is not the full rotation story. Pair it with a restart controller, an explicit rollout, or an application that can re-read config without a restart.

Auth should stay keyless

ESO supports static AWS access keys through secretRef, but on EKS that should be the fallback, not the plan. IRSA or EKS Pod Identity keeps cloud credentials short-lived and cuts out another Kubernetes secret full of long-lived credentials.

When ESO is the right call

ESO is a good fit when you already have a proper secret manager, want less secret material in git, and still need applications to consume standard Kubernetes Secret objects.

It is a weaker fit when the only thing you need is encrypted configuration checked into git and you do not want another controller in the cluster. In that case, SOPS or Sealed Secrets may be the simpler tool because they solve the narrower problem directly.

The useful way to think about it is this: Sealed Secrets and SOPS protect secrets on the way into the cluster. ESO manages the handoff from an external source of truth into the cluster. If your platform already centres on AWS Secrets Manager or Vault, ESO is usually the more natural shape.

The main lesson is not that ESO makes secrets easy. Secrets are still awkward. It makes the awkwardness live in a place that scales better: IAM policy, backend secret-manager ownership, and a controller whose job is only to synchronise values into the format Kubernetes workloads already understand.