Kubernetes Secrets - from Sealed-Secrets to Sops+Age
Right after spinning up my first (and recently retired) Kubernetes cluster, one of the first problems I faced was how to manage secrets for the applications I wanted to run on it. If I had chosen to deploy my cluster with a cloud provider, the simple and obvious choice would be to use whatever secrets management platform they had on offer. I was not afforded that luxury with selfhosting. Likewise I could've created secrets imperatively with kubectl create secret and called it a day. That felt even dirtier than signing up for AWS just to use Secrets Manager. I was going to do it the "right" way: Kubernetes, with blackjack and gitops!
There were a few ways I could solve this thorny problem, including sealed-secrets, sops, and Hashicorp Vault. Vault was so far beyond overkill I couldn't see it over the horizon, and running it to manage a couple of api keys was out of the question. At the time, 1password didn't support api access to secrets in Kubernetes on any sort of plan obtainable by a puny individual subscriber, so integrating directly with my password manager was also out of the question.
Sops, originally created by Mozilla, seemed like a good solution. Pass it a GPG key (or several other formats) and it could encrypt portions of files at rest, making them safe to commit to a git repository. It felt like ansible-vault, but capable of standalone operation. For reasons unknown to me at the time of writing, I decided it was too much headache to manage keys and partial repo encryption, and went looking elsewhere.
Eventually, I landed on the sealed-secrets controller developed by bitnami. It ran a controller inside Kubernetes that generated a key on first boot, and kubernetes secrets were encrypted via a cli tool to store at rest in git. The key was (in theory) unextractable, tying an encrypted secret manifest to a cluster. Even better, sealed-secrets included a CRD for the encrypted secret resource which made secret management with ArgoCD a breeze.
Sealed-Secrets, and Why I Left them Behind
When I only had a single cluster, life was easy. Deploying the controller was a breeze. I created manifests, stored them in git, and wrote a short helper script to manage secret encryption with sealed-secrets. I went on with my day deploying workloads to my shiny new Kubernetes cluster.
#!/usr/bin/env bash
if [; then
fi
if [; then
fi
tmpl_file=""
tmpl_dir=
tmpl_filename=
tmp_file="/.tmp.yaml"
sealed_file="/.sealed.yaml"
# regular expression to check for any base64 encoded values in input file
# if a base64 value is found, assume secret is properly formatted.
# runs with grep -Po, to force perl mode regular expression evaluation for positive lookback support to avoid matching other k8s keys/values
# positive lookback: '(?<=\: )' -> only match if ': ' exists before the base64 text
if ! ; then
| |
else
fi
||
Two years in, I needed to deploy my first workload that needed to be open to the public internet: Nextcloud. Not wanting to open the internet floodgate to my hand-rolled, no automation kubeadm managed house of jank that was my first attempt at Kubernetes, I decided it best to separate fully public workloads from private ones with a dedicated cluster in a separate, isolated network with minimal permissions. Great idea, right?
It was. Except for a neat little conundrum I unwittingly committed future me to solving. My manifests for core cluster components like cert-manager weren't completely portable. The sealed-secrets controller's decryption key couldn't be replicated to the new cluster (without work and a lot of jank), so the easiest path forward was to pull secrets off the private cluster one-by-one and re-encrypt them for the new cluster. (Yes, I could've made new API keys for some of these services, I have now. Not the point here.)
Now, what happens when my third (or fourth) cluster come along? The problem continues to compound. Add on the fact that secrets can't be decrypted locally for inspection and rotating secrets requires accessing every cluster to re-encrypt the template? The friction is unsustainable.
Back to Basics: sops + age
Ironically, I'm looking to the same solution I dismissed early on as "too complex" for salvation. "Too complex" isn't quite right; my past self describing sops as "too difficult" would be more accurate. It was reasonable at the time to pick a tooling ecosystem with a low barrier to entry for which I already had a strong mental model, but now it's time to pick something simple.
Sops on its own won't offer me salvation, but it gets me close. Sops handles encryption and decryption of yaml files on disk (including partial encryption by encrypting only the actually sensitive keys, which is awesome!), but on its own it neither knows or cares about Kubernetes. It needs some help.
Sops also has support for a few different types of encryption keys, but I'll be using age as it's a well-tested modern encryption solution that integrates cleanly with sops. With the incredible ssh-to-age I can even use machine-specific ssh keys to encrypt and decrypt secret data! No sharing keys across every development device!
Kubernetes with ksops
My main concern with switching to sops was integrating my encrypted secrets data with my deployment tools of choice: kustomize and ArgoCD. Luckily, the folks over at Viaduct AI created kustomize-sops (ksops), a kustomize plugin written in go that can handle the decryption of sops secrets during a kustomize build execution. They even included a handy guide for integrating their plugin into ArgoCD via an initContainer.
I won't cover integrations with other tools like Helm here. I already make heavy use of Kustomize in my repository for patches and management of remote manifests from upstream projects. I haven't been publicly vocal about my distaste for Helm before, but at best Helm is a mistake. I won't turn this into a rant, it deserves its own post. Stay tuned for that.
Tangent aside, let's move on to the fun stuff.
Lets Encrypt some Secrets
Any type of secret, not just TLS certs (i'm sorry, the pun was too good to omit).
Let's look at a practical deployment of sops + age, integrated with ArgoCD.
Setting up sops + age on NixOS
I'll walk through how I use sops with Kubernetes, but you don't need to copy my implementation to implement this yourself.
All of my workstations (and soon, all of my servers!) run on NixOS. The "everything is declarative" approach to package management and configuration is amazing for reproducing environments across multiple machines. That extends to nix-shell too, a method of defining a "devshell" specific to a project. I have the following shell.nix file in my Kubernetes manifests/argocd git repo:
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
pkgs.mkShellNoCC {
buildInputs = [
# sops tooling
sops
age
ssh-to-age
kustomize-sops
kustomize
];
# thanks github:starcraft66/infrastructure
KUSTOMIZE_PLUGIN_HOME = pkgs.buildEnv {
name = "kustomize-plugins";
paths = with pkgs; [
kustomize-sops
];
postBuild = ''
mv $out/lib/* $out
rm -r $out/lib
'';
pathsToLink = [ "/lib" ];
};
shellHook = ''
export SOPS_AGE_KEY="$(ssh-to-age -private-key -i $HOME/.ssh/id_ed25519)"
'';
}
This does a few things, and credit to github user starcraft66's infrastructure repo for the KUSTOMIZE_PLUGIN_HOME variable creation example. This nix shell sets up the following:
- installs necessary packages (
sops,age,kustomize,ssh-to-age) - installs
kustomize-sopsand places the go modules in a plugin directory, exported asKUSTOMIZE_PLUGIN_HOME - exports
SOPS_AGE_KEYbased on the current user's ssh key
This can all be configured manually on any linux distribution with the right packages, but checking in the declarative shell definition with my manifests makes my life so much simpler.
Patching ksops
When I first started working on this post, the kustomizes-sops package in nixpkgs was broken. The go module was renamed in v3, and there was a 2 year old PR to address it (also by starcraft66, you rock). At the time, I created a nix derivation for kustomize-sops inside my repo and included it in my shell.nix:
final: prev: {
kustomize-sops = prev.kustomize-sops.overrideAttrs(previousAttrs: {
# Based on unresolved patch here from 2022: https://github.com/NixOS/nixpkgs/pull/175539/files
# module name is expected as ksops/ksops, not ksops-exec/ksops-exec after version 2
installPhase = (previousAttrs.installPhase or "") + ''
mkdir -p $out/lib/viaduct.ai/v1/ksops/
cp $out/lib/viaduct.ai/v1/ksops-exec/ksops-exec $out/lib/viaduct.ai/v1/ksops/ksops
'';
});
}
- { pkgs ? import <nixpkgs> {} }:
+ { pkgs ? import <nixpkgs> { overlays = [ (import ./.nix/overlays/kustomize-sops.nix) ]; } }:
Generating Age Keys
(The keys in this post were only ever used for demonstration purposes. If you want to YOLO, go ahead and use it at your own risk.)
Generating an age key is as simple as running age-keygen with the optional -o output.txt parameter. This will generate a public/private age keypair that we can use with sops:
# created: 2025-11-12T15:11:27-08:00
# public key: age1jg5r9uynw5xp52lyac7nh0e8puxs74yqskh5fx6txd34fnkpkv8qlzzq0w
We can also use ssh-to-age to generate public and private age keys based on an ed25519 ssh key. I'll generate a new ssh key pair and then use it to generate an age key pair here:
)
)
Configuring Sops
Now that we have some keys to work with, we can start looking at sops itself. I'll be reusing the keys generated in the previous section here.
Before we can encrypt anything, sops has to be told what should be considered secret and how to encrypt it. Sops automatically searches for its config file (.sops.yaml) in the current directory tree. Instead of rambling about how the configuration works, let's take a look at an example config file.
keys:
- age1jg5r9uynw5xp52lyac7nh0e8puxs74yqskh5fx6txd34fnkpkv8qlzzq0w
- age1qejhjaqlwncvkhylgjnzssujjxk4mfztz8jxcdmu3frrs3vjte5s2uk8uu
creation_rules:
- unencrypted_regex: "^(apiVersion|metadata|kind|type)$"
path_regex: .*\.(yaml|json|env|ini)$
key_groups:
- age:
- *admin
- *argocd
This basic sops config does two things:
- defines two age public keys as sops identities, named
adminandargocdargocd's key is the plain age key generated earlieradmin's key is the key derived from an ssh key generated earlier
- for any file named
secret*with the extensions.yaml,.json,.env, or.iniin the directorymanifests/, apply the following rules:- encrypt the values of each key in the file such that either the
adminorargocdprivate key can decrypt them - do not encrypt the keys
apiVersion,metadata,kind, ortypeif they're present in the document
- encrypt the values of each key in the file such that either the
Sops's config can go much deeper than this, but that's outside the scope of this post. For now, this is enough to highlight how to encrypt and decrypt secrets for use with Kubernetes. The official documentation for .sops.yaml is here.
Encrypting a Kubernetes Secret
With our sops config defined, we can create a secret manifest for Kubernetes and encrypt it! Let's create a basic secret, and encrypt it with sops:
apiVersion: v1
kind: Secret
metadata:
name: my-secret
type: Opaque
stringData:
SECRET_PASSWORD: hunter2
This is a perfectly valid Kubernetes secret resource, but obviously we wouldn't want to commit this to git in its current state (even though there's no way you could see my password in that yaml file, right?!).
I've saved the secret in the same directory as .sops.yaml, and when I run sops --encrypt, it spits out a yaml file with the contents of stringData encrypted! (I could also use --in-place to overwrite the file instead of directing it to stdout, which is what I would normally do working in my repo).
|
CHm2mmskmgaBNOlNFhll/CsYwEvxPb3ehf1dxj6QP335ShmRK2qo0A==
|
kfS9Bsi0bRA2/xD4W0ChuNgvwmBOLwZcSSJYvnZxp82OZBzoJPbpYQ==
|||)
But wait! there are two blocks containing BEGIN AGE ENCRYPTED FILE, but we only have one piece of encrypted data!
Because both identities are listed in the same key group in .sops.yaml, either key on it's own can decrypt the value. If multiple key groups are present, at least one key from each group must be present to decrypt the data.
Deploying Encrypted Secrets with ArgoCD
Telling Kustomize about our Secrets
Encrypting our secrets is one thing, but deploying them is another. Sops encryption protects the secrets at rest in git but they must be decrypted before they're applied to a Kubernetes cluster. I mentioned ksops above; it's a kustomize plugin that handles on-the-fly decryption of sops-encrypted files when running commands like kustomize build. Since ArgoCD understands kustomize, ksops can handle decryption both on my local machine and in ArgoCD.
Let's take this typical kustomization.yaml file as an example, and add an encrypted secret to it.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- resources/cluster-issuer.yaml
- resources/wildcard-cert.yaml
- resources/cluster-wildcard-cert.yaml
This applies two Certificates and one ClusterIssuer cert-manager resource to create two wildcard TLS certificates for my ingress to use. The ClusterIssuer needs an API key for a supported DNS service like Cloudflare or Digital Ocean to create proof of ownership DNS records for the domain. Let's pretend our test-secret.yaml holds the API key for Cloudflare, and is currently in secrets/test-secret.yaml. If it was added to the resources: block directly in kustomization.yaml, Kubernetes wouldn't understand what to do with it. To tell ksops to decrypt it, we add a secret generator:
# secret-generator.yaml
---
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: secret-generator
files:
- secrets/test-secret.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
+generators:
+ - secret-generator.yaml
resources:
- resources/cluster-issuer.yaml
- resources/wildcard-cert.yaml
- resources/cluster-wildcard-cert.yaml
With the plugin installed, running kustomize build will decrypt the secret and add it to the output.
I thought this was about ArgoCD...
Now that Kustomize can read and decrypt the secrets, it's time to give ArgoCD the same power. The steps remaining are as follows:
- Generate an age key for argocd, different from our other keys
- Add the key to
.sops.yaml - Run
sops updatekeys $filefor each file, or to run against every sops file:
GIT_BASE=
for; do
||
done
- Add the key as a secret to ArgoCD's namespace (you can encrypt it with sops and keep it in the repo, but you must
kubectl apply -n argocd -fthe secret from a decrypted state the first time) - Add the ksops plugin to argocd
The first four steps are pretty self-explanatory and have already been covered above, so those will remain an exercise to the reader.
Instead of rebuilding an argocd image (a valid way to handle this), we'll use an initContainer to install the binary in a volume, and mount that volume into the argocd-repo-server pod. Let's create a Kustomize patch to add ksops support to the argocd repo server, mount our Kubernetes age key secret as a volume, and add the SOPS_AGE_KEY_FILE environment variable.
# argocd/patches/argocd-repo-server-ksops.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
# 1. Define an emptyDir volume which will hold the custom binaries
volumes:
- name: kustomize-plugins
emptyDir:
- name: sops-age
secret:
secretName: sops-age
# 2. Use an init container to download/copy custom binaries into the emptyDir
initContainers:
- name: install-ksops
image: viaductoss/ksops:v4.3.2
command:
args:
- echo "Installing KSOPS...";
mkdir -p /kustomize-plugins/viaduct.ai/v1/ksops/;
mv ksops /kustomize-plugins/viaduct.ai/v1/ksops/ksops;
echo "Done.";
volumeMounts:
- mountPath: /kustomize-plugins
name: kustomize-plugins
# 3. Volume mount the custom binary to the bin directory (overriding the existing version)
containers:
- name: argocd-repo-server
env:
- name: SOPS_AGE_KEY_FILE
value: /secret/age.key
- name: KUSTOMIZE_PLUGIN_HOME
value: /kustomize-plugins
volumeMounts:
- mountPath: /kustomize-plugins
name: kustomize-plugins
#subPath: ksops
- name: sops-age
mountPath: /secret
To add a file patch (which does a strategicMerge), add it like so:
# argocd/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
generators:
- secret-generator.yaml
resources:
- ./custom
- ./external
+patches:
+ - path: patches/argocd-repo-server-ksops.yaml
With a final kustomize build argocd | kubectl apply -f -, ArgoCD can now read sops-encrypted secrets and deploy them like any other resource.