kubernetes kubeseal
===================

In my homelab kubernetes cluster I am using kubeseal to encrypt secrets. I have been using it successfully for a few months now wtih great success. It allows...

Date: March 27, 2024

In my homelab kubernetes cluster I am using kubeseal to encrypt secrets. I
have been using it successfully for a few months now wtih great success. It
allows me to commit all of my secrets manifests to git with out risk of leaking
secrets.

You see kubeseal encrypts your secrets with a private key only stored in your
cluster, so only the cluster itself can decrypt them using the kubeseal
controller.

![kubeseal-post.png](https://dropper.waylonwalker.com/api/file/833e8681-8220-4096-b211-80c33eb10c1c.png)

## KubeSeal

[https://sealed-secrets.netlify.app/](https://sealed-secrets.netlify.app/){.hoverlink}

 [1]

## installation

Installation happens in two steps. You need the kubernetes controller and the
client side cli to create a sealed secret.

For a more complete instruction see the
[docs#installation](]

## installation - controller

!!! warning
 **context**
 Make sure that you are in the right context before running any kubectl commands.

``` bash
kubectl config current-context
```

sealed-secrets is installed using the helm package manager. To install
sealed-secrets run the following command.

``` bash
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller sealed-secrets/sealed-secrets
```

## installation - client

For the client you can check your OS package manager, brew, or the
[github-releases](https://github.com/bitnami-labs/sealed-secrets/releases/).
For me I found it in the main arch repos.

``` bash
paru -S kubeseal
# or
sudo pacman -S kubeseal
# or
brew install kubeseal
```

!!! note
 You will need to install kubeseal on every device that you will want to
 create sealed secrets on.

## Example

Most of these commands come straight from the docs. From my experience I have
always specified the namespace, my projects go per namespace and I don't have
any reason that other namepsaces should see the secret, and if they do I deploy
another secret in that namespace.

``` bash
# Create a json/yaml-encoded Secret somehow:
# (note use of `--dry-run` - this is just a local file!)
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o yaml -n thenamespace > mysecret.yaml
```

> note that the key of the secret is `foo` and the value is `bar`

results

``` yaml
apiVersion: v1
data:
 foo: YmFy
kind: Secret
metadata:
 creationTimestamp: null
 name: mysecret
 namespace: thenamespace
```

!!! note
 The data is base64 encoded.

 ``` bash
 echo -n bar | base64
 # YmFy
 ```

``` bash
# This is the important bit:
kubeseal -f mysecret.yaml -w mysealedsecret.yaml

# At this point mysealedsecret.json is safe to upload to Github,
# post on Twitter, etc.

# Eventually:
kubectl create -f mysealedsecret.yaml -n thenamespace
# sealedsecret.bitnami.com/mysecret created

# Profit!
kubectl get secret mysecret
kubectl get secret mysecret -n thenamespace
# NAME TYPE DATA AGE
# mysecret Opaque 1 27s

cat mysealedsecret.yaml | kubeseal --validate

```

``` bash
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o yaml \
 | kubeseal -o yaml -n thenamespace > mysealedsecret.yaml
echo -n baz | kubectl create secret generic mysecret --dry-run=client --from-file=bar=/dev/stdin -o yaml \
 | kubeseal -o yaml -n thenamespace --merge-into mysealedsecret.yaml
```

Results

``` yaml
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
 creationTimestamp: null
 name: mysecret
 namespace: thenamespace
spec:
 encryptedData:
 bar: AgBLkamltcLfH1dC1JxQ3qd8lJ8aBZF2ARoq3uo055hnzXOy8g2T5liTx5UPvyPV8yyWqABU8eOnwjNhDtzSATvYeBB3fGkucdOZziWEoiNsWTR9ZtFEkod7Ya6uGkzZOJwi3IkrHFVIT9oWZQUxxJZ6vFhPiFcx9Dorr8TNSzG4KOug25+PhWPPiHDgSup5N3CkWCZaYOF7dbZRVSA4nGP1fZxjFByHP4AsdjLCHptyVbkpLRKeiXTkLxfLX4K+JLZGM41S1On5bSP56mCfv1daTJx619kDXkRLw9l21Ot283/L0NMNAiw781AefYMVoO3aHmYgcT6wAtsQAKje9fyL7DQRHt8a5NZOWukp/P6XjdXRz/nfQasQlbSTrRkDpplKIM5/WdPcBoKi+yyoOL0rZ8x1X7YzUI3BggZmzWyEPD01BK1YAHGZnYIZbbCy1JSm8JCBvP+xWMg+i0Z/DCD8nclAhH1GX2Q7/NrNHF//589AJfuriymd2+mk7uaLA4RRsY0l5QeZD6HVAqSv5jWsVQQtSftWmI9vn9oL/Pno7sEUjSDpXPfF4nnsULhxsPEe2DFAMm1kZAjdF06ueF4/x2Fdy80ZQNyycaDx2CWm4z3b14A75WGyOXl2wJZQqxrFCz8el4hD2nH3zQFEzd6AIh49myqVAGuu2qGlYP4p94LJghVa+mQjztLD/2ZUZjY+anQ=
 foo: AgAducXW6iUCY900cPDdmRfuj7tKnh2hY4C1+2hFoAtjyvjepsKNWsiPJ81t8anaMfFPat4ta060l5VtTrceFE8oS/rViz1tvNWGPBjL+GwL/QjkGl6H8ju87vKERKQn5qw7B/V5j24VM8AxOd+/vJNt/IeRIHLubvFft4hyMq4b0xmIxaemBSTxchQX/5364T3VJH2kHaqpqd+JJgQnTbiTQe/XnyZokDX8GSxw4rAbJUJSRUtY9DB9ZDu2zC5VngX+GJjbwHGbv9EKs8LShJIPrD8xHqrDmlSXGkkP01D4A6268Qoi3x5S0H5aqDgtrgBiWsNkzdKwjfrTNx7pKecOi41lyFdffHOGUew4aPPMqjzWR2TEms9WNNQXwnBdDHKMkFsisocF52BolEkcjF8g/u5h2Af92abMu6k16VybJrB7TV1set5A9W7rqG1iXI4+1W6XQfFnpja8xL/zJBvZcyHgeYMNaxa3C3s6PANhPzAUVaXV9eedAkptGJLN13IZj4LujpoAxRKo6bEdydv/5P23R3fx5PgTOpVI7riECAOIg2PThFsEoVCUwStmKCvIx1I2+YixIlv/OiaUWo4lrI/3ve5WGp4ZnuiJPk34JoYAlRbR6+sX14d8Ek6viq/pJUUIfVpNIkNMboUL4u+KpT47eyQ/mWih/KFduQyX9II/vQ+/IJGzEEHIipxAhdmV+K4=
 template:
 metadata:
 creationTimestamp: null
 name: mysecret
 namespace: thenamespace
```

## backing up your sealing key

``` bash
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >main.key
```

## converting .env files to a secret

Working with web applications .env is a common way to store credentials. Let's
look at how we can convert these to secrets.

``` bash
kubectl create secret generic mysecret --from-env-file=.env -n thenamespace --dry-run=client -o yaml > mysecret.yaml
```

Now you have a secret that looks like this.

``` yaml
apiVersion: v1
data:
 foo: YmFy
kind: Secret
metadata:
 creationTimestamp: null
 name: mysecret
 namespace: thenamespace
```

Seal it up just like before.

``` bash
kubeseal -f mysecret.yaml -w mysealedsecret.yaml
```

## Using the secrets

I typically use the secrets in the container spec.

``` yaml
containers:
 - name: myapp
 envFrom:
 - secretRef:
 name: mysecret
 # You can still have other env vars
 env:
 - name: foo
 value: bar
```

Sometimes I want to mount the secret as a volume.

``` yaml
containers:
 - name: myapp
 volumeMounts:
 - name: mysecret
 mountPath: /mysecret
volumes:
 - name: mysecret
 secret:
 secretName: mysecret
```

## Image Pull Secrets

I also need to use imagePullSecrets. Let's walk through the whole process. Starting with the secret.

``` bash
kubectl create secret docker-registry regcred --docker-server=myprivateregistry.example.com --docker-username=foo --docker-password=bar --dry-run=client -o yaml
```

Generates the following secret.

``` yaml
apiVersion: v1
data:
 .dockerconfigjson: eyJhdXRocyI6eyJteXByaXZhdGVyZWdpc3RyeS5leGFtcGxlLmNvbSI6eyJ1c2VybmFtZSI6ImZvbyIsInBhc3N3b3JkIjoiYmFyIiwiYXV0aCI6IlptOXZPbUpoY2c9PSJ9fX0=
kind: Secret
metadata:
 creationTimestamp: null
 name: regcred
type: kubernetes.io/dockerconfigjson

---
```

> the secret

Now we we can seal that secret.

``` bash
kubeseal -f regcred.yaml -w regcred-sealed.yaml
```

And that gives us the following sealed secret that we can deploy into our cluster.

``` yaml
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
 creationTimestamp: null
 name: regcred
 namespace: default
spec:
 encryptedData:
 .dockerconfigjson: AgATYVEywkyYEaoErQbJo6xEZfOnRn1ydNTLkO3Jt/NF/UH+0o9lHpDecRpN0XnVu8xJUcdjkgD9q2XkwP8e6qQDS2mMPTiTNIN+8gbJx97WrD1YQDT0lXBuoyi9I/iwlXxx6MgH/6GY6CeGTz5SRlvoU0Xhlt4d11s7/xapdE8QMLsAReqPEv8oZHEyAxDRrjXX0V+tO8dV+G+GXjUDMBBceLael9rvGzSKIwDVXACVqQhLkB6FoP98M+yyBE46RBNnSnS0ShQM5PprL24HKpRZ43x4RM53KBrQ7R/MxeshafY+B6vUvrolmVox4sud8xngMOcjTO28LLOrck5V8ZiDabhN7ajHEf03IESr1o/ADGf5k9988Vv1txJtsZW0K2mpRu0D7/BLVL9KzbZ5ywULqIoD/Ur2GIGnZqMAKOq4laGp/GJtMKLrhmEvekT397wC/Gf/xdDKVhHf2p4ocsPu7LKFuS5H/Auel/Q5grdn8L5wwrO4VWRv3eJroKh/Hux7Qd7f64O7qdi0XthDocf+gmtjys+Gy72M7tyf8f/O+3oKbS4CWQVTj4ZThMc9znrFnHqt2q/7pAyytTQCpk51wlzOsNvOhCueJM/jmeahaL0LuBrqngqISpnd65sgVzBcZpwK9i2Fckyt0DrZLH+NoIuvaqNhzlF+OMbAft/ylWWKCH4WUP+FKG+1LXM7ud7AA3MMbGSBxHL0/WK/INa7MB56xZKMqqyvvLLQHFTQUROJjkgkzsumdOgwZTRgIFnAZ4+vOX3/1Rtt3mAs3vdoJhL4GuKUYCnEHt908eKkWEVEs7eMk5SdSRtIsbaXO2s0dtADwg==
 template:
 metadata:
 creationTimestamp: null
 name: regcred
 namespace: default
 type: kubernetes.io/dockerconfigjson
```

Now that we have our sealed registry secret, we can deploy it into our cluster.

``` bash
kubectl apply -f regcred-sealed.yaml
```

Now we can use it to pull images from our private registry.

``` yaml
containers:
 imagePullSecrets:
 - name: regcred
```

## Full example

Here is a full deployment example using all the secrets we have created.

* regcred
* mounting a secret
* envFrom secret

``` yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
 creationTimestamp: null
 labels:
 service: myservice
 name: myservice
 namespace: mynamespace
spec:
 replicas: 1
 selector:
 matchLabels:
 service: myservice
 strategy:
 type: Recreate
 template:
 metadata:
 creationTimestamp: null
 labels:
 service: myservice
 spec:
 containers:
 - envFrom:
 - secretRef:
 name: mysecret
 env:
 - name: foo
 value: bar
 image: private-registry.io/myimage:1.0.0
 name: myimage
 ports:
 - containerPort: 5000
 protocol: TCP
 resources: {}
 volumeMounts:
 - mountPath: /mysecret
 name: mysecret
 restartPolicy: Always
 volumes:
 - name: mysecret
 secret:
 secretName: mysecret
 imagePullSecrets:
 - name: regcred
```

## Downside

Now the main downside I see with kubeseal is that it does not provide a way to
store your secrets in a way that you can access outside of your cluster. So
you need to make sure that you have another solution in place to store your
secrets so that you still have them if you ever were to take the cluster down
or move from k8s to something else.

Overall the likelyhood of you loosing a production cluster is pretty low, so
maybe it's ok to just trust it depending on what the secrets are. Especially
for things you control and can rotate anyways its fine.

References:
[1]: https://sealed-secrets.netlify.app/