Encrypt your Kubernetes Secrets with Mozilla SOPS

By default, Kubernetes Secrets (secrets) are stored with base64 encoding in YAMLfiles. The lack of encryption for secrets often leads to the question of how to store secrets securely. Obviously, you don’t want to put your sensitive configuration data into a git repository, because it is just encoded. echo <base64_representation> | base64 -d.

A typical solution is using services like Azure Key Vault, or HashiCorp Vault to persist sensitive data. Those services can be integrated with Kubernetes by using the Secrets Store CIS driver. However, relying on an additional service means that you have to manage and maintain that service in addition to Kubernetes. Additionally, depending on the service you use to store your sensitive data, some sensitive configuration must be stored somewhere to configure the CIS driver.

As an alternative, you can use Mozilla SOPS (SOPS) to encrypt and decrypt your Kubernetes secret files. Secrets that are encrypted via SOPS can be stored in source control. Encrypted secrets will be decrypted locally just before they’ll be deployed to Kubernetes. This article demonstrates how to encrypt and decrypt Kubernetes secrets (YAML files) using SOPS in combination with Azure Key Vault, which allows you to store your secrets along with other Kubernetes manifests directly in git.

What is Mozilla SOPS

Mozilla SOPS (Secrets OPerationS) is a platform-agnostic CLI that is used to edit encrypted files of different formats - including yaml, json, ini, binary, and others. SOPS supports multiple backends to consume keys for encryption and decryption. Here are the five most popular supported scenarios:

PGP
Azure Key Vault
AWS KMS
GCP KMS
HashiCorp Vault

For authenticating against Azure Key Vault, SOPS tries several authentication patterns in the following order:

Client credentials
Client certificate
Username & password
Managed Service Identity (MSI)
Azure CLI authentication

Although Azure CLI authentication is frictionless, I would encourage you to use an Azure Service Principal (SP) on your local development machine. To decrypt secrets with SOPS in Kubernetes (for example, if you use a GitOps operator such as Flux), you should consider using a combination of Managed Service Identity (MSI) and Azure AD Pod Identity. (Consider the official flux documentation for a detailed walk-through).

Provision an Azure Service Principal (SP)

To create a new Azure SP, use the following command:

# create a service principal
az ad sp create-for-rbac -n sp-sops-keyvault -o json
# {
#   "appId": "00000000-0000-0000-000000000000",
#   "displayName": "http://sp-sops-keyvault",
#   "name": "http://sp-sops-keyvault",
#   "password": "00000000-0000-0000-000000000000",
#   "tenant": "<your_tenant_identifier>
# }

SOPS (and some of the upcoming commands) require authentication information stored in environment variables. That said, you can quickly store appId, tenant, and password in local environment variables using the export command:

export AZURE_CLIENT_ID=<appId>
export AZURE_CLIENT_SECRET=<password>
export AZURE_TENANT_ID=<tenant>

If you want to integrate SOPS into your CI pipeline, consider using repository secrets (that’s how sensitive configuration data is called in the context of GitHub) or a competitive solution to deal with sensitive data in your CI system.

Provisioning an Azure Key Vault instance

You need an Azure Key Vault instance. Provisioning a new Azure Key Vault is straightforward using Azure CLI, as shown in the following snippet:

# create a new Resource Group
az group create -n rg-sops-sample -l germanywestcentral

# create a Key Vault instance
az keyvault create -n kv-sops-sample \
   -g rg-sops-sample \
   -l germanywestcentral 

# create an access policy for the SP
az keyvault set-policy -n kv-sops-sample \
   -g rg-sops-sample \
   --spn $AZURE_CLIENT_ID \
   --key-permissions encrypt decrypt

Create a key in Azure Key Vault for encryption & decryption

At this point, you have to create the actual key used for encryption and decryption in our newly created instance of Azure Key Vault:

# create an key for encryption / decryption 
az keyvault key create -n sops-sample-key \
    --vault-name kv-sops-sample \
    --ops encrypt decrypt \
    --protection software

On top of authentication information, SOPS also requires the identifier of the key we just created. Again, use Azure CLI and store the identifier in an environment variable:

# read and store key identifier
export KEY_ID=$(az keyvault key show -n sops-sample-key \
    --vault-name kv-sops-sample \
    --query key.kid -o tsv)

Install Mozilla SOPS

As already mentioned, SOPS is a cross-platform CLI. You can download the CLI from GitHub at https://github.com/mozilla/sops/releases. Ensure that you put the executable in a folder that is part of your PATH variable.

# download sops cli for macOS
curl -O -L -C - https://github.com/mozilla/sops/releases/download/v3.7.1/sops-v3.7.1.darwin

# move and rename the cli to /usr/bin
sudo mv sops-v3.7.1.darwin /usr/bin/sops

# make it executable
sudo chmod +x /usr/bin/sops

# latest macOS may prevent you from using SOPS CLI
# use System Preferences > Security & Privacy to whitelist SOPS

Encrypt Kubernetes Secrets

First, create a regular Kubernetes secret using kubectl:

# create the Kubernetes secret
kubectl create secret generic demo \
    --from-literal mysecret=secret_value \
    -o yaml \
    --dry-run=client > secret.encoded.yml

# print the contents of secret.encoded.yml
cat secret.encoded.yml

# apiVersion: v1
# data:
#   mysecret: c2VjcmV0X3ZhbHVl
# kind: Secret
# metadata:
#   creationTimestamp: null
#   name: demo

As you can see, the secret is stored in its base64 representation. Now, use SOPS CLI to create an encrypted variation of the secret. Especially when considering decrypting secrets in Flux, ensure that you provide --encrypted-regex argument and limit encryption just to values stored in data and stringData.

# encrypt secret.encoded.yml using SOPS
sops --encrypt --encrypted-regex '^(data|stringData)$' \
    --azure-kv $KEY_ID secret.encoded.yml > secret.encrypted.yml

# print the contents of secret.encrypted.yml
cat secret.encrypted.yml

# apiVersion: v1
# data:
#    mysecret: ENC[AES256_GCM,data:gz/WAjWte3bCnNm6e+G4ow==,iv:VB4pAv833tDdD4n76h4CqEZNpGdwA3V1QGWp7PK/Jfc=,tag:CcUy3rti4XcWArmHANVS8Q==,type:str]
# kind: Secret
# metadata:
#     creationTimestamp: null
#     name: demo
# sops:
#     kms: []
#     gcp_kms: []
#     azure_kv:
#         - vault_url: https://kv-sops-sample.vault.azure.net
#           name: sops-sample-key
#           version: ee44c0c0cc9e4620aa4f4c86c4942047
#           created_at: "2021-08-02T20:55:40Z"
#           enc: EjszDACgiDP8rW3wzs-7fAmFzlAhCq0-R9YlA9cuPcq78EXEeNTC8OnlSdXQAGdGrgE9oylu1HKZa4RB9GxzzVDav8uNVPp67NPmC4-teeA5iRE4jqlp1An6sG6CpkZGcAmKWpfj_DEWecqrNGWSLTA2hI_HKwG5xNkFh9Myik6732W-XL65IFqgepcFrNIzeHetznO0j1iISNXqMeJjeCnZ6Qq0jcXUMIfQnXjAllKfjSukiT3A3GlWxP0j50Z328t-JHi5RowYHT-hC8FDOdR_U95sqnFd27RgEXmbDIU6IGvP3vmCiZJz4YQCPXaGhySvFY6qCEoCbCSC4RaoWw
#     hc_vault: []
#     age: []
#     lastmodified: "2021-08-02T20:55:41Z"
#     mac: ENC[AES256_GCM,data:AmKRnzoImfIzPa3JBcuxUKRrse5uZwJGukpLj1wxed3R7lsUN+QAV1+WkfNyeMoW5C3ek7j20Xpbvzi+MgP8zcQOwWSwA79Svgz3hKMn9eTRTfgU+4jYezIIHCwkv61MTN8RGW5AhOInYP8oRPW3zKD+SbBO/Jeu7SC+/oVn07I=,iv:S4Th+0quL84lhJtA/lugEv+iLc+WhWEYPSlXGWKhd/M=,tag:CUGg8+UM7gNSzfjJx1Ua1w==,type:str]
#     pgp: []
#     encrypted_regex: ^(data|stringData)$
#     version: 3.7.1

By default, the encrypted version of the secret contains essential information about Azure Key Vault and the key used to encrypt and decrypt. This information makes decrypting easy, as you will see in a few minutes. However, you can also provide a custom .sops.yaml configuration file to remove this metadata from the actual secrets file. At this point, you can delete the encoded version of the secret, and add the encrypted one to git using the following commands:

# delete encoded version of the secret
rm secret.encoded.yml

# add encrypted secret to source control and commit it
git add secret.encrypted.yml
git commit -m 'chore: add encrypted secret'

Decrypt Kubernetes Secrets for deployment

To deploy the secret to Kubernetes, you must decrypt it. Again, SOPS CLI is here to help. The decrypted secret can be piped directly to kubectl for deployment as shown in this snippet:

# decrypt and deploy the secret
sops --decrypt secret.encrypted.yml | kubectl apply -f -

Deploy an echo server to Kubernetes

Having the secret stored in Kubernetes, you can provision the echo container, which will print basic information about the actual environment via HTTP. Notice the envFrom part of the manifest. The previously created secret is used to populate environment variables:

apiVersion: v1
kind: Pod
metadata:
  name: echo
  labels:
    app: echo
spec:
    containers:
    - image: thorstenhans/env-via-http:0.0.1
      name: main
      ports:
        - containerPort: 5000
          protocol: TCP
      envFrom:
        - secretRef:
            name: demo
            optional: true
      resources:
        requests:
          cpu: 50m
          memory: 32Mi
        limits:
          cpu: 100m
          memory: 48Mi

Once deployed, use simple port-forwarding and curl to verify the deployment:

# deploy the sample pod 
kubectl apply -f pod.yml

# verify pod is running
kubectl get po
#
# NAME                    READY   STATUS    RESTARTS   AGE
# echo-74bc9c6d74-vh5lr   1/1     Running   0          4m38s

# activate port forwarding
kubectl port-forward echo-74bc9c6d74-vh5lr --port 8080:5000
#
# Forwarding from 127.0.0.1:8080 -> 5000
# Forwarding from [::1]:8080 -> 5000

# start a new terminal session
# issue an HTTP request to the echo pod with curl
curl http://localhost:8080 | jq

You should get a JSON object containing all environment variables available inside the Pod, including the mysecret environment variable with its decrypted value of secret_value.

{
  "hostName": "echo-74bc9c6d74-vh5lr",
  "envVars": [
    "KUBERNETES_SERVICE_PORT=443",
    "KUBERNETES_PORT=tcp://10.0.0.1:443",
    "HOSTNAME=echo-74bc9c6d74-vh5lr",
    "SHLVL=1",
    "PORT=5000",
    "HOME=/root",
    "mysecret=secret_value",
    "KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1",
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "KUBERNETES_PORT_443_TCP_PORT=443",
    "KUBERNETES_PORT_443_TCP_PROTO=tcp",
    "KUBERNETES_SERVICE_PORT_HTTPS=443",
    "KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443",
    "KUBERNETES_SERVICE_HOST=10.0.0.1",
    "PWD=/"
  ]
}

Conclusion

Being able to store secrets securely in source control is excellent. SOPS makes the process of encrypting and decrypting secrets painless. The seamless integration with services such as Azure Key Vault is fantastic. Teams will find this approach very useful, especially during the transition from legacy continuous deployment towards GitOps. It is also worth mentioning that SOPS is seamlessly integrated with Flux, one of the most popular continuous delivery solutions based on the GitOps Toolkit.