Kubernetes Centralized External Secrets Management with HashiCorp Vault

Hands-on Tutorial

Segesta

Kubernetes might not meet all the requirements for an enterprise-grade secret store. The enterprise world has high standards for security compliance, especially the financial services sector. This is where external providers like HashiCorp Vault or Cyberark come to the rescue.

HashiCorp Vault is an open-source project by HashiCorp and likely one of the most popular secret management solutions in the cloud native space. HCP Vault is ideal for companies obsessed with standardizing secrets management across all platforms, not just Kubernetes, since it is integrating with a variety of common products in the cloud (i.e. AWS IAM) and DevOps space (i.e. GitLab). Here is a HashiCorp official depiction of some of the various pluggable components that allow you to integrate with external systems:

There are two ways to use the Vault: either as self-managed, or the existing HashiCorp cloud-hosted solution, aka HCP Vault. Here are the differences as presented by HashiCorp:

Self-managed vs. HCP Vault cluster

Hands-On Tutorial

To exemplify the K8s — Vault integration with the injection integration way of secrets retrieval, we will stand up a local Vault OSS server and use a Minikube K8s cluster to deploy a sample application that will access the secrets.

The Vault helm chart can deploy only the Vault Agent Injector service configured to target an external Vault. The injector service enables the authentication and secret retrieval for the applications by adding Vault Agent containers as they are written to the pod automatically, including specific annotations.

The Injection Integration approach runs an initContainer and/or sidecar container to communicate with the external Vault. Secrets can be injected into the pod’s filesystem, making them available to all containers in the pod. Secrets can also be made available as Environment Variables, both ways ensuring decoupling the secret store from the application. When pods are created, based on annotations, the vault-agent-injector adds an initContainer (used to retrieve the initial secret) and optionally a sidecar container to keep secrets updated in real-time, if required.

Assumptions: OS X with existing helm, jq, Xcode, minikube, and kubectl.

Start a Local Vault Server

1. Install HashiCorp Vault CLI

brew tap hashicorp/tap
brew install hashicorp/tap/vault

2. Start a Vault development server with root as the root token that listens for requests at 0.0.0.0:8200

vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

3. Validate

export VAULT_ADDR=http://0.0.0.0:8200

vault login root

In your browser open http://0.0.0.0:8200

NOTE: Use token “root” to login

Start K8s Cluster

4. Start minikube

minikube start

5. Get minikube external Vault URL

EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')

6. Install vault-agent-injector using the official helm with the following parameter:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install vault hashicorp/vault --set "injector.externalVaultAddr=https://$EXTERNAL_VAULT_ADDR:8200" --set "tlsDisable=true"

Integrate K8s cluster with Vault

7. Extract the values needed to configure the K8s-Vault integration

VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-token-")).name')
TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)
KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

8. Enable K8s cluster - Vault Authentication

NOTE: Replace <cluster-name> in all the steps below with your cluster name. In our case, it will be minikube. The path option is needed when integrating multiple clusters with Vault.

vault auth enable -path=<cluster-name> kubernetes
vault write auth/<cluster-name>/config \
     token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
     kubernetes_host="$KUBE_HOST" \
     kubernetes_ca_cert="$KUBE_CA_CERT" \
     issuer="https://kubernetes.default.svc.cluster.local"

The UI will reflect these changes as well:

9. Create a secret store and add a couple of key/value pairs:

NOTE: A Vault secret store, policy and role naming convention should be adopted to better organize and identify these assets.

vault kv put secret/<cluster-name>/config username='admin' password='secret'

10. Create a Vault policy to specify permissions for the secret store above

vault policy write <cluster-name>-policy - <<EOF
path "secret/data/<cluster-name>/config" {
  capabilities = ["read"]
}
EOF

11. Create a Kubernetes authentication role for a specific K8s namespace and K8s service account

NOTE: ServiceAccount is mandatory, NameSpace is optional

vault write auth/<cluster-name>/role/<cluster-name>-role \
     bound_service_account_names=test-sa \
     bound_service_account_namespaces=default \
     policies=<cluster-name>-policy \
     ttl=24h

Deploy sample application for testing

12. Create ServiceAccount and deploy sample application.

kubectl create sa test-sa
kubectl apply -f sample.yaml 

#sample.yaml
apiVersion: v1
kind: Pod
metadata:
  name: sample
  labels:
    app: sample
  annotations:
    # This will ensure the initial secret injection (initContainer)
    vault.hashicorp.com/agent-inject: 'true'  
    # Optional: Ensure real-time updating of secrets  (sidecar will be running at all times)      
    vault.hashicorp.com/agent-inject-status: 'update'   
    # Vault Auth Path
    vault.hashicorp.com/auth-path: 'auth/minikube'
    # Namespace
    vault.hashicorp.com/namespace: 'default'
    # Bounds injection to Vault role defined above
    vault.hashicorp.com/role: 'minikube-role' 
    # Mount secrets on the /vault/secrets/credentials.txt
    vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/data/minikube/config' 
spec:
  serviceAccountName: test-sa
  containers:
    - name: app
      image: nginx:latest
      ports:       
      - containerPort: 80

13. Validate by ensuring the /vault/secrets/credentials.txt was populated with the secret from the Vault server:

kubectl exec -it sample -c app -- cat /vault/secrets/credentials.txt

data: map[password:secret username:admin]

14. To demonstrate the vault-agent sidecar functionality, which runs in the same pod and updates the secrets if they are changing on the Vault server, modify the secret from the UI (http://0.0.0.0:8200)

NOTE: username value was changed to “admin2"

Allow a couple of minutes for the sidecar to sync with the Vault server and try again to display the secrets file on the pod:

kubectl exec -it sample -c app -- cat /vault/secrets/credentials.txt

data: map[password:secret username:admin2]

This credentials rotation at Vault level just saved you one or multiple redeployments.

The sky is the limit and this is just a basic example. There are multiple ways to provide these secrets to the pod. Here is the documentation link.

The official documentation for the above tutorial can be found here.