K8s Deployment Automatic Rollout Restart When Referenced Secrets and ConfigMaps are Updated

Problem Description

Mounting Secrets and ConfigMaps as environment variables inside a pod is one of the most common ways to connect these decoupled Kubernetes resources. However, these environment variables are mounted at pod startup time and Deployment itself is not aware of updates to these external Secrets and ConfigMaps. Environment variables do not get updated unless the pod is restarted.

When you are using K8s Secrets that are being created via External Secrets or Hashicorp Vault and updated at their original source on the fly by external resources (i.e. AWS RDS) at random times, your applications deployed in K8s that depend on these Secrets might become unresponsive.

Rollout restarts in Kubernetes provide a controlled mechanism to refresh workloads without modifying their configuration or code, ensuring minimal disruption to application availability. They allow deployments to apply updates to mounted ConfigMaps or Secrets, recover from state inconsistencies, or integrate changes to runtime environments. By triggering a pod recreation, rollout restarts ensure that the latest configurations are reflected while maintaining rolling update strategies to avoid downtime. This approach offers a simple, reliable way to manage updates and maintain consistency without requiring manual pod deletions or extensive redeployments.

But how can you monitor updates to Secrets and ConfigMaps and start rollout restarts of the affected deployments?

Solution 1

Use existing solution out there, i.e. Reloader

However, if your organization does not trust this due to security concerns or is very strict about not using unsupported Open Source solutions, then maybe Solution 2 might be a better approach.

Solution 2

Build your own Kubernetes operator to monitor Secrets and ConfigMaps updates and trigger a rollout restart of affected deployments.

Very important, for additional safety and for larger environments where multiple applications are deployed to K8s, I choose to only restart deployments IF they are in the same namespace and have this custom label defined: restart-on-config-update: “true”.

For this article I chose to write this Kubernetes Operator in Python leveraging the kopf library (https://kopf.readthedocs.io/en/stable/).

Without further introduction, the code is minimal. This is a working example I composed on a Saturday afternoon; do your own and research to make this production grade. The operator will be deployed within the desired namespace; it will listen for Secrets and ConfigMap updates and rollout restart the affected deployment if they contain the custom label restart-on-config-update: “true”. There are a few other ways of doing this; you can create a custom label for the ConfigMap and Secrets too, so you can filter even more the scope of the operator.

# Handler for ConfigMap or Secrets updates
@kopf.on.update("configmaps")
@kopf.on.update("secrets")
def on_configmap_update(body, spec, meta, namespace, **kwargs):
    try:
        # Log debug information
        kopf.info(body, reason="ConfigMapOrSecretUpdate", message=f"Handling update: {meta['name']} in namespace: {namespace}")

        # Only restart deployments in the same namespace 
        # and if the custom label was defined 
        label_selector = "restart-on-config-update=true"
        deployments = apps_v1_api.list_namespaced_deployment(namespace=namespace, label_selector=label_selector)

        for deployment in deployments.items:
            deployment_name = deployment.metadata.name
            kopf.info(body, reason="RestartDeployment", message=f"Restarting deployment: {deployment_name}")

            # Get current timestamp
            current_timestamp = datetime.utcnow().isoformat()

            # Patch deployment to trigger a rollout restart
            apps_v1_api.patch_namespaced_deployment(
                name=deployment_name,
                namespace=namespace,
                body={
                    "spec": {
                        "template": {
                            "metadata": {
                                "annotations": {
                                    "kubectl.kubernetes.io/restartedAt": current_timestamp
                                }
                            }
                        }
                    }
                }
            )
    except client.exceptions.ApiException as e:
        kopf.warning(body, reason="ApiException", message=f"Kubernetes API exception: {e}")
    except Exception as e:
        kopf.warning(body, reason="Exception", message=f"Unexpected error: {e}")

You can build your operator in a Docker container with this Dockerfile

FROM python:3.11

WORKDIR /app
COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt
COPY operator.py /app/

CMD ["kopf", "run", "operator.py", "--namespace", "."]

Build and publish image

docker build -t <your-docker-registry>/configmap-secret-operator:latest .

docker push <your-docker-registry>/configmap-secret-operator:latest

And deploy via an old fashioned K8s Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: operator
  labels:
    app: operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: operator
  template:
    metadata:
      labels:
        app: operator
    spec:
      serviceAccountName: restart-operator-sa
      containers:
      - name: operator
        image: <your-registry>/configmap-secret-operator:latest
        imagePullPolicy: Always
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

Notice the serviceAccountName: restart-operator-sa above, RBAC configuration is necessary:

My defined role has the following permissions:

rules:
- apiGroups: [""]
  resources: ["configmaps", "secrets", "events"] 
  verbs: ["get", "list", "watch", "create", "patch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "update", "patch"]

Testing

Deploy a sample application via manifest (most important, custom label has to be present restart-on-config-update: “true”)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-deployment
  labels:
    app: example-app
    restart-on-config-update: "true"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
        - name: example-container
          image: nginx:latest
          env:
            - name: CONFIG_VALUE
              valueFrom:
                configMapKeyRef:
                  name: example-configmap
                  key: example-key
            - name: SECRET_VALUE
              valueFrom:
                secretKeyRef:
                  name: example-secret
                  key: example-secret-key

Sample ConfigMap and secret to go with the deployment

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-configmap
data:
  example-key: "example-value-test"

---

apiVersion: v1
kind: Secret
metadata:
  name: example-secret
type: Opaque
data:
  example-secret-key: Y29uZmlkZW50aWFsLXN0cmluZw== # Base64-encoded

Assuming you deployed your operator and the sample application above, by simply editing the ConfigMap or Secret, the operator will trigger a rollout restart of your application.

kubectl edit cm/example-configmap -n <your-namespace>

kubectl get all -n <your-namespace>

You will see your operator and newly restarted application with an AGE of a few seconds.

Optionally, you can create a helm chart to deploy the operator with the following a values.yml

image:
  repository: <your-docker-registry>/configmap-secret-operator
  tag: latest

replicaCount: 1

resources:
  limits:
    cpu: 500m
    memory: 1Gi
  requests:
    cpu: 250m
    memory: 512Mi

serviceAccount:
  create: true
  name: restart-operator-sa

rbac:
  create: true