SeniorDebuggingOccasionalNot answered yet

Every Pod is CreateContainerConfigError after a config change — find the root cause.

After a config change, every new Pod is stuck in CreateContainerConfigError and never starts. Explain what this state is (and how it differs from a crash), read the artifact, and give the fix.

$ kubectl get pod api-77b
NAME      READY   STATUS                       RESTARTS   AGE
api-77b   0/1     CreateContainerConfigError   0          1m

$ kubectl describe pod api-77b | sed -n '/State/,/Events/p ; /Warning/p'
    State:  Waiting
      Reason:  CreateContainerConfigError
Events:
  Warning  Failed  kubelet  Error: couldn't find key DB_PASSWORD in Secret default/api-secrets

Diagnose and fix.

CreateContainerConfigError is a creation error, not a crash: the kubelet cannot build the container because a referenced ConfigMap/Secret or key is missing. Here an env var maps DB_PASSWORD from a Secret, but the change dropped that key — restore it or fix the reference.

✗Reading CreateContainerConfigError as an app crash with logs to read
✗Missing that a dropped Secret/ConfigMap key blocks container creation
✗Not treating it as a Waiting state like ImagePullBackOff

→How does marking an env valueFrom as optional: true change this failure?
→Why do a Secret rename and a dropped key produce the same Pod-level symptom?

Contents

Solution

1. What the state is

CreateContainerConfigError — the container is not created because the kubelet could not assemble its configuration: an env.valueFrom/envFrom/volume references a ConfigMap, Secret, or key that does not exist. It is a Waiting state (like ImagePullBackOff), not a crash: the process never ran, so there are no app logs.

2. Read the cause

Error: couldn't find key DB_PASSWORD in Secret default/api-secrets

The config change dropped/renamed the DB_PASSWORD key, while the Deployment still maps it into an env var. Hence every new Pod is in the error.

3. Fix

restore the DB_PASSWORD key in Secret api-secrets; or
correct the reference (right key/Secret name); or
if the variable is non-critical, set optional: true on the secretKeyRef.

After the fix, new Pods create normally; any old Pods kept running throughout.

Open topic in roadmap →