The GitOps controller Argo CD keeps reverting a manual hotfix — diagnose
Production is degraded. An engineer ran kubectl edit deploy/api to bump replicas from 2 to 6 as an emergency fix. Minutes later Argo CD shows the app OutOfSync and the replica count is back to 2. They edit again; it reverts again. Auto-sync with self-heal is enabled and Git still declares replicas: 2.
$ argocd app get api
Name: api
Sync Status: OutOfSync
Health: Progressing
GROUP KIND NAME STATUS MESSAGE
Deployment api OutOfSync replicas: live 6, desired 2
Sync Policy: Automated (selfHeal: true, prune: true)
14:02:11 live edit detected → 14:02:12 auto-synced back to desiredExplain what is happening and give the correct fix.
This is Argo CD working as designed, not a bug. Git says replicas: 2, so the manual kubectl edit is drift, and self-heal reverts it to the Git state — editing the cluster never sticks under auto-sync self-heal. Fix it in Git: commit replicas: 6 and let Argo CD sync.
- ✗Treating self-heal reverting drift as an Argo CD bug
- ✗Trying to fix desired state by editing the cluster instead of Git
- ✗Disabling self-heal as the fix rather than committing the change to Git
- →When is temporarily disabling self-heal justified, and what is the risk of doing so?
- →How would you make an emergency scale-up durable through the GitOps flow itself?
Solution
What is happening
Nothing is broken — this is GitOps working correctly. Argo CD continuously reconciles the live cluster against the desired state in Git. Git declares replicas: 2, so the kubectl edit to 6 is drift to the controller. With selfHeal: true it reverts drift on every reconcile (the log shows the edit at 14:02:11 → auto-sync at 14:02:12). Editing the cluster fundamentally cannot stick while Git says otherwise.
The correct fix — change Git, not the cluster
# deploy/api.yaml — in Git
spec:
replicas: 6 # was 2; commit, review the PR, mergeOnce merged, Argo CD sees the new desired state and drives the cluster to 6 replicas itself — the fix sticks and lands in the Git audit trail. If you truly need it live before the merge, temporarily clear selfHeal (accepting that drift is no longer healed), but the right path is a fast PR.
