A reverse proxy is throwing 502/503/504 in bursts — diagnose and fix it
A reverse proxy in front of your backend pool starts returning gateway errors in bursts. You loop a health probe and read the proxy log. Explain what each of 502, 503 and 504 means here — which points at a dead upstream, which at a timeout, and which at no healthy backend — and give the fix for each.
$ for i in $(seq 5); do curl -so /dev/null -w '%{http_code}\n' https://api.internal/; done
502
504
503
502
504
# nginx error.log
[error] connect() failed (111: Connection refused) while connecting to upstream, upstream: "http://10.0.0.7:8080"
[error] upstream timed out (110: timed out) while reading response header from upstream
[error] no live upstreams while connecting to upstream, host: "api.internal"Diagnose each code and give the fix.
Each code is a different upstream fault. 502 Bad Gateway — the upstream replied invalidly or refused the connection (crashed or wrong port). 504 Gateway Timeout — the upstream accepted but missed the proxy timeout (slow or overloaded). 503 — no healthy backend. Fix per cause: repair the backend, the timeout, or the health check.
- ✗Confusing 502 (bad reply) with 504 (upstream timeout)
- ✗Restarting the proxy when the fault is in the upstream
- ✗Treating 503 as a client error rather than no healthy backend
- →Which proxy setting most often causes a 504 under load?
- →How does a misconfigured health check produce a 503 with healthy pods?
Solution
1. Read the codes as distinct causes
All three are gateway (proxy/LB) responses about the upstream, not the proxy itself:
502 Bad Gateway → upstream replied invalidly OR refused (Connection refused)
504 Gateway Timeout → upstream accepted but did not answer in time (upstream timed out)
503 Service Unavailable → no live upstreams in the pool (no live upstreams)The log maps them directly: connect() failed (111: Connection refused) → 502; upstream timed out (110) → 504; no live upstreams → 503.
2. Diagnose each
- 502 — backend
10.0.0.7:8080crashed or listens on the wrong port: connection refused. - 504 — backend is alive but slow/overloaded: it missed
proxy_read_timeout. - 503 — the health check pulled every backend out of the pool (or they truly are all dead).
3. Fix
502 → restart/repair the backend, check the port and that the process listens
504 → speed up the upstream OR raise proxy_read_timeout; check load
503 → fix the health check (threshold/path/interval), return live instances to the poolDo not blindly restart the proxy — it is just faithfully reporting the upstream's state.
