Response times degrade only under load, while a single request stays fast. Diagnose the cause.
Production response times climb from 120 ms to 8 s at peak and recover afterwards. A single request against the same box is fast, and CPU on the web servers sits at 30%.
Constraints: the code did not change, database CPU is also low, and nothing is logged as an error. You have the PHP-FPM status page and the slow log for the peak window.
pool: www
process manager: dynamic
max children: 20
active processes: 20
idle processes: 0
listen queue: 312
max listen queue: 512
slow requests: 847
[slow-log] script /index.php request_uri=/api/orders
[0x00] curl_exec() /app/Service/Warehouse.php:41Diagnose the cause.
The pool is saturated, not slow: 20 of 20 children are active and 312 requests sit in the listen backlog, so most of the 8 s is spent waiting for a worker. The slow log names the culprit — a blocking curl_exec() to the warehouse service holds a worker for seconds. Raising pm.max_children only moves the queue; add a timeout and take that call out of the request.
- ✗Reading a long listen queue as a network problem rather than as pool exhaustion
- ✗Raising
pm.max_childrenpast the RAM budget and turning a queue into swapping - ✗Ignoring the slow log because one request in isolation looks perfectly fast
- →Why does a blocking third-party call hurt far more under load than in a single request?
- →What does a
curltimeout guarantee about the worst-case time a worker is held?
active processes: 20 out of max children: 20 with listen queue: 312 is not "slow code" — it is an exhausted pool. A request waits in the socket backlog until a worker frees up, which is exactly why a single request stays fast: it gets a worker immediately.
The low CPU confirms it — the workers are not computing, they are waiting.
active processes: 20 / 20 → no free workers
listen queue: 312 → 312 requests waiting for one
CPU: 30% → workers are blocked, not busyThe slow log names what they wait on: a blocking curl_exec() to the warehouse service. While it hangs, that worker is held and serves nobody.
// Warehouse.php:41 — a call with no timeout holds the worker indefinitely
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT_MS, 300);
curl_setopt($ch, CURLOPT_TIMEOUT_MS, 800); // worst case is now bounded
$body = curl_exec($ch);The order of work: bound the worst case with a timeout, then take the call out of the request — onto a queue, or behind a cached warehouse response. Raising pm.max_children first is wrong: the queue simply moves inside the machine and the RAM runs out.
