About 2% of requests return 500, always from one host, and it never reproduces locally. Diagnose the cause.
Production returns a 500 on roughly 2% of requests. It never reproduces locally or on staging, and all four app servers were deployed from the same release tag an hour ago.
Constraints: the load balancer is healthy, the database is healthy, and the failing requests are spread evenly across routes rather than concentrated on one endpoint. You have the error tracker and shell access to every host.
[error tracker] 1 issue, 2141 events in 1h server_name: app-03 (100%)
Error: Call to undefined method App\Billing\Invoice::totalWithTax()
/var/www/current/app/Http/Controllers/InvoiceController.php:88
release: 20260710-1a2b3c
[app-03] readlink /var/www/current -> /var/www/releases/20260710-1a2b3c
[app-03] php-fpm master started -> 31 days ago
[app-03] opcache.validate_timestamps -> 0
[app-03] opcache_get_status(): cached script
/var/www/current/app/Billing/Invoice.php timestamp 2026-07-09Diagnose the cause.
The symlink on app-03 does point at the new release, but its FPM master was never restarted and opcache.validate_timestamps is 0, so its workers still execute the previous release's compiled Invoice — a class with no totalWithTax(). The post-flip OPcache reset silently failed on that host. Reset OPcache or reload FPM there, and make the deploy fail when a host does not confirm it.
- ✗Reading a fault confined to one host as a hardware problem rather than as a stale runtime on that host
- ✗Trusting that a symlink pointing at the new release means the new code is what actually executes
- ✗Letting a deploy report success when its OPcache reset step failed on one machine of the fleet
- →With
opcache.validate_timestampsat 0, what is the only thing that makes PHP re-read a changed file? - →Where in the deploy does the OPcache reset belong, and how do you prove it ran on every host?
Three signals add up to one conclusion.
readlink /var/www/current -> new release (symlink was flipped)
php-fpm master started -> 31 days ago (FPM never reloaded)
opcache.validate_timestamps -> 0 (mtime is never rechecked)
opcache_get_status() -> Invoice.php dated 2026-07-09 (previous release)The symlink moved, but it is not what runs. With validate_timestamps=0 OPcache never compares a file's mtime: once compiled, a script lives in shared memory until an explicit reset. The FPM master on app-03 has not been restarted for 31 days, so no reset happened there — and its workers keep executing the previous release's Invoice, which has no totalWithTax() yet. That is both the Call to undefined method on exactly one host and the "2% of requests": it is the traffic share the balancer sends to app-03.
It does not reproduce locally for the same reason — dev runs validate_timestamps=1.
# fix it on the box
cachetool opcache:reset --fcgi=127.0.0.1:9000 # or systemctl reload php8.3-fpm
# and put it in the deploy as a verified step, not a best-effort one
for host in app-01 app-02 app-03 app-04; do
ssh "$host" 'cachetool opcache:reset --fcgi=127.0.0.1:9000' || exit 1
doneThe point is that the deploy must fail when even one host does not confirm the reset. A step that is silently skipped leaves one server on the previous release, and that does not look like a broken deploy — it looks like a flaky bug.
