A disk-full alert fires on a production database host — diagnose and respond
An alert pages you: the data partition of a PostgreSQL host is nearly full. Below is what you see on the box. Walk through how you triage and respond — what you check first, the immediate action to keep the DB up, and the durable fix — without simply deleting data files by hand.
ALERT: disk usage on /var/lib/postgresql = 96% (480G/500G)
$ df -h /var/lib/postgresql → 96% used
$ du -sh /var/lib/postgresql/* → base/ 410G, pg_wal/ 60G
largest tables show heavy UPDATE/DELETE churn; autovacuum last ran days agoDiagnose the cause and give the response.
First confirm with df/du what is eating space — data, bloat from dead tuples, WAL, or logs. Immediate relief: free safe space (rotate logs, archive old WAL) to keep the DB writable. Durable fix for the bloat shown: run VACUUM (ANALYZE) — it reclaims dead-tuple space and refreshes planner stats; then investigate the lagging autovacuum and right-size the disk. Never delete files in the data directory by hand.
- ✗Deleting files inside the data directory by hand, corrupting the DB
- ✗Skipping triage and acting before knowing what consumes the space
- ✗Treating
VACUUMas a query tuner rather than space reclamation
- →Why does
VACUUMreclaim space for reuse butVACUUM FULLis needed to shrink the file? - →What can cause autovacuum to fall behind on a high-churn table?
Solution
1. Triage — find what eats the disk
$ df -h /var/lib/postgresql # confirm the partition is filling
$ du -sh /var/lib/postgresql/* # base/ (data+bloat) vs pg_wal/ (WAL) vs logsHere base/ holds 410G with heavy UPDATE/DELETE and a lagging autovacuum — the classic bloat case: dead row versions never reclaimed.
2. Immediate relief (keep the DB writable)
Free safe space: rotate and clear logs, archive old WAL. The 60G WAL partition — check it is not piling up from an inactive replication slot.
3. Durable fix
VACUUM (ANALYZE) big_table; -- reclaim dead-tuple space for reuseVACUUM returns the space under dead tuples for reuse inside the file; ANALYZE refreshes planner statistics. Investigate why autovacuum lagged (load, threshold settings). Then right-size the disk or move pg_wal to its own volume.
⚠️ Never rm files inside the data directory — it corrupts the database.
