du reports 1 GB but df reports 10 GB used on the same mount — diagnose
A monitoring alert fires: the /var filesystem is 90% full, yet adding up the files there with du accounts for only a fraction of the used space. The two tools disagree:
$ du -sh /var
1.0G /var
$ df -h /var
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 10G 9.0G 500M 95% /varNo new files appear, restarting nothing helps yet, and the gap will not clear on its own.
Diagnose the cause and state how to reclaim the space.
A process holds an open descriptor to a file deleted from the directory tree. du walks names and no longer sees it, but the inode's blocks survive until the last descriptor closes, so df still counts the space. Find it with lsof | grep deleted; reclaim by restarting or signalling the holding process, or truncating its fd via /proc/<pid>/fd.
- ✗Running
fsckon a live mount instead of finding the process holding the deleted file - ✗Assuming the space is gone permanently rather than pinned by an open descriptor
- ✗Deleting more files when the real fix is to release or restart the holding process
- →Why does truncating
/proc/<pid>/fd/<n>free the space without killing the process? - →Which common service typically causes this with a deleted-but-still-written log file?
What is happening
du and df measure used space differently:
duwalks the directory tree and sums the sizes of files it can reach by name.dfasks the filesystem how many blocks are marked used, regardless of whether any name still references them.
When a process opens a file and the file is then deleted (unlink) from its directory, the name disappears — but the inode and its blocks stay alive as long as at least one descriptor is open. du no longer sees the file, while df keeps counting its blocks. The classic case is a service writing to a log, the log getting deleted (or rotated wrong), and the service still holding the descriptor and writing on.
Diagnosis
# show deleted-but-still-open files
lsof | grep '(deleted)'
# example line: nginx 1234 www-data 5w REG 8,2 9663676416 /var/log/app.log (deleted)The columns give the PID (1234) and the descriptor number (5).
Reclaiming the space
# Option 1 — restart/signal the holding process (closes the descriptor)
systemctl restart nginx
# Option 2 — truncate the descriptor without killing the process
: > /proc/1234/fd/5⚠️ fsck on a mounted, live filesystem is unnecessary and dangerous here — there is no corruption.
