Production throws Too many open files — diagnose it from the trace and lsof
A report-export service has been up for six days. Under normal load it starts throwing the error below; a restart clears it for about a day, then it returns. The artifacts come from the live process.
Constraints:
- the box has spare CPU, memory and disk; nothing is swapping and the volume is not full
- the code has run unchanged for two years, but export volume grew about 5x this quarter
- you are allowed to raise the limit — say whether that is the fix
java.io.FileNotFoundException: /var/data/reports/r-88213.csv (Too many open files)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:236)
at com.acme.report.ExportJob.writeReport(ExportJob.java:74)
$ ulimit -n
4096
$ lsof -p 8123 | wc -l
4092
$ lsof -p 8123 | awk '{print $5}' | sort | uniq -c | sort -rn | head -3
3968 REG <- /var/data/reports/r-*.csv
94 IPv4 <- postgres connections
18 CHR
$ jcmd 8123 GC.run
$ lsof -p 8123 | wc -l
231 <- back to ~4000 within the hourDiagnose the cause.
This is a file-descriptor leak, not an undersized limit: 3968 of the 4092 handles are report CSVs the process has finished writing. A forced GC drops the count to 231, which proves the streams are closed only by their Cleaner when collected — writeReport never calls close(). Wrap it in try-with-resources; raising ulimit -n only postpones the failure.
- ✗Raising
ulimit -nand calling it fixed, while the handle count keeps climbing back to the new ceiling - ✗Ignoring the clue that a forced GC releases the handles — the signature of a stream closed only by its cleaner
- ✗Assuming a
FileOutputStreamis closed when its variable goes out of scope
- →Why does try-with-resources still close the stream when
writeReportthrows halfway through? - →Which JDK-level metric would have shown the handle count growing before the first failure?
Solution
Three clues line up. First: almost all 4092 handles are REG files under /var/data/reports/r-*.csv — reports the process has already finished writing. Second: a forced jcmd GC.run drops the count to 231. Descriptors released by garbage collection means exactly one thing: close() is never called, and the handle is only surrendered by the stream's Cleaner when the object is collected. Third: a restart buys about a day, so the count grows monotonically with the number of exports.
That is a descriptor leak, not a small limit. The 5x growth in volume only made the ceiling arrive sooner.
// ❌ before: the stream is closed only when it is garbage-collected
void writeReport(Path path, List<Row> rows) {
var out = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(path.toFile()), UTF_8));
for (Row r : rows) {
out.write(r.toCsv()); // ⚠️ throw here and close() never runs
out.newLine();
}
out.flush(); // flush is there, close is not
}
// ✅ after: deterministic close, including on the exception path
void writeReport(Path path, List<Row> rows) throws IOException {
try (var out = Files.newBufferedWriter(path, UTF_8)) {
for (Row r : rows) {
out.write(r.toCsv());
out.newLine();
}
}
}Why ulimit is not the fix. A higher limit just postpones the failure: the leak rate is unchanged, so the count will climb to 65536 all the same, only later. Raise the limit separately and deliberately — once you have proven every descriptor genuinely needs to be open at once.
Catching it earlier. The UNIX flavour of OperatingSystemMXBean exposes getOpenFileDescriptorCount() and getMaxFileDescriptorCount(). Publish that gauge and alert on a rising trend rather than on the ceiling being hit.
