Diagnose why two worker threads stopped making progress in this thread dump
A payment service stops processing transfers. The pool's other threads still serve requests, CPU usage is near zero, no exception is logged, and the two workers below never move again. You take a thread dump.
Constraints:
- reason only from the dump — you may not attach a debugger to production
- explain what the two threads are waiting for, and name the fix
"worker-1" #21 prio=5 tid=0x00007f9c0a12c800 nid=0x2b03 waiting for monitor entry
java.lang.Thread.State: BLOCKED (on object monitor)
at com.acme.billing.Ledger.credit(Ledger.java:88)
- waiting to lock <0x000000076ab3f2c8> (a com.acme.billing.Account)
- locked <0x000000076ab41180> (a com.acme.billing.Account)
at com.acme.billing.Ledger.transfer(Ledger.java:61)
"worker-2" #22 prio=5 tid=0x00007f9c0a12f000 nid=0x2b04 waiting for monitor entry
java.lang.Thread.State: BLOCKED (on object monitor)
at com.acme.billing.Ledger.credit(Ledger.java:88)
- waiting to lock <0x000000076ab41180> (a com.acme.billing.Account)
- locked <0x000000076ab3f2c8> (a com.acme.billing.Account)
at com.acme.billing.Ledger.transfer(Ledger.java:61)
Diagnose the cause.
Both threads are BLOCKED on a monitor, and their monitor addresses cross: worker-1 holds 0x…41180 and waits for 0x…3f2c8, while worker-2 holds 0x…3f2c8 and waits for 0x…41180. That is a circular wait on two Account monitors — a deadlock, because transfer locks the accounts in the caller's argument order, so two opposite transfers grab them in opposite orders. Fix it with a consistent global lock order (e.g. by account id) or a timed tryLock with back-off.
- ✗Reading
BLOCKEDas CPU starvation or scheduling delay instead of a monitor wait - ✗Ignoring the monitor addresses, which are what prove the wait is circular
- ✗Proposing a bigger pool or higher priority, which only adds more stuck threads
- →Which
jstackline would confirm the diagnosis automatically without reading the frames? - →How would switching
Accountmonitors to a timedtryLockchange this dump?
Analysis
The dump gives you everything in four lines. Each thread has a locked / waiting to lock pair:
worker-1: locked <0x…41180> waiting to lock <0x…3f2c8>
worker-2: locked <0x…3f2c8> waiting to lock <0x…41180>
The addresses cross: the monitor one thread holds is exactly the one the other is waiting for, and vice versa. Both are BLOCKED (on object monitor), so neither will release what it holds until it gets what it wants. That closed wait cycle is a deadlock. The CPU is idle because a thread blocked on a monitor does not spin — it sleeps.
Where the cycle comes from. transfer(from, to) synchronizes on from first and to second — the caller's argument order. Two opposite transfers (A→B and B→A) take the same two Account monitors in opposite orders, and the cycle closes.
// ❌ the caller decides the lock order
void transfer(Account from, Account to, long amount) {
synchronized (from) {
synchronized (to) { from.debit(amount); to.credit(amount); }
}
}
// ✅ one global order — the cycle cannot form
void transfer(Account from, Account to, long amount) {
Account first = from.id() < to.id() ? from : to;
Account second = from.id() < to.id() ? to : from;
synchronized (first) {
synchronized (second) { from.debit(amount); to.credit(amount); }
}
}
When a global order cannot be defined, the alternative is ReentrantLock.tryLock with a timeout: a thread that fails to take the second lock releases the first and retries after a randomized back-off.
⚠️ Trap: growing the pool or raising thread priority does not help — the new threads just queue behind the same monitors. For this case jstack also prints a ready-made Found one Java-level deadlock: line listing the members of the cycle.