You receive a production thread dump with hundreds of BLOCKED threads — how do you identify the root cause?

Look for the small number of threads actually holding the contested lock (state RUNNABLE or waiting on I/O) versus the many threads BLOCKED waiting for it — that lock holder, and what it's doing (a slow query, an external call), is almost always the actual bottleneck, not the blocked threads themselves.