Disk I/O stall on cloud-PfN5fm — jbd2 stuck in D state, SSH locked out by MaxStartups

Instance cloud-PfN5fm (61.107.200.123) went fully unresponsive. Requesting a check on the storage backing /dev/vda.

Symptoms

  • New SSH refused: banner line 0: Exceeded MaxStartups, then connection reset.
  • Existing sessions alive, but ss, journalctl, sudo hang for >1 min.
  • Plenty of free disk space.

I/O pressure only — CPU and memory idle

/proc/pressure/io      some avg60=85.18  full avg60=79.03
/proc/pressure/cpu     some avg60=0.04
/proc/pressure/memory  some avg60=0.00
/proc/loadavg          183.37 177.75 166.55  17/30434

Load ~183 with only 17 runnable; the rest is D state (uninterruptible I/O wait).

ext4 journal and writeback threads stuck

$ ps -eo stat,pid,etime,comm | grep '^D'
D   2295   jbd2/vda1-8
D<s 2383   systemd-journal
D  11761   kworker/u384:12+flush-252:0   (37m)
D  11764   kworker/u384:15+flush-252:0   (37m)

jbd2/vda1-8 is blocked, so everything touching the filesystem stalls behind it. sshd children hit D state pre-auth, so LoginGraceTime (2m, stock config) never fires and the MaxStartups slots are exhausted.

/dev/vda — /proc/diskstats, cumulative

  • ~42.8 ms/write (1,073,834,968 ms / 25,080,458 writes)
  • queue depth ~19 while busy, ~82 MB/s effective
  • 21 I/Os in flight at sample time

Notes

  • Single volume /dev/vda1 (ext4), no network storage.
  • Still rw despite errors=remount-ro, so no filesystem-level I/O error was detected.
  • nvme0n1–nvme7n1 all show 0 reads/writes. Are they recognized and usable?
  • ~20 days uptime, but almost all PSI accumulated in the last 2–3 hours.

Solved.

1 docker container penalizes the whole I/O process, so I killed it and the connection is now alive.