Turns out I was wrong. It does still happen with a single display, but it takes
much longer. After over 6 days of uptime, my single display failed to wake up.
I'll attach dmesg output for that case.

It appears that the numbers in the debug message don't go way up when the
problem happens with a single display. So I guess maybe that isn't as much of a
clue as I'd hoped.