XFCE4, Intel HD graphics, switching DVI output loses signal

tl;dr:
If you have problems under XFCE4 that X does not come up again after switching your monitor (off and on, or using a KVM switch) then add the following line to the Device section of xorg.conf:
Option "HotPlug" "Off"

This stops xfsettingsd from getting signalled on monitor changes. This will probably work only for single monitor environments, not for setups where you switch from a computer monitor to an entertainment display (TV).