I have a GeChic  onlap 1002 screen:
https://www.gechic.com/en-portable-touch-monitor-onlap1002-overview.html

and a Radeon HD 6320 graphic card:
00:01.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
[AMD/ATI] Wrestler [Radeon HD 6320] [1002:9806]

When X starts, I get and out-of-range message on the screen and no output. If I
then try to connect my system to another screen (high end asus), I also get an
out-o-range message and no output.

When I try the vtc calculated modeline:

xrandr --newmode 1280x800_cvt  83.50  1280 1352 1480 1680  800 803 809 831
-hsync +vsyn
xrandr --addmode DisplayPort-0 1280x800_cvt
xrandr --output DisplayPort-0  --mode 1280x800_cvt

the screen shows a warning, but there is an image.

I can patch the kernel and/or provide any debug info you need.