In 3.4, radeon worked with a glitch - window titles were see-throug (not drawn). In 3.5-rc5, radeon driver seems to be more careful and disables acceleration on this system at all. Full dmesg below.
Does it always do it the same? got the dmesg from 3.4 and/or 2.6.32?
That was a good question. I did some more tries, and it was not repeatable. Sometimes it worked like 3.4 did - rv100 initialized fine, no borders except when maximized, otherwise worked.
Re-tested 3.4.0-rc7 that I had availble from 3.4 times, it worked 4 out of 4 times.
Initial windows border problem seems to have been introduced between 2.6.37 and 2.6.38, bisect is slow.
It's actually more complicated than that. Old kernel images started misbehaving from around 2.6.35-rc5 and any kernel older than that was OK. When I recompiled the older kernels with squeeze gcc (migh have been lenny gcc before, or different answers to make oldconfig), anything from current git down to 2.6.33 is broken with radeon.modeset=1 and works (I get window titles) with radeon.modeset=0.
2.6.32 works and has kernel modesetting enabled from dmesg but I think I did not notice screen going fbcon during boot... maybe it only used the modesetting when X started.
So there are 2 different problems (maybe related), the new one above and the old one of not seeing some window borders when modesetting and acceleration are in use. Maximized windows do have borders.