I'm a bit confused. You mention this happens after starting X, and also mention you don't use Xorg. Can you clarify? What kernel are you using?