[PATCH] drm: don't unnecessarily enable the polling work