Here are the utility functions again. Rebased and updated slightly.
I adapted drm_framebuffer_check() to handle multiple handles, and I changed some of the error values it returns.
The plane options ioctl is a bit of an open question. I started by adding separate ioctls for some subsets of the fucntionality, but then I was just writing so much duplicated ioctl handling code that I decided it doesn't seem right. I'm not really happy with this monster ioctl either.
I would actually like to replace the whole KMS API with some tag-length-value (TLV) type of thing. Basically you'd just feed the ioctl a KMS object ID and bunch of attribute IDs + values. The ioctl could take multiple such "streams" for any number of KMS objects.
It would be easily extendible simply by adding new types of attributes.
It would also support atomic mode setting changes across an arbitrary set of KMS objects, without having to do some kind begin+set+set+set...+commit multi ioctl sequence with all the state management nightmares that it would entail.
It would also cleanly handle cases where some attributes would be somehow tied together in hardware (eg. you have to set one attribute to a specific value, before another atribute can be set). With a multi ioctl approach you would have to accept invalid intermediate states, and delay error checking until the commit ioctl.
Of course you'd need to parse the "stream" and feed the resulting state to the driver somehow. I'm think the common code would pass the stream to the driver attribute at a time, and the driver would build up the state, possibly doing some early error checking. Finally when the whole state is built up, it would would be passed to the driver for final error checking, and if everything looks good the driver could push the state to the hardware. The driver could then attempt to do its thing in an atomic fashion, if possible.
Well, that's my utopian idea. Would require quite a bit of work though. Anyone else thinking along similar lines?