On Wed, May 11, 2022 at 12:12 PM Linus Torvalds torvalds@linux-foundation.org wrote:
On Wed, May 11, 2022 at 12:08 PM Linus Torvalds torvalds@linux-foundation.org wrote:
The kernel tree might have just the expected *failures* listed, if there are any. Presumably the ci tree has to have the expected results anyway, so what's the advantage of listing non-failures?
.. put another way: I think a list of "we are aware that these currently fail" is quite reasonable for a development tree, maybe even with a comment in the commit that created them about why they currently fail.
That also ends up being very nice if you fix a problem, and the fix commit might then remove the failure for the list, and that all makes perfect sense.
But having just the raw output of "these are the expected CI results" that is being done and specified by some other tree entirely - that seems pointless and just noise to me. There's no actual reason to have that kind of noise - and update that kind of noise - that I really see.
Yeah, the only reason we have full results is that the current tool to check for pass/fail of the entire CI job is 'diff' ;-)
It has the nice benefit of generating a patch for you to squash into whatever commit to update the expectation files, I suppose. But we have something more clever on the mesa-ci side of things where we list skips/flakes/expected-fails but not expected-passes. To be fair, the # of tests on the mesa side is something on the order of 750,000, I don't expect to ever get close to that # on the kernel side.
BR, -R
Linus