On Fr, 2020-02-28 at 10:47 +0100, Daniel Vetter wrote:
On Fri, Feb 28, 2020 at 10:29 AM Erik Faye-Lund erik.faye-lund@collabora.com wrote:
On Fri, 2020-02-28 at 13:37 +1000, Dave Airlie wrote:
On Fri, 28 Feb 2020 at 07:27, Daniel Vetter daniel.vetter@ffwll.ch wrote:
Hi all,
You might have read the short take in the X.org board meeting minutes already, here's the long version.
The good news: gitlab.fd.o has become very popular with our communities, and is used extensively. This especially includes all the CI integration. Modern development process and tooling, yay!
The bad news: The cost in growth has also been tremendous, and it's breaking our bank account. With reasonable estimates for continued growth we're expecting hosting expenses totalling 75k USD this year, and 90k USD next year. With the current sponsors we've set up we can't sustain that. We estimate that hosting expenses for gitlab.fd.o without any of the CI features enabled would total 30k USD, which is within X.org's ability to support through various sponsorships, mostly through XDC.
Note that X.org does no longer sponsor any CI runners themselves, we've stopped that. The huge additional expenses are all just in storing and serving build artifacts and images to outside CI runners sponsored by various companies. A related topic is that with the growth in fd.o it's becoming infeasible to maintain it all on volunteer admin time. X.org is therefore also looking for admin sponsorship, at least medium term.
Assuming that we want cash flow reserves for one year of gitlab.fd.o (without CI support) and a trimmed XDC and assuming no sponsor payment meanwhile, we'd have to cut CI services somewhere between May and June this year. The board is of course working on acquiring sponsors, but filling a shortfall of this magnitude is neither easy nor quick work, and we therefore decided to give an early warning as soon as possible. Any help in finding sponsors for fd.o is very much appreciated.
a) Ouch.
b) we probably need to take a large step back here.
I kinda agree, but maybe the step doesn't have to be *too* large?
I wonder if we could solve this by restructuring the project a bit. I'm talking purely from a Mesa point of view here, so it might not solve the full problem, but:
- It feels silly that we need to test changes to e.g the i965 driver
on dragonboards. We only have a big "do not run CI at all" escape- hatch.
- A lot of us are working for a company that can probably pay for
their own needs in terms of CI. Perhaps moving some costs "up front" to the company that needs it can make the future of CI for those who can't do this
- I think we need a much more detailed break-down of the cost to make
educated changes. For instance, how expensive is Docker image uploads/downloads (e.g intermediary artifacts) compared to build logs and final test-results? What kind of artifacts?
We have logs somewhere, but no one yet got around to analyzing that. Which will be quite a bit of work to do since the cloud storage is totally disconnected from the gitlab front-end, making the connection to which project or CI job caused something is going to require scripting. Volunteers definitely very much welcome I think.
It's very surprising to me that this kind of cost monitoring is treated as an afterthought, especially since one of the main jobs of the X.Org board is to keep spending under control and transparent.
Also from all the conversations it's still unclear to me if the google hosting costs are already over the sponsored credits (so is burning a hole into X.org bank account right now) or if this is only going to happen at a later point in time.
Even with CI disabled it seems that the board estimates a cost of 30k annually for the plain gitlab hosting. Is this covered by the credits sponsored by google? If not, why wasn't there a board voting on this spending? All other spending seem to require pre-approval by the board. Why wasn't gitlab hosting cost discussed much earlier in the public board meetings, especially if it's going to be such an big chunk of the overall spending of the X.Org foundation?
Regards, Lucas