[Changing subject/recipients, to avoid hijacking the thread]
Hi Dan,
On Wed, 22 Aug 2018 at 17:29, Daniel Stone daniel@fooishbar.org wrote:
Hi,
On Wed, 22 Aug 2018 at 16:02, Emil Velikov emil.l.velikov@gmail.com wrote:
On 22 August 2018 at 12:44, Daniel Vetter daniel.vetter@ffwll.ch wrote:
I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
- fd.o admins want to deprecate shell accounts and hand-rolled
infrastructure, because it's a pain to keep secure&updated.
- gitlab will allow us to add committers on our own, greatly
simplifying that process (and offloading that task from fd.o admins).
Random thought - I really wish the admins spoke early and louder about issues. From infra to manpower and adhoc tool maintenance.
I thought I mostly had it covered, but fair enough. What knowledge are you missing and how should it best be delivered?
One first-order issue is that as documented at https://www.freedesktop.org/wiki/AccountRequests/ creating accounts requires manual admin intervention. You can also search the 'freedesktop.org' product on Bugzilla to see the amount of time we spend shuffling around GPG & SSH keys, which is about the worst possible use of my time. Many other people have had access to drive the ancient shell-script frontend to LDAP before, but for some reason they mostly aren't very enthusiastic about doing it all the time.
In the mesa-dev@ thread about Mesa's migration, which is linked from my blog post, I went into quite a lot of detail about why Jenkins was not suitable to roll out across fd.o globally. That knowledge was gained from trial & error, which was a lot of time burnt. The end result is that we don't have any CI, except if people hang Travis/AppVeyor off GitHub mirrors.
You've personally seen what's involved in setting up Git repository hooks so we can build docs. We can't give access to let people work on those, without giving them direct access to the literal Git repository itself on disk. The hooks mostly involve bespoke sets of rsync jobs and hand-managed SSH credentials and external services to build docs and so on and so forth. None of this is accessible to a newcomer who wants to make a non-code contribution: you have to find someone with access to the repo to go bash some shell scripts directly and hope you didn't screw it up. None of this is auditable, so if the repo mysteriously gets wiped, then hopefully there are some backups somewhere. But there definitely aren't any logs. Luckily we prevent most people from having access to most repositories via a mandatory 'shell' which only allows people to push Git; this was written by hand by us 12 years ago.
What else? Our fork of Patchwork was until recently based on an ancient fork of Django, in a bespoke unreproducible production environment. Bugzilla is patched for spam and templates (making upgrades complex), yet we still have a surprising amount of spam pass through. There's no way to delete spam, but you have to manually move every bug to the 'spam' group, then go through and delete the user which involves copying & pasting the email and a few clicks per user. Mailman is patched to support Recaptcha, bringing more upgrade fun. ikiwiki breaks all the time because it's designed to have the public-access web server on the same host as the writeable Git repositories.
Our servers are several years old and approaching life expiry, and we have no more spare disk. You can see in #freedesktop IRC the constant network connectivity issues people have to PSU almost every day. Our attempts to root-cause and solve those have got nowhere.
I could go on, but the 'moved elsewhere' list in https://gitlab.freedesktop.org/freedesktop/freedesktop/issues/2 indicates that we do have problems to solve, and that changing peoples' SSH keys for them isn't the best way for us to be spending the extremely limited time that we do have.
Starting with the most important - regardless of what may come as nitpicking, I do admire the time, patience and effort that you've been putting in fd.o.
Based on your blog-post, there are many issues beyond users usability (think people using cgit/annongit, pw failing to parse email, etc). And yes, I've read it a couple of times as it came out.
You mentioned many of those, so let me respin that a bit: - old hacky/adhoc scripts - throw those over a git repo - annoying and/or admin requiring workflow - throw some suggestions about tools in ^^ - ageing servers - increasing maintenance
People working on graphics [more or less familiar with some issues] may not be the best admin/tools engineers out there. Hence publicity, be that via blog post/XDC talk/other, is very important IMHO.
Don't recall a blog or XDC (lighting) talk over the last 5 years - and seemingly during that time, more issues have been pilling up.
I believe that people would have responded to such publicity, either by writing/fixing tools, monetary or otherwise. Perhaps not to a massive extend, but another thing to explore is funding from X.org to hide contractors to help with some issues. IIRC that was the case with the wiki changes a few years back.
Admittedly, I'm not familiar with the finances that X.org has.
It feels that you (and fellow admins) have been testing the limits of your perseverance, against a mounting amount of problems and pressure. And although you were exploring alternatives, only a limited group of people knew the struggles you're faced with.
Thus my initial statement. I hope you'll see what i am aiming here
It could be that I'm too naive about the impact that would have? Yet without trying, one will never know.
HTH Emil