Hi all,
I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
- fd.o admins want to deprecate shell accounts and hand-rolled infrastructure, because it's a pain to keep secure&updated.
- gitlab will allow us to add committers on our own, greatly simplifying that process (and offloading that task from fd.o admins).
There's also some more benefits we might want to reap, like better CI integration for basic build testing - no more "oops didn't build drm-misc defconfigs" or "sry, forgot make check in maintainer-tools". But that's all fully optional.
For the full in-depth writeup of everything, see
https://www.fooishbar.org/blog/gitlab-fdo-introduction/
I think now is also a good time, with mesa, xorg, wayland/weston and others moved, to start thinking about how we'll move drm. There's a few things to figure out though:
- We probably want to split out maintainer-tools. That would address the concern that there's 50+ committers to an auto-updating shell script ...
- We need to figure out how to handle the ACL trickery around drm-tip in gitlab.
- Probably good to stage the migration, with maintainer-tools, igt leading. That will also make fd.o admins happy, who want to rework their cloud infrastructure a bit before migrating the big kernel repos over.
- Figuring out the actual migration - we've been adding a pile of committers since fd.o LDAP was converted to gitlab once back in spring. We need to at least figure out how to move the new accounts/committers.
- Similar, maintainer-tools needs to move. We probably want to move all the dim maintained kernel repos in one go, to avoid headaches with double-accounts needed for committers.
- CI, linux-next and everyone else should be fine, since the cgit/non-ssh paths will keep working (they'll be read-only mirrors). Need to double-check that with everyone.
- Some organization structure would be good.
https://cgit.freedesktop.org/drm
libdrm won't be part of the gitlab drm group because that's already moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
https://gitlab.freedesktop.org/mesa/drm
But there's also drm_hwcomposer, which we might want to migrate into drm too - gitlab requires a containing group, and drm_hwcomposer/drm_hwcomposer is a bit silly.
Note: Access rights can be done at any level in the hierarchy, the organization is orthogonal to commit rights.
- Anything else I've forgotten.
A lot of this still needs to be figured out first. As a first step I'm looking for volunteers who want to join the fun, besides comments and thoughts on the overall topic of course.
Cheers, Daniel
On Wed, Aug 22, 2018 at 01:44:56PM +0200, Daniel Vetter wrote:
Hi all,
I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
- fd.o admins want to deprecate shell accounts and hand-rolled
infrastructure, because it's a pain to keep secure&updated.
- gitlab will allow us to add committers on our own, greatly
simplifying that process (and offloading that task from fd.o admins).
There's also some more benefits we might want to reap, like better CI integration for basic build testing - no more "oops didn't build drm-misc defconfigs" or "sry, forgot make check in maintainer-tools". But that's all fully optional.
For the full in-depth writeup of everything, see
https://www.fooishbar.org/blog/gitlab-fdo-introduction/
I think now is also a good time, with mesa, xorg, wayland/weston and others moved, to start thinking about how we'll move drm. There's a few things to figure out though:
- We probably want to split out maintainer-tools. That would address
the concern that there's 50+ committers to an auto-updating shell script ...
/me laughs nervously
We need to figure out how to handle the ACL trickery around drm-tip in gitlab.
Probably good to stage the migration, with maintainer-tools, igt
leading. That will also make fd.o admins happy, who want to rework their cloud infrastructure a bit before migrating the big kernel repos over.
- Figuring out the actual migration - we've been adding a pile of
committers since fd.o LDAP was converted to gitlab once back in spring. We need to at least figure out how to move the new accounts/committers.
- Similar, maintainer-tools needs to move. We probably want to move
all the dim maintained kernel repos in one go, to avoid headaches with double-accounts needed for committers.
- CI, linux-next and everyone else should be fine, since the
cgit/non-ssh paths will keep working (they'll be read-only mirrors). Need to double-check that with everyone.
They can also pull the trees from git://gitlab.fd.o/blah as normal, just need to give them new pointers once we're stable.
- Some organization structure would be good.
https://cgit.freedesktop.org/drm
libdrm won't be part of the gitlab drm group because that's already moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
https://gitlab.freedesktop.org/mesa/drm
But there's also drm_hwcomposer, which we might want to migrate into drm too - gitlab requires a containing group, and drm_hwcomposer/drm_hwcomposer is a bit silly.
This seems fine to me. Our expansion plans likely aren't big enough to warrant a separate group.
Note: Access rights can be done at any level in the hierarchy, the organization is orthogonal to commit rights.
- Anything else I've forgotten.
A lot of this still needs to be figured out first. As a first step I'm looking for volunteers who want to join the fun, besides comments and thoughts on the overall topic of course.
I'm pretty keen on getting this done, so I'll volunteer some cycles if there's something that needs doing.
Sean
Cheers, Daniel
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx
On Wed, 22 Aug 2018, Daniel Vetter daniel.vetter@ffwll.ch wrote:
Hi all,
I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
- fd.o admins want to deprecate shell accounts and hand-rolled
infrastructure, because it's a pain to keep secure&updated.
- gitlab will allow us to add committers on our own, greatly
simplifying that process (and offloading that task from fd.o admins).
There's also some more benefits we might want to reap, like better CI integration for basic build testing - no more "oops didn't build drm-misc defconfigs" or "sry, forgot make check in maintainer-tools". But that's all fully optional.
For the full in-depth writeup of everything, see
https://www.fooishbar.org/blog/gitlab-fdo-introduction/
I think now is also a good time, with mesa, xorg, wayland/weston and others moved, to start thinking about how we'll move drm. There's a few things to figure out though:
- We probably want to split out maintainer-tools. That would address
the concern that there's 50+ committers to an auto-updating shell script ...
We need to figure out how to handle the ACL trickery around drm-tip in gitlab.
Probably good to stage the migration, with maintainer-tools, igt
leading. That will also make fd.o admins happy, who want to rework their cloud infrastructure a bit before migrating the big kernel repos over.
- Figuring out the actual migration - we've been adding a pile of
committers since fd.o LDAP was converted to gitlab once back in spring. We need to at least figure out how to move the new accounts/committers.
- Similar, maintainer-tools needs to move. We probably want to move
all the dim maintained kernel repos in one go, to avoid headaches with double-accounts needed for committers.
- CI, linux-next and everyone else should be fine, since the
cgit/non-ssh paths will keep working (they'll be read-only mirrors). Need to double-check that with everyone.
- Some organization structure would be good.
https://cgit.freedesktop.org/drm
libdrm won't be part of the gitlab drm group because that's already moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
https://gitlab.freedesktop.org/mesa/drm
But there's also drm_hwcomposer, which we might want to migrate into drm too - gitlab requires a containing group, and drm_hwcomposer/drm_hwcomposer is a bit silly.
Note: Access rights can be done at any level in the hierarchy, the organization is orthogonal to commit rights.
- Anything else I've forgotten.
A lot of this still needs to be figured out first. As a first step I'm looking for volunteers who want to join the fun, besides comments and thoughts on the overall topic of course.
Just a couple of concerns from drm/i915 perspective for starters:
- Patchwork integration. I think we'll want to keep patchwork for at least intel-gfx etc. for the time being. IIUC the one thing we need is some server side hook to update patchwork on git push.
- Sticking to fdo bugzilla and disabling gitlab issues for at least drm-intel for the time being. Doing that migration in the same go is a bit much I think. Reassignment across bugzilla and gitlab will be an issue.
BR, Jani.
On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
- Sticking to fdo bugzilla and disabling gitlab issues for at least drm-intel for the time being. Doing that migration in the same go is a bit much I think. Reassignment across bugzilla and gitlab will be an issue.
Can you elaborate a bit on the issues here? The actual move-the-bugs process has been pretty painless for the parts of xorg we've done so far.
- ajax
On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
- Sticking to fdo bugzilla and disabling gitlab issues for at least drm-intel for the time being. Doing that migration in the same go is a bit much I think. Reassignment across bugzilla and gitlab will be an issue.
Can you elaborate a bit on the issues here? The actual move-the-bugs process has been pretty painless for the parts of xorg we've done so far.
I guess there is nothing against moving the bugs there. The concern is only on doing everything at once.
I'm in favor of moving gits for now and after we are confident that everything is there and working we move the bugs.
One question about the bugzilla:
Will all the referrences on all commit messages get outdated after bugzilla is dead? Or bugzilla will stay up for referrence but closed for interaction? or all old closed stuff are always moved and bugzilla.fd.o as well and bugzilla.fd.o will be mirroring gitlab?
Thanks, Rodrigo.
- ajax
dim-tools mailing list dim-tools@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dim-tools
Hi Rodrigo,
On Wed, 22 Aug 2018 at 17:06, Rodrigo Vivi rodrigo.vivi@intel.com wrote:
On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
- Sticking to fdo bugzilla and disabling gitlab issues for at least drm-intel for the time being. Doing that migration in the same go is a bit much I think. Reassignment across bugzilla and gitlab will be an issue.
Can you elaborate a bit on the issues here? The actual move-the-bugs process has been pretty painless for the parts of xorg we've done so far.
I guess there is nothing against moving the bugs there. The concern is only on doing everything at once.
I'm in favor of moving gits for now and after we are confident that everything is there and working we move the bugs.
As Daniel alluded to, the only issue I really have is moving _all_ the kernel repos at once. At the end of the year we'll have easy automatic scaling thanks to the independent services being separated. As it is, all the GitLab services (apart from CI runners) run on a single machine, so we have limited options if it becomes overwhelmed with load.
Do you have a particular concern about the repos? e.g. what would you check for to make sure things are 'there and working'?
One question about the bugzilla:
Will all the referrences on all commit messages get outdated after bugzilla is dead? Or bugzilla will stay up for referrence but closed for interaction? or all old closed stuff are always moved and bugzilla.fd.o as well and bugzilla.fd.o will be mirroring gitlab?
When bugs are migrated from Bugzilla to GitLab, only open bugs are migrated. Closed ones are left in place, as is; open ones have a comment at the end saying that the bug has moved to GitLab, a URL linking to the new GitLab issue, and telling them to please chase it up there.
Even when we move everyone completely off Bugzilla, we will keep it as a read-only mirror forever. Even with Phabricator, which very few people ever used, has had all its bugs and code review captured and archived, so we can continue to preserve all the old content and links, without having to run the actual service.
Cheers, Daniel
On Wed, Aug 22, 2018 at 05:37:22PM +0100, Daniel Stone wrote:
Hi Rodrigo,
On Wed, 22 Aug 2018 at 17:06, Rodrigo Vivi rodrigo.vivi@intel.com wrote:
On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
- Sticking to fdo bugzilla and disabling gitlab issues for at least drm-intel for the time being. Doing that migration in the same go is a bit much I think. Reassignment across bugzilla and gitlab will be an issue.
Can you elaborate a bit on the issues here? The actual move-the-bugs process has been pretty painless for the parts of xorg we've done so far.
I guess there is nothing against moving the bugs there. The concern is only on doing everything at once.
I'm in favor of moving gits for now and after we are confident that everything is there and working we move the bugs.
As Daniel alluded to, the only issue I really have is moving _all_ the kernel repos at once. At the end of the year we'll have easy automatic scaling thanks to the independent services being separated. As it is, all the GitLab services (apart from CI runners) run on a single machine, so we have limited options if it becomes overwhelmed with load.
Do you have a particular concern about the repos?
no concerns from my side about the repos itself. From my committer perspective on libdrm, mesa the migration was really smooth.
e.g. what would you check for to make sure things are 'there and working'?
more in terms of other committers getting used to it, dim working for most of committers, links in documentations and wikis updated...
but no concerns with the infra itself.
One question about the bugzilla:
Will all the referrences on all commit messages get outdated after bugzilla is dead? Or bugzilla will stay up for referrence but closed for interaction? or all old closed stuff are always moved and bugzilla.fd.o as well and bugzilla.fd.o will be mirroring gitlab?
When bugs are migrated from Bugzilla to GitLab, only open bugs are migrated. Closed ones are left in place, as is; open ones have a comment at the end saying that the bug has moved to GitLab, a URL linking to the new GitLab issue, and telling them to please chase it up there.
Even when we move everyone completely off Bugzilla, we will keep it as a read-only mirror forever. Even with Phabricator, which very few people ever used, has had all its bugs and code review captured and archived, so we can continue to preserve all the old content and links, without having to run the actual service.
Great! Thanks for all clarification, Rodrigo.
Cheers, Daniel
On Wed, 22 Aug 2018, Rodrigo Vivi rodrigo.vivi@intel.com wrote:
On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
- Sticking to fdo bugzilla and disabling gitlab issues for at least drm-intel for the time being. Doing that migration in the same go is a bit much I think. Reassignment across bugzilla and gitlab will be an issue.
Can you elaborate a bit on the issues here? The actual move-the-bugs process has been pretty painless for the parts of xorg we've done so far.
I guess there is nothing against moving the bugs there. The concern is only on doing everything at once.
No, it's not just that.
We have some automation using the bugzilla APIs directly, and someone(tm) needs to figure out how this should work with gitlab. Maybe we have a better chance of doing things with gitlab's APIs, maybe we can reduce our dependence on external logic altogether.
We have special "i915 platform" and "i915 features" fields in bugzilla. We use a mailing list default assignee. Some of us use the "whiteboard" and "keywords" fields. Etc.
I don't think figuring all this out is rocket science, but someone needs to actually do it, and get our workflows straightened out *before* we flip the switch. I'm just trying to figure out if that is a blocker to migrating the repos.
BR, Jani.
On Fri, Aug 24, 2018 at 8:52 AM, Jani Nikula jani.nikula@linux.intel.com wrote:
On Wed, 22 Aug 2018, Rodrigo Vivi rodrigo.vivi@intel.com wrote:
On Wed, Aug 22, 2018 at 10:19:19AM -0400, Adam Jackson wrote:
On Wed, 2018-08-22 at 16:13 +0300, Jani Nikula wrote:
- Sticking to fdo bugzilla and disabling gitlab issues for at least drm-intel for the time being. Doing that migration in the same go is a bit much I think. Reassignment across bugzilla and gitlab will be an issue.
Can you elaborate a bit on the issues here? The actual move-the-bugs process has been pretty painless for the parts of xorg we've done so far.
I guess there is nothing against moving the bugs there. The concern is only on doing everything at once.
No, it's not just that.
We have some automation using the bugzilla APIs directly, and someone(tm) needs to figure out how this should work with gitlab. Maybe we have a better chance of doing things with gitlab's APIs, maybe we can reduce our dependence on external logic altogether.
We have special "i915 platform" and "i915 features" fields in bugzilla. We use a mailing list default assignee. Some of us use the "whiteboard" and "keywords" fields. Etc.
I don't think figuring all this out is rocket science, but someone needs to actually do it, and get our workflows straightened out *before* we flip the switch. I'm just trying to figure out if that is a blocker to migrating the repos.
I think the mesa approach of flat-out disabling gitlab issues and merge request is solid. That way there's no confusions with contributions and bug reports stranded in the wrong place. This should allow us to handle all the different feature sets gitlab provides independently and as we see fit: Repo hosting, documentation/artifact hosting, CI integration, patch submissions, issue tracking. We can choose&pick what we think is good, and ignore everything else.
And of course if 1-2 years down the road things change, we can reevaluate. I think for a rough timeline if things go well we'll have igt and maintainer-tools migrated by end of year (just the repos, nothing else), and a solid plan for migrating all the drm* repos next year. Everything else is too far away to have solid plans for, or even just a timeline. -Daniel
On Wed, Aug 22, 2018 at 3:13 PM, Jani Nikula jani.nikula@linux.intel.com wrote:
On Wed, 22 Aug 2018, Daniel Vetter daniel.vetter@ffwll.ch wrote:
Hi all,
I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
- fd.o admins want to deprecate shell accounts and hand-rolled
infrastructure, because it's a pain to keep secure&updated.
- gitlab will allow us to add committers on our own, greatly
simplifying that process (and offloading that task from fd.o admins).
There's also some more benefits we might want to reap, like better CI integration for basic build testing - no more "oops didn't build drm-misc defconfigs" or "sry, forgot make check in maintainer-tools". But that's all fully optional.
For the full in-depth writeup of everything, see
https://www.fooishbar.org/blog/gitlab-fdo-introduction/
I think now is also a good time, with mesa, xorg, wayland/weston and others moved, to start thinking about how we'll move drm. There's a few things to figure out though:
- We probably want to split out maintainer-tools. That would address
the concern that there's 50+ committers to an auto-updating shell script ...
We need to figure out how to handle the ACL trickery around drm-tip in gitlab.
Probably good to stage the migration, with maintainer-tools, igt
leading. That will also make fd.o admins happy, who want to rework their cloud infrastructure a bit before migrating the big kernel repos over.
- Figuring out the actual migration - we've been adding a pile of
committers since fd.o LDAP was converted to gitlab once back in spring. We need to at least figure out how to move the new accounts/committers.
- Similar, maintainer-tools needs to move. We probably want to move
all the dim maintained kernel repos in one go, to avoid headaches with double-accounts needed for committers.
- CI, linux-next and everyone else should be fine, since the
cgit/non-ssh paths will keep working (they'll be read-only mirrors). Need to double-check that with everyone.
- Some organization structure would be good.
https://cgit.freedesktop.org/drm
libdrm won't be part of the gitlab drm group because that's already moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
https://gitlab.freedesktop.org/mesa/drm
But there's also drm_hwcomposer, which we might want to migrate into drm too - gitlab requires a containing group, and drm_hwcomposer/drm_hwcomposer is a bit silly.
Note: Access rights can be done at any level in the hierarchy, the organization is orthogonal to commit rights.
- Anything else I've forgotten.
A lot of this still needs to be figured out first. As a first step I'm looking for volunteers who want to join the fun, besides comments and thoughts on the overall topic of course.
Just a couple of concerns from drm/i915 perspective for starters:
Patchwork integration. I think we'll want to keep patchwork for at least intel-gfx etc. for the time being. IIUC the one thing we need is some server side hook to update patchwork on git push.
Sticking to fdo bugzilla and disabling gitlab issues for at least drm-intel for the time being. Doing that migration in the same go is a bit much I think. Reassignment across bugzilla and gitlab will be an issue.
Good points, forgot about both. Patchwork reading the mailing list should keep working as-is, but the update hook needs looking into.
Disabling gitlab issues is a non-brainer, same with merge requests. Mesa is already doing that. For bugs I think it's best to entirely leave them out for now, and maybe reconsider when/if mesa has moved. Before that I don't think gitlab issues make any sense at all.
For merge requests I think best approach is to enable them very selectively at first for testing out, and then making a per-subproject decision whether they make sense. E.g. I think for maintainer-tools integrating make check and the doc building into gitlab CI would be sweet, and worth looking into gitlab merge requests just to automate that. Again best left out of scope for the initial migration. -Daniel
Hi,
On Wed, 22 Aug 2018 at 15:44, Daniel Vetter daniel.vetter@ffwll.ch wrote:
On Wed, Aug 22, 2018 at 3:13 PM, Jani Nikula jani.nikula@linux.intel.com wrote:
Just a couple of concerns from drm/i915 perspective for starters:
Patchwork integration. I think we'll want to keep patchwork for at least intel-gfx etc. for the time being. IIUC the one thing we need is some server side hook to update patchwork on git push.
Sticking to fdo bugzilla and disabling gitlab issues for at least drm-intel for the time being. Doing that migration in the same go is a bit much I think. Reassignment across bugzilla and gitlab will be an issue.
Good points, forgot about both. Patchwork reading the mailing list should keep working as-is, but the update hook needs looking into.
All the hooks are retained. gitlab.fd.o pushes to git.fd.o, and git.fd.o still executes all the old hooks. This includes Patchwork, readthedocs, AppVeyor, and whatever else.
For merge requests I think best approach is to enable them very selectively at first for testing out, and then making a per-subproject decision whether they make sense. E.g. I think for maintainer-tools integrating make check and the doc building into gitlab CI would be sweet, and worth looking into gitlab merge requests just to automate that. Again best left out of scope for the initial migration.
You don't need MRs to do that. You can build a .gitlab-ci.yml file which will run make check or build the docs or whatever, and have that run on pushes. Anyone can then fork the repo, push their changes to that fork, and see the CI results from there. It's like Travis: the CI configuration is a (mutable) part of the codebase, not a separate 'thing' hanging off a specific repo. So if you write the CI pipeline first, you can have people running CI on push completely independently of switching the workflow to use MRs.
Cheers, Daniel
Hi Dan,
On 22 August 2018 at 12:44, Daniel Vetter daniel.vetter@ffwll.ch wrote:
Hi all,
I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
- fd.o admins want to deprecate shell accounts and hand-rolled
infrastructure, because it's a pain to keep secure&updated.
- gitlab will allow us to add committers on our own, greatly
simplifying that process (and offloading that task from fd.o admins).
Random thought - I really wish the admins spoke early and louder about issues.
From infra to manpower and adhoc tool maintenance.
There's also some more benefits we might want to reap, like better CI integration for basic build testing - no more "oops didn't build drm-misc defconfigs" or "sry, forgot make check in maintainer-tools". But that's all fully optional.
For the full in-depth writeup of everything, see
https://www.fooishbar.org/blog/gitlab-fdo-introduction/
I think now is also a good time, with mesa, xorg, wayland/weston and others moved, to start thinking about how we'll move drm. There's a few things to figure out though:
- We probably want to split out maintainer-tools. That would address
the concern that there's 50+ committers to an auto-updating shell script ...
We need to figure out how to handle the ACL trickery around drm-tip in gitlab.
Probably good to stage the migration, with maintainer-tools, igt
leading. That will also make fd.o admins happy, who want to rework their cloud infrastructure a bit before migrating the big kernel repos over.
- Figuring out the actual migration - we've been adding a pile of
committers since fd.o LDAP was converted to gitlab once back in spring. We need to at least figure out how to move the new accounts/committers.
As a observer, allow me to put some ideas. You've mostly covered them all, my emphasis is to seriously stick with _one_ thing at a time. Attempting to do multiple things in parallel will end up with sub-optimal results.
- (at random point) cleanup the committers list - people who have not contributed in the last 1 year? - setup drm group, copy/migrate accounts - one could even reuse the existing credentials - move git repos to gitlab, the push URL change, cgit mirror preserves the normal fetch ones as well as PW hooks - work out how new accounts are handled - still in bugzilla, otherwise
At this stage only workflow change is a) once-off account setup and b) pushURL update As a follow-up one can setup anything fancy. - migrate PW/other hooks - migrate bugs - if applicable - add new hooks - DRM docs, other - etc
- Similar, maintainer-tools needs to move. We probably want to move
all the dim maintained kernel repos in one go, to avoid headaches with double-accounts needed for committers.
One should be able to create a separate repo for these. And then either: - one by one add the required features into the gitlab MR machinery, - or, wire the execution in pre/post merge stage.
IIRC there are some upstream requests about the former.
- CI, linux-next and everyone else should be fine, since the
cgit/non-ssh paths will keep working (they'll be read-only mirrors). Need to double-check that with everyone.
- Some organization structure would be good.
https://cgit.freedesktop.org/drm
libdrm won't be part of the gitlab drm group because that's already moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
https://gitlab.freedesktop.org/mesa/drm
But there's also drm_hwcomposer, which we might want to migrate into drm too - gitlab requires a containing group, and drm_hwcomposer/drm_hwcomposer is a bit silly.
It did strike me a lot when drm_hwcomposer/drm_hwcomposer was introduced. Fortunately moving repos in gitlab is reasonably pain-free
HTH Emil
Hi,
On Wed, 22 Aug 2018 at 16:02, Emil Velikov emil.l.velikov@gmail.com wrote:
On 22 August 2018 at 12:44, Daniel Vetter daniel.vetter@ffwll.ch wrote:
I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
- fd.o admins want to deprecate shell accounts and hand-rolled
infrastructure, because it's a pain to keep secure&updated.
- gitlab will allow us to add committers on our own, greatly
simplifying that process (and offloading that task from fd.o admins).
Random thought - I really wish the admins spoke early and louder about issues. From infra to manpower and adhoc tool maintenance.
I thought I mostly had it covered, but fair enough. What knowledge are you missing and how should it best be delivered?
One first-order issue is that as documented at https://www.freedesktop.org/wiki/AccountRequests/ creating accounts requires manual admin intervention. You can also search the 'freedesktop.org' product on Bugzilla to see the amount of time we spend shuffling around GPG & SSH keys, which is about the worst possible use of my time. Many other people have had access to drive the ancient shell-script frontend to LDAP before, but for some reason they mostly aren't very enthusiastic about doing it all the time.
In the mesa-dev@ thread about Mesa's migration, which is linked from my blog post, I went into quite a lot of detail about why Jenkins was not suitable to roll out across fd.o globally. That knowledge was gained from trial & error, which was a lot of time burnt. The end result is that we don't have any CI, except if people hang Travis/AppVeyor off GitHub mirrors.
You've personally seen what's involved in setting up Git repository hooks so we can build docs. We can't give access to let people work on those, without giving them direct access to the literal Git repository itself on disk. The hooks mostly involve bespoke sets of rsync jobs and hand-managed SSH credentials and external services to build docs and so on and so forth. None of this is accessible to a newcomer who wants to make a non-code contribution: you have to find someone with access to the repo to go bash some shell scripts directly and hope you didn't screw it up. None of this is auditable, so if the repo mysteriously gets wiped, then hopefully there are some backups somewhere. But there definitely aren't any logs. Luckily we prevent most people from having access to most repositories via a mandatory 'shell' which only allows people to push Git; this was written by hand by us 12 years ago.
What else? Our fork of Patchwork was until recently based on an ancient fork of Django, in a bespoke unreproducible production environment. Bugzilla is patched for spam and templates (making upgrades complex), yet we still have a surprising amount of spam pass through. There's no way to delete spam, but you have to manually move every bug to the 'spam' group, then go through and delete the user which involves copying & pasting the email and a few clicks per user. Mailman is patched to support Recaptcha, bringing more upgrade fun. ikiwiki breaks all the time because it's designed to have the public-access web server on the same host as the writeable Git repositories.
Our servers are several years old and approaching life expiry, and we have no more spare disk. You can see in #freedesktop IRC the constant network connectivity issues people have to PSU almost every day. Our attempts to root-cause and solve those have got nowhere.
I could go on, but the 'moved elsewhere' list in https://gitlab.freedesktop.org/freedesktop/freedesktop/issues/2 indicates that we do have problems to solve, and that changing peoples' SSH keys for them isn't the best way for us to be spending the extremely limited time that we do have.
For the full in-depth writeup of everything, see
If you haven't seen this, or the post it was linked from, they would be a good start: https://lists.freedesktop.org/archives/freedesktop/2018-July/000370.html
There's also the long thread on mesa-dev a long time back which covers a lot of this ground already.
- Figuring out the actual migration - we've been adding a pile of
committers since fd.o LDAP was converted to gitlab once back in spring. We need to at least figure out how to move the new accounts/committers.
As a observer, allow me to put some ideas. You've mostly covered them all, my emphasis is to seriously stick with _one_ thing at a time. Attempting to do multiple things in parallel will end up with sub-optimal results.
- (at random point) cleanup the committers list - people who have not
contributed in the last 1 year?
libdrm was previously covered under the Mesa ACL. Here are the other committer lists, which you can see via 'getent group' on annarchy or another machine:
amdkfd:x:922:fxkuehl,agd5f,deathsimple,danvet,jazhou,jbridgman,hwentland,tstdenis,gitlab-mirror,rui drm-meson:x:936:narmstrong drm:x:940:airlied,danvet drm-intel:x:920:daniels,ickle,danvet,jwrdegoede,pzanoni,mlankhorst,ideak,vivijim,vsyrjala,jani,miku,mattrope,tursulin,dolphin,llandwerlin,lyudess,dpandiya,mthierry,mdnavare drm-misc:x:932:daniels,daenzer,anholt,agd5f,ickle,deathsimple,kraxel,danvet,jwrdegoede,mlankhorst,tagr,mperes,vivijim,dvdhrm,vsyrjala,pinchartl,jani,evelikov,lynxeye,dolphin,sumits,architt,seanpaul,llandwerlin,lyudess,padovan,narmstrong,hwentland,bbrezillion,shawnguo,ahajda,lukas,vinceab,benjamin.gaignard,patrik,pcornu,notro,linusw,mripard,hjc,anushas,mmind,hyunk,tomba,jsarha,wens,andr2000,alexg1,dliviu,ayankh,dlech,mdnavare,shashank,ph5
- setup drm group, copy/migrate accounts - one could even reuse the
existing credentials
Yes, dealing with accounts is covered here: https://gitlab.freedesktop.org/freedesktop/freedesktop/wikis/home#how-do-i-c...
That page is linked from the blog post as well as the freedesktop@ post.
- move git repos to gitlab, the push URL change, cgit mirror
preserves the normal fetch ones as well as PW hooks
Yes, this is what happens. The old repos give you a long message explaining how to set up your GitLab account and change your Git remote if you try to push to them.
- work out how new accounts are handled - still in bugzilla, otherwise
At this stage only workflow change is a) once-off account setup and b) pushURL update As a follow-up one can setup anything fancy.
- migrate PW/other hooks
- migrate bugs - if applicable
- add new hooks - DRM docs, other
- etc
- Similar, maintainer-tools needs to move. We probably want to move
all the dim maintained kernel repos in one go, to avoid headaches with double-accounts needed for committers.
One should be able to create a separate repo for these. And then either:
- one by one add the required features into the gitlab MR machinery,
- or, wire the execution in pre/post merge stage.
IIRC there are some upstream requests about the former.
He's just talking about migrating repository hosting (where people push), not opening up for MRs (how people review code); see the reply to Jani upthread for more detail.
- CI, linux-next and everyone else should be fine, since the
cgit/non-ssh paths will keep working (they'll be read-only mirrors). Need to double-check that with everyone.
- Some organization structure would be good.
https://cgit.freedesktop.org/drm
libdrm won't be part of the gitlab drm group because that's already moved under mesa (and you can't symlink/mulit-home anymore on gitlab):
https://gitlab.freedesktop.org/mesa/drm
But there's also drm_hwcomposer, which we might want to migrate into drm too - gitlab requires a containing group, and drm_hwcomposer/drm_hwcomposer is a bit silly.
It did strike me a lot when drm_hwcomposer/drm_hwcomposer was introduced. Fortunately moving repos in gitlab is reasonably pain-free
It is, but if they don't share an ACL and the only benefit is cosmetic, then ... meh? Weston doesn't live in drm/ either and that's OK.
Cheers, Daniel
[Changing subject/recipients, to avoid hijacking the thread]
Hi Dan,
On Wed, 22 Aug 2018 at 17:29, Daniel Stone daniel@fooishbar.org wrote:
Hi,
On Wed, 22 Aug 2018 at 16:02, Emil Velikov emil.l.velikov@gmail.com wrote:
On 22 August 2018 at 12:44, Daniel Vetter daniel.vetter@ffwll.ch wrote:
I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
- fd.o admins want to deprecate shell accounts and hand-rolled
infrastructure, because it's a pain to keep secure&updated.
- gitlab will allow us to add committers on our own, greatly
simplifying that process (and offloading that task from fd.o admins).
Random thought - I really wish the admins spoke early and louder about issues. From infra to manpower and adhoc tool maintenance.
I thought I mostly had it covered, but fair enough. What knowledge are you missing and how should it best be delivered?
One first-order issue is that as documented at https://www.freedesktop.org/wiki/AccountRequests/ creating accounts requires manual admin intervention. You can also search the 'freedesktop.org' product on Bugzilla to see the amount of time we spend shuffling around GPG & SSH keys, which is about the worst possible use of my time. Many other people have had access to drive the ancient shell-script frontend to LDAP before, but for some reason they mostly aren't very enthusiastic about doing it all the time.
In the mesa-dev@ thread about Mesa's migration, which is linked from my blog post, I went into quite a lot of detail about why Jenkins was not suitable to roll out across fd.o globally. That knowledge was gained from trial & error, which was a lot of time burnt. The end result is that we don't have any CI, except if people hang Travis/AppVeyor off GitHub mirrors.
You've personally seen what's involved in setting up Git repository hooks so we can build docs. We can't give access to let people work on those, without giving them direct access to the literal Git repository itself on disk. The hooks mostly involve bespoke sets of rsync jobs and hand-managed SSH credentials and external services to build docs and so on and so forth. None of this is accessible to a newcomer who wants to make a non-code contribution: you have to find someone with access to the repo to go bash some shell scripts directly and hope you didn't screw it up. None of this is auditable, so if the repo mysteriously gets wiped, then hopefully there are some backups somewhere. But there definitely aren't any logs. Luckily we prevent most people from having access to most repositories via a mandatory 'shell' which only allows people to push Git; this was written by hand by us 12 years ago.
What else? Our fork of Patchwork was until recently based on an ancient fork of Django, in a bespoke unreproducible production environment. Bugzilla is patched for spam and templates (making upgrades complex), yet we still have a surprising amount of spam pass through. There's no way to delete spam, but you have to manually move every bug to the 'spam' group, then go through and delete the user which involves copying & pasting the email and a few clicks per user. Mailman is patched to support Recaptcha, bringing more upgrade fun. ikiwiki breaks all the time because it's designed to have the public-access web server on the same host as the writeable Git repositories.
Our servers are several years old and approaching life expiry, and we have no more spare disk. You can see in #freedesktop IRC the constant network connectivity issues people have to PSU almost every day. Our attempts to root-cause and solve those have got nowhere.
I could go on, but the 'moved elsewhere' list in https://gitlab.freedesktop.org/freedesktop/freedesktop/issues/2 indicates that we do have problems to solve, and that changing peoples' SSH keys for them isn't the best way for us to be spending the extremely limited time that we do have.
Starting with the most important - regardless of what may come as nitpicking, I do admire the time, patience and effort that you've been putting in fd.o.
Based on your blog-post, there are many issues beyond users usability (think people using cgit/annongit, pw failing to parse email, etc). And yes, I've read it a couple of times as it came out.
You mentioned many of those, so let me respin that a bit: - old hacky/adhoc scripts - throw those over a git repo - annoying and/or admin requiring workflow - throw some suggestions about tools in ^^ - ageing servers - increasing maintenance
People working on graphics [more or less familiar with some issues] may not be the best admin/tools engineers out there. Hence publicity, be that via blog post/XDC talk/other, is very important IMHO.
Don't recall a blog or XDC (lighting) talk over the last 5 years - and seemingly during that time, more issues have been pilling up.
I believe that people would have responded to such publicity, either by writing/fixing tools, monetary or otherwise. Perhaps not to a massive extend, but another thing to explore is funding from X.org to hide contractors to help with some issues. IIRC that was the case with the wiki changes a few years back.
Admittedly, I'm not familiar with the finances that X.org has.
It feels that you (and fellow admins) have been testing the limits of your perseverance, against a mounting amount of problems and pressure. And although you were exploring alternatives, only a limited group of people knew the struggles you're faced with.
Thus my initial statement. I hope you'll see what i am aiming here
It could be that I'm too naive about the impact that would have? Yet without trying, one will never know.
HTH Emil
dri-devel@lists.freedesktop.org
-
Adam Jackson -
Daniel Stone -
Daniel Vetter -
Emil Velikov -
Jani Nikula -
Rodrigo Vivi -
Sean Paul