[all] Curating the openstack org on GitHub
Hi everyone, As you know, for practical and "code marketing" reasons, we maintain mirrors of the opendev.org/openstack repositories on GitHub: https://github.com/openstack/ A few months ago jroll and I took on the task of maintaining the "openstack" organization on GitHub... In particular pushing "moved" final commits to repositories that are now maintained elsewhere on Opendev, and marking them retired. It's all a bit of a challenge. GitHub does not let you set the "mirror" flag by API (you have to ask GitHub support to manually do it, and they don't like it when you ask for it to be set on 1,831 repositories). The "pinned repositories" UI fails miserably and does not let us select "Nova" (again, probably too many repositories, GitHub support does not have a solution for us). There are limited opportunities to describe the fact that we actually only use GitHub as a mirror, leading to confusion. For inactive repositories (no longer under OpenStack governance), the situation is even more confusing. Some are noted "MOVED" or "RETIRED" in their description, some have a closing commit (some others don't), some have the "archived" flag. Some don't have anything and appear active while they are not (see attached CSV for those who like data). There is also the openstack-dev and openstack-attic organizations, which are leftovers from ancient times. In summary, for a "code marketing" opportunity, it does not paint a great picture. I'd like to make the following suggestion: - aggressively delete all non-openstack things from the openstack org, keeping only official, active repositories - delete the openstack-dev and openstack-attic organizations We shied away from doing that in the past, mostly to not break people who may have cloned those repositories... But I think the benefits of cleaning up now outweigh the drawbacks. The reference code always exists at opendev.org anyway. Also maybe once we are back to a more reasonable number of visible repositories, the pins UI will work again. And GitHub will actually be OK with setting the mirror flag. Thoughts, comments ? -- Thierry Carrez (ttx)
SGTM, as someone associated with a now-inactive repo ( https://github.com/openstack/networking-calico). On Wed, Apr 1, 2020 at 2:32 PM Thierry Carrez <thierry@openstack.org> wrote:
Hi everyone,
As you know, for practical and "code marketing" reasons, we maintain mirrors of the opendev.org/openstack repositories on GitHub:
A few months ago jroll and I took on the task of maintaining the "openstack" organization on GitHub... In particular pushing "moved" final commits to repositories that are now maintained elsewhere on Opendev, and marking them retired.
It's all a bit of a challenge. GitHub does not let you set the "mirror" flag by API (you have to ask GitHub support to manually do it, and they don't like it when you ask for it to be set on 1,831 repositories). The "pinned repositories" UI fails miserably and does not let us select "Nova" (again, probably too many repositories, GitHub support does not have a solution for us). There are limited opportunities to describe the fact that we actually only use GitHub as a mirror, leading to confusion.
For inactive repositories (no longer under OpenStack governance), the situation is even more confusing. Some are noted "MOVED" or "RETIRED" in their description, some have a closing commit (some others don't), some have the "archived" flag. Some don't have anything and appear active while they are not (see attached CSV for those who like data). There is also the openstack-dev and openstack-attic organizations, which are leftovers from ancient times. In summary, for a "code marketing" opportunity, it does not paint a great picture.
I'd like to make the following suggestion:
- aggressively delete all non-openstack things from the openstack org, keeping only official, active repositories - delete the openstack-dev and openstack-attic organizations
We shied away from doing that in the past, mostly to not break people who may have cloned those repositories... But I think the benefits of cleaning up now outweigh the drawbacks. The reference code always exists at opendev.org anyway. Also maybe once we are back to a more reasonable number of visible repositories, the pins UI will work again. And GitHub will actually be OK with setting the mirror flag.
Thoughts, comments ?
-- Thierry Carrez (ttx)
Hello Thierry, I'm all for it, however would try to talk to GitHub/Microsoft and get their support on keeping our mirrors there up and running officially. Would we may be need to have a formal contract with them? Whenever I got-clone code, I use opendev. But what I really like in GitHub in comparison to the Gitea we use, is advanced search functionality and git repo statistics (commits volume, per-contributor commits volume, etc.). Best regards, Roman Gorshunov On Wed, Apr 1, 2020 at 3:40 PM Thierry Carrez <thierry@openstack.org> wrote:
Hi everyone,
As you know, for practical and "code marketing" reasons, we maintain mirrors of the opendev.org/openstack repositories on GitHub:
A few months ago jroll and I took on the task of maintaining the "openstack" organization on GitHub... In particular pushing "moved" final commits to repositories that are now maintained elsewhere on Opendev, and marking them retired.
It's all a bit of a challenge. GitHub does not let you set the "mirror" flag by API (you have to ask GitHub support to manually do it, and they don't like it when you ask for it to be set on 1,831 repositories). The "pinned repositories" UI fails miserably and does not let us select "Nova" (again, probably too many repositories, GitHub support does not have a solution for us). There are limited opportunities to describe the fact that we actually only use GitHub as a mirror, leading to confusion.
For inactive repositories (no longer under OpenStack governance), the situation is even more confusing. Some are noted "MOVED" or "RETIRED" in their description, some have a closing commit (some others don't), some have the "archived" flag. Some don't have anything and appear active while they are not (see attached CSV for those who like data). There is also the openstack-dev and openstack-attic organizations, which are leftovers from ancient times. In summary, for a "code marketing" opportunity, it does not paint a great picture.
I'd like to make the following suggestion:
- aggressively delete all non-openstack things from the openstack org, keeping only official, active repositories - delete the openstack-dev and openstack-attic organizations
We shied away from doing that in the past, mostly to not break people who may have cloned those repositories... But I think the benefits of cleaning up now outweigh the drawbacks. The reference code always exists at opendev.org anyway. Also maybe once we are back to a more reasonable number of visible repositories, the pins UI will work again. And GitHub will actually be OK with setting the mirror flag.
Thoughts, comments ?
-- Thierry Carrez (ttx)
On 01/04/2020 14:32, Thierry Carrez wrote:
Hi everyone,
<snip>
I'd like to make the following suggestion:
- aggressively delete all non-openstack things from the openstack org, keeping only official, active repositories
I think we have gotten to a point where this is warranted
- delete the openstack-dev and openstack-attic organizations
If we do delete these orgs, could we maybe leave the org in place with a single repo pointing people to opendev / or a table of old repo -> new repos? I agree with the clean up, but I don't want to break people and leave them with no guidance.
We shied away from doing that in the past, mostly to not break people who may have cloned those repositories... But I think the benefits of cleaning up now outweigh the drawbacks. The reference code always exists at opendev.org anyway. Also maybe once we are back to a more reasonable number of visible repositories, the pins UI will work again. And GitHub will actually be OK with setting the mirror flag.
Thoughts, comments ?
What do you think about creating a new openstack archive organization github.com/openstack-archive/ (or thing like that) and tranfert them to this organization? Github properly manage redirection and transfert, it could be a proper way to keep projects reachable for searching with the github search engine (for some reasons) and to allow us to clean things on github.com/openstack. Le mer. 1 avr. 2020 à 16:38, Graham Hayes <gr@ham.ie> a écrit :
On 01/04/2020 14:32, Thierry Carrez wrote:
Hi everyone,
<snip>
I'd like to make the following suggestion:
- aggressively delete all non-openstack things from the openstack org, keeping only official, active repositories
I think we have gotten to a point where this is warranted
- delete the openstack-dev and openstack-attic organizations
If we do delete these orgs, could we maybe leave the org in place with a single repo pointing people to opendev / or a table of old repo -> new repos?
I agree with the clean up, but I don't want to break people and leave them with no guidance.
We shied away from doing that in the past, mostly to not break people who may have cloned those repositories... But I think the benefits of cleaning up now outweigh the drawbacks. The reference code always exists at opendev.org anyway. Also maybe once we are back to a more reasonable number of visible repositories, the pins UI will work again. And GitHub will actually be OK with setting the mirror flag.
Thoughts, comments ?
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
On 4/1/20 10:28 AM, Herve Beraud wrote:
What do you think about creating a new openstack archive organization github.com/openstack-archive/ <http://github.com/openstack-archive/> (or thing like that) and tranfert them to this organization?
This is basically what openstack-attic was. It would be nicer for anyone who has cloned these repos to move them there instead of deleting. It does still leave a bunch of essentially dead repos lying around though, so I'm not sure if that completely addresses the cleanup aspect of this effort. I'm also curious if all of the repos involved here are inactive, or if there are still things in openstack-dev that were moved to the openstack namespace but are still mirrored to github in the old org.
Github properly manage redirection and transfert, it could be a proper way to keep projects reachable for searching with the github search engine (for some reasons) and to allow us to clean things on github.com/openstack <http://github.com/openstack>.
Le mer. 1 avr. 2020 à 16:38, Graham Hayes <gr@ham.ie <mailto:gr@ham.ie>> a écrit :
On 01/04/2020 14:32, Thierry Carrez wrote: > Hi everyone, >
<snip>
> I'd like to make the following suggestion: > > - aggressively delete all non-openstack things from the openstack org, > keeping only official, active repositories
I think we have gotten to a point where this is warranted
> - delete the openstack-dev and openstack-attic organizations
If we do delete these orgs, could we maybe leave the org in place with a single repo pointing people to opendev / or a table of old repo -> new repos?
I agree with the clean up, but I don't want to break people and leave them with no guidance.
> We shied away from doing that in the past, mostly to not break people > who may have cloned those repositories... But I think the benefits of > cleaning up now outweigh the drawbacks. The reference code always exists > at opendev.org <http://opendev.org> anyway. Also maybe once we are back to a more reasonable > number of visible repositories, the pins UI will work again. And GitHub > will actually be OK with setting the mirror flag. > > Thoughts, comments ? >
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Ben Nemec wrote:
On 4/1/20 10:28 AM, Herve Beraud wrote:
What do you think about creating a new openstack archive organization github.com/openstack-archive/ <http://github.com/openstack-archive/> (or thing like that) and tranfert them to this organization?
This is basically what openstack-attic was. [...]
Except openstack-attic was defined and replicated from our side, not the result of a GitHub transfer. So if we were to do the same (rename all projects on opendev, sync them to GitHub), users of old repositories on GitHub would not be automagically transferred. -- Thierry Carrez (ttx)
On 4/1/20 11:07 AM, Thierry Carrez wrote:
Ben Nemec wrote:
On 4/1/20 10:28 AM, Herve Beraud wrote:
What do you think about creating a new openstack archive organization github.com/openstack-archive/ <http://github.com/openstack-archive/> (or thing like that) and tranfert them to this organization?
This is basically what openstack-attic was. [...]
Except openstack-attic was defined and replicated from our side, not the result of a GitHub transfer. So if we were to do the same (rename all projects on opendev, sync them to GitHub), users of old repositories on GitHub would not be automagically transferred.
Would it be simpler in this case though. We would just: 1. Stop mirroring retired repos from our gitea to GitHub 2. Manually do a repo transfer from openstack/ to openstack-attic/ on GitHub 3. Set the openstack-attic/repo to Archived Then anyone that still tries to clone from the old openstack namespaces GitHub location will still get the files, but from that point on we don't need to worry about mirroring or ongoing maintenance. It's basically just a historical record. And if someone wants to for whatever reason, they can fork that repo and do whatever they want to do with it. Sean
On 2020-04-01 11:15:37 -0500 (-0500), Sean McGinnis wrote: [...]
1. Stop mirroring retired repos from our gitea to GitHub [...]
At the moment, there is legacy configuration in place instructing OpenDev's Gerrit to replicate all repositories with names matching ^openstack/.* to the openstack organization on GitHub. Gerrit is going to continue to try to (re)replicate all these retired repos within the openstack namespace. This is probably the time to talk about switching OpenStack's active deliverables to using a Zuul job for replicating to GitHub like some of the other namespaces in OpenDev have been doing. We'd dearly love to drop that GitHub remote from our Gerrit replication config. -- Jeremy Stanley
On 01/04/2020 17:32, Jeremy Stanley wrote:
On 2020-04-01 11:15:37 -0500 (-0500), Sean McGinnis wrote: [...]
1. Stop mirroring retired repos from our gitea to GitHub [...]
At the moment, there is legacy configuration in place instructing OpenDev's Gerrit to replicate all repositories with names matching ^openstack/.* to the openstack organization on GitHub. Gerrit is going to continue to try to (re)replicate all these retired repos within the openstack namespace.
This is probably the time to talk about switching OpenStack's active deliverables to using a Zuul job for replicating to GitHub like some of the other namespaces in OpenDev have been doing. We'd dearly love to drop that GitHub remote from our Gerrit replication config.
This is probably a good thing regardless of what we decide here - in theory, it would be removing the remotes, and then injecting a new job into all openstack/ projects that looks something like [1], and runs as a post job, right? 1 - https://opendev.org/recordsansible/ara/src/branch/master/.zuul.d/jobs.yaml#L...
On 2020-04-01 18:20:08 +0100 (+0100), Graham Hayes wrote:
On 01/04/2020 17:32, Jeremy Stanley wrote: [...]
This is probably the time to talk about switching OpenStack's active deliverables to using a Zuul job for replicating to GitHub like some of the other namespaces in OpenDev have been doing. We'd dearly love to drop that GitHub remote from our Gerrit replication config.
This is probably a good thing regardless of what we decide here - in theory, it would be removing the remotes, and then injecting a new job into all openstack/ projects that looks something like [1], and runs as a post job, right? [...]
Basically, yes. We might be able to further centralize some of that so as to reduce the amount of boilerplate projects would need. -- Jeremy Stanley
Sean McGinnis wrote:
[...] We would just:
1. Stop mirroring retired repos from our gitea to GitHub 2. Manually do a repo transfer from openstack/ to openstack-attic/ on GitHub 3. Set the openstack-attic/repo to Archived
Then anyone that still tries to clone from the old openstack namespaces GitHub location will still get the files, but from that point on we don't need to worry about mirroring or ongoing maintenance. It's basically just a historical record. And if someone wants to for whatever reason, they can fork that repo and do whatever they want to do with it.
Assuming Matt is right and org-to-org transfer does not generate manual confirmation if they have a shared owner, that's definitely a possibility. I prefer openstack-archive because openstack-attic actually exists on opendev so having it in two places but containing different things is likely to be confusing. I'd rather transfer openstack-attic to openstack-archive as well. -- Thierry Carrez (ttx)
On 01/04/2020 17:36, Thierry Carrez wrote:
Sean McGinnis wrote:
[...] We would just:
1. Stop mirroring retired repos from our gitea to GitHub 2. Manually do a repo transfer from openstack/ to openstack-attic/ on GitHub 3. Set the openstack-attic/repo to Archived
Then anyone that still tries to clone from the old openstack namespaces GitHub location will still get the files, but from that point on we don't need to worry about mirroring or ongoing maintenance. It's basically just a historical record. And if someone wants to for whatever reason, they can fork that repo and do whatever they want to do with it.
Assuming Matt is right and org-to-org transfer does not generate manual confirmation if they have a shared owner, that's definitely a possibility.
I prefer openstack-archive because openstack-attic actually exists on opendev so having it in two places but containing different things is likely to be confusing. I'd rather transfer openstack-attic to openstack-archive as well.
That seems sane to me
On 4/1/20 11:36 AM, Thierry Carrez wrote:
Sean McGinnis wrote:
[...] We would just:
1. Stop mirroring retired repos from our gitea to GitHub 2. Manually do a repo transfer from openstack/ to openstack-attic/ on GitHub 3. Set the openstack-attic/repo to Archived
Then anyone that still tries to clone from the old openstack namespaces GitHub location will still get the files, but from that point on we don't need to worry about mirroring or ongoing maintenance. It's basically just a historical record. And if someone wants to for whatever reason, they can fork that repo and do whatever they want to do with it.
Assuming Matt is right and org-to-org transfer does not generate manual confirmation if they have a shared owner, that's definitely a possibility.
I prefer openstack-archive because openstack-attic actually exists on opendev so having it in two places but containing different things is likely to be confusing. I'd rather transfer openstack-attic to openstack-archive as well.
Good point. +1 from me.
On Wed, 2020-04-01 at 18:36 +0200, Thierry Carrez wrote:
Sean McGinnis wrote:
[...] We would just:
1. Stop mirroring retired repos from our gitea to GitHub 2. Manually do a repo transfer from openstack/ to openstack-attic/ on GitHub 3. Set the openstack-attic/repo to Archived
Then anyone that still tries to clone from the old openstack namespaces GitHub location will still get the files, but from that point on we don't need to worry about mirroring or ongoing maintenance. It's basically just a historical record. And if someone wants to for whatever reason, they can fork that repo and do whatever they want to do with it.
Assuming Matt is right and org-to-org transfer does not generate manual confirmation if they have a shared owner, that's definitely a possibility.
I prefer openstack-archive because openstack-attic actually exists on opendev so having it in two places but containing different things is likely to be confusing. I'd rather transfer openstack-attic to openstack-archive as well.
LGTM. Thanks for the work on the cleanup. It's welcomed, or should I dare to say ... even necessary! Regards, Jean-Philippe Evrard
Herve Beraud wrote:
What do you think about creating a new openstack archive organization github.com/openstack-archive/ <http://github.com/openstack-archive/> (or thing like that) and tranfert them to this organization?
Github properly manage redirection and transfert, it could be a proper way to keep projects reachable for searching with the github search engine (for some reasons) and to allow us to clean things on github.com/openstack <http://github.com/openstack>.
The trick is that transfer is (if documentation is to be trusted) an async process involving the new owner clicking a link in an email to accept the transferred repository. Since this has to be done for 1,130 repositories (not even counting openstack-dev and openstack-attic), I don't think that would be practical... -- Thierry Carrez (ttx)
On Wed, Apr 01, 2020 at 06:04:48PM +0200, Thierry Carrez wrote:
Herve Beraud wrote:
What do you think about creating a new openstack archive organization github.com/openstack-archive/ <http://github.com/openstack-archive/> (or thing like that) and tranfert them to this organization?
Github properly manage redirection and transfert, it could be a proper way to keep projects reachable for searching with the github search engine (for some reasons) and to allow us to clean things on github.com/openstack <http://github.com/openstack>.
The trick is that transfer is (if documentation is to be trusted) an async process involving the new owner clicking a link in an email to accept the transferred repository. Since this has to be done for 1,130 repositories (not even counting openstack-dev and openstack-attic), I don't think that would be practical...
This is true unless the user doing the transfer is an owner on both sides of the transfer. Then it should just be a matter of pushing the button/making the API request and there is no confirmation needed (I did this just the other day, transferring a repo from my personal account to an organization I'm an owner of). -Matt Treinish
On 2020-04-01 12:26:57 -0400 (-0400), Matthew Treinish wrote: [...]
This is true unless the user doing the transfer is an owner on both sides of the transfer. Then it should just be a matter of pushing the button/making the API request and there is no confirmation needed (I did this just the other day, transferring a repo from my personal account to an organization I'm an owner of).
In fact, this was the only way to do a transfer between orgs for quite some time (so you had to temporarily add the user to one or the other org), and the confirmation workflow was only added more recently. -- Jeremy Stanley
On 01.04.20 15:32, Thierry Carrez wrote:
[...] I'd like to make the following suggestion:
- aggressively delete all non-openstack things from the openstack org, keeping only official, active repositories
Please double check your list, you missed all the repos listed in https://opendev.org/openstack/governance/src/branch/master/reference/sigs-re... - those should be treated as official as well, Andreas -- Andreas Jaeger aj@suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB
Andreas Jaeger wrote:
On 01.04.20 15:32, Thierry Carrez wrote:
[...] I'd like to make the following suggestion:
- aggressively delete all non-openstack things from the openstack org, keeping only official, active repositories
Please double check your list, you missed all the repos listed in https://opendev.org/openstack/governance/src/branch/master/reference/sigs-re... - those should be treated as official as well,
Good catch! I'll fix it before activating... the goal of the early analysis was to expose the variety of cases. -- Thierry Carrez (ttx)
OK, so to summarize, the now-proposed plan is to: 0. Create an openstack-archive organization on GitHub before some org-squatter steals it [DONE] 1. Build a list of official openstack repositories, not forgetting to include SIG, board and UC-owned ones 2. Remove openstack namespace-wide mirroring, replace it with repo-specific jobs for official repositories 3. Move the (no-longer replicated) non-official repositories from the openstack org to the openstack-archive one, mark them "Archived" if they are not already 4. Move the repositories from openstack-attic and openstack-dev organizations on GitHub to openstack-archive as well I will start proceeding on those tasks unless there are new objections posted before end of week. Open questions: - We have a bunch of stale repositories under the openstack-infra organization on GitHub, should we also move them to openstack-archive ? - After the repo transfer, should we destroy the no-longer used openstack-attic and openstack-dev (and openstack-infra) organizations, or are they somehow needed for the automagic redirection to happen ? -- Thierry Carrez (ttx)
On 02/04/2020 13:07, Thierry Carrez wrote:
OK, so to summarize, the now-proposed plan is to:
0. Create an openstack-archive organization on GitHub before some org-squatter steals it [DONE]
1. Build a list of official openstack repositories, not forgetting to include SIG, board and UC-owned ones
2. Remove openstack namespace-wide mirroring, replace it with repo-specific jobs for official repositories
3. Move the (no-longer replicated) non-official repositories from the openstack org to the openstack-archive one, mark them "Archived" if they are not already
4. Move the repositories from openstack-attic and openstack-dev organizations on GitHub to openstack-archive as well
I will start proceeding on those tasks unless there are new objections posted before end of week.
Open questions:
- We have a bunch of stale repositories under the openstack-infra organization on GitHub, should we also move them to openstack-archive ?
- After the repo transfer, should we destroy the no-longer used openstack-attic and openstack-dev (and openstack-infra) organizations, or are they somehow needed for the automagic redirection to happen ?
If only to avoid someone squatting on the org, I think we should keep them. Not sure if we need them for the redirects, but I think it is safer to keep an empty org. - Graham
On Thu, 2020-04-02 at 13:39 +0100, Graham Hayes wrote:
If only to avoid someone squatting on the org, I think we should keep them. Not sure if we need them for the redirects, but I think it is safer to keep an empty org.
I don't think it hurts to keep them, and point to archive in their description/displayed name. Regards, JP
Thierry Carrez <thierry@openstack.org> writes:
OK, so to summarize, the now-proposed plan is to:
0. Create an openstack-archive organization on GitHub before some org-squatter steals it [DONE]
1. Build a list of official openstack repositories, not forgetting to include SIG, board and UC-owned ones
2. Remove openstack namespace-wide mirroring, replace it with repo-specific jobs for official repositories
Mohammed was asking about how to make this more efficient using nodeless jobs; here's an idea: We should be able to add a nodeless job in one of the trusted repos (either opendev/base-jobs or openstack/project-config) and users can supply a secret in the repo. That will reduce the complexity and improve the efficiency (since the push happens from the executors). I propose: * Create such a job and add it to opendev/base-jobs so it's available to every tenant. It should accept a secret that not only has an ssh key but also a regex to apply to the project to determine if that project is allowed to use the secret and/or what the target project name should be. This can be used to mitigate the fact that there are non-openstack projects in the openstack zuul tenant. The documentation promote jobs have something similar. * Create a job in openstack/project-config which inherits from it and supplies the secret for the ssh key which grants access to the openstack org so that no openstack project has to deal with that individually. This secret would specify "^openstack/.*" as the project regex mentioned above to restrict it to official openstack projects. * OpenStack projects would simply add that job to their post pipelines (either in-repo or in project-config). * Any non-openstack project can use the job from opendev/base-jobs and provide their own secret. I think we should set that up (and confirm it works) before we do any mass replication job changes. -Jim
On 2020-04-09 16:53:09 -0700 (-0700), James E. Blair wrote:
Thierry Carrez <thierry@openstack.org> writes: [...] * Create a job in openstack/project-config which inherits from it and supplies the secret for the ssh key which grants access to the openstack org so that no openstack project has to deal with that individually.
Something like the openstack-mirror-on-github job added by https://review.opendev.org/718479 but adding...
This secret would specify "^openstack/.*" as the project regex mentioned above to restrict it to official openstack projects.
Because as you pointed out in IRC, this job can actually be added to any project in-repo right now and since it ignored the namespace part of the repo name but hard-codes the destination to the openstack org, it allows a potential x/nova repo to fight with openstack/nova over replication to the same target and all the possible security implications thereof. Reverted Thierry's PoC for the moment with https://review.opendev.org/718839 but we should repropose following the plan you've outlined.
* OpenStack projects would simply add that job to their post pipelines (either in-repo or in project-config). [...]
In project-config I guess, because we'll want to also replicate on tag events and implicit branch matching for branched projects will prevent that from working if added in-repo.
I think we should set that up (and confirm it works) before we do any mass replication job changes.
I absolutely agree. The idea was to test carefully before adding this to any non-test repos anyway. -- Jeremy Stanley
Jeremy Stanley wrote:
On 2020-04-09 16:53:09 -0700 (-0700), James E. Blair wrote: [...]
* Create a job in openstack/project-config which inherits from it and supplies the secret for the ssh key which grants access to the openstack org so that no openstack project has to deal with that individually.
Something like the openstack-mirror-on-github job added by https://review.opendev.org/718479 but adding...
This secret would specify "^openstack/.*" as the project regex mentioned above to restrict it to official openstack projects.
Also adding nodeless operation and moving it to opendev/base-jobs.
Because as you pointed out in IRC, this job can actually be added to any project in-repo right now and since it ignored the namespace part of the repo name but hard-codes the destination to the openstack org, it allows a potential x/nova repo to fight with openstack/nova over replication to the same target and all the possible security implications thereof.
Reverted Thierry's PoC for the moment with https://review.opendev.org/718839 but we should repropose following the plan you've outlined.
* OpenStack projects would simply add that job to their post pipelines (either in-repo or in project-config). [...]
In project-config I guess, because we'll want to also replicate on tag events and implicit branch matching for branched projects will prevent that from working if added in-repo.
I think we should set that up (and confirm it works) before we do any mass replication job changes.
I absolutely agree. The idea was to test carefully before adding this to any non-test repos anyway.
That all sounds good to me. Regarding implementation, could someone who knows what they are doing create that nodeless secret-driven-regexped git-mirroring job in opendev/base-jobs? I'll be happy to take it from there :) -- Thierry Carrez (ttx)
On Fri, Apr 10, 2020 at 8:06 AM Thierry Carrez <thierry@openstack.org> wrote:
Jeremy Stanley wrote:
On 2020-04-09 16:53:09 -0700 (-0700), James E. Blair wrote: [...]
* Create a job in openstack/project-config which inherits from it and supplies the secret for the ssh key which grants access to the openstack org so that no openstack project has to deal with that individually.
Something like the openstack-mirror-on-github job added by https://review.opendev.org/718479 but adding...
This secret would specify "^openstack/.*" as the project regex mentioned above to restrict it to official openstack projects.
Also adding nodeless operation and moving it to opendev/base-jobs.
Because as you pointed out in IRC, this job can actually be added to any project in-repo right now and since it ignored the namespace part of the repo name but hard-codes the destination to the openstack org, it allows a potential x/nova repo to fight with openstack/nova over replication to the same target and all the possible security implications thereof.
Reverted Thierry's PoC for the moment with https://review.opendev.org/718839 but we should repropose following the plan you've outlined.
* OpenStack projects would simply add that job to their post pipelines (either in-repo or in project-config). [...]
In project-config I guess, because we'll want to also replicate on tag events and implicit branch matching for branched projects will prevent that from working if added in-repo.
I think we should set that up (and confirm it works) before we do any mass replication job changes.
I absolutely agree. The idea was to test carefully before adding this to any non-test repos anyway.
That all sounds good to me. Regarding implementation, could someone who knows what they are doing create that nodeless secret-driven-regexped git-mirroring job in opendev/base-jobs? I'll be happy to take it from there :)
opendev/base-jobs work is done and landed: https://review.opendev.org/#/c/719032/ openstack/project-config base job is pending one more +W: https://review.opendev.org/#/c/719047/ once that is done, we should be good to go to test it and move towards it :)
-- Thierry Carrez (ttx)
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. https://vexxhost.com
Thierry Carrez wrote:
OK, so to summarize, the now-proposed plan is to:
0. Create an openstack-archive organization on GitHub before some org-squatter steals it [DONE]
1. Build a list of official openstack repositories, not forgetting to include SIG, board and UC-owned ones
2. Remove openstack namespace-wide mirroring, replace it with repo-specific jobs for official repositories
Tasks 1 and 2 were recently completed. I'll move on to task 3 and 4 in the coming days:
3. Move the (no-longer replicated) non-official repositories from the openstack org to the openstack-archive one, mark them "Archived" if they are not already
4. Move the repositories from openstack-attic and openstack-dev organizations on GitHub to openstack-archive as well. Keep the orgs empty.
We still have one open question:
Open questions:
- We have a bunch of stale repositories under the openstack-infra organization on GitHub, should we also move them to openstack-archive ?
-- Thierry Carrez (ttx)
participants (13)
-
Andreas Jaeger
-
Ben Nemec
-
corvus＠inaugust.com
-
Graham Hayes
-
Herve Beraud
-
Jean-Philippe Evrard
-
Jeremy Stanley
-
Matthew Treinish
-
Mohammed Naser
-
Neil Jerram
-
Roman Gorshunov
-
Sean McGinnis
-
Thierry Carrez