[infra] Missing releases from opendev.org/opendev/git-review/tags
Greetings:
I noticed that the list of tags for git-review in the website does not include the 2.2.x tags, which exist in git:
[zaitcev@niphredil git-review-0]$ grep url .git/config url = https://opendev.org/opendev/git-review.git [zaitcev@niphredil git-review-0]$ git tag -l --format='%(objectname) %(objecttype) %(refname)'| grep 2\.2 bb81051c2481d4ac983ec0b28bd62b590d361141 tag refs/tags/2.2.0 [zaitcev@niphredil git-review-0]$ git show bb81051c2481d4ac983ec0b28bd62b590d361141 | head -5 tag 2.2.0 Tagger: Jeremy Stanley fungi@yuggoth.org Date: Wed Nov 24 02:20:05 2021 +0000
Release 2.2.0 [zaitcev@niphredil git-review-0]$
The 2.2.0 is missing from https://opendev.org/opendev/git-review/tags
Coincidentally, it is also missing from https://tarballs.opendev.org/openstack/git-review/
Is this something that the infrastructure team did on purpose? Or is this just a bug to be reported?
-- Pete
On 2022-01-04 14:37:55 -0600 (-0600), Pete Zaitcev wrote:
I noticed that the list of tags for git-review in the website does not include the 2.2.x tags, which exist in git:
[...]
The 2.2.0 is missing from https://opendev.org/opendev/git-review/tags
Coincidentally, it is also missing from https://tarballs.opendev.org/openstack/git-review/
Is this something that the infrastructure team did on purpose? Or is this just a bug to be reported?
A little of both.
For the first URL I suspect that's a bug in Gitea not correctly reflecting mirrored tags. I don't recall noticing this before, but it definitely seems like Gitea is serving that view from its database and not updating whenever a new tag is pushed in. It doesn't seem to only affect the git-review repo. For example, the most recent nova tag displayed is from 2019-06-18. Skimming the OpenDev meeting agendas from around then, it looks like roughly a month later we were rebuilding the Gitea server farm (all the machines currently seem to have creation dates between 2019-07-23 and 2019-07-29), so it's quite probable that recreating the database is the only thing which has been populating the tags table in their DB. We'll have to take a look at it and probably file a bug with the Gitea maintainers.
As for the tarballs URL, we moved git-review from the openstack Git namespace to the opendev Git namespace, but don't appear to have set up a redirect for those files on the tarballs site nor moved them to the opendev tarballs directory in AFS like when we moved bindep. I'll work on cleaning that up, thanks for bringing it to my attention!
We also moved gating of git-review from the openstack Zuul tenant to the opendev Zuul tenant, where tarballs site uploads are not included in the Python release job (though I'm not opposed to adding it, probably in a child job). It's expected that the place to obtain new git-review tarballs is PyPI, as mentioned in our release announcements on the service-announce mailing list:
http://lists.opendev.org/pipermail/service-announce/2021-November/000028.htm...
I agree, all of this is a bit confusing, and could stand to get fixed up.
On Tue, 4 Jan 2022 21:20:24 +0000 Jeremy Stanley fungi@yuggoth.org wrote:
Thanks for the detailed explanations, I think I understood everything. One thing caught my attention:
It's expected that the place to obtain new git-review tarballs is PyPI, as mentioned in our release announcements on the service-announce mailing list:
http://lists.opendev.org/pipermail/service-announce/2021-November/000028.htm...
As it happens, just a short time back, I ran into an issue with PyPI.[1] Basically, it's possible to upload something there and nobody knows anything about it. Is that loss of audit trail a concern for our releases?
-- Pete
On 2022-01-04 16:13:17 -0600 (-0600), Pete Zaitcev wrote: [...]
As it happens, just a short time back, I ran into an issue with PyPI.[1] Basically, it's possible to upload something there and nobody knows anything about it. Is that loss of audit trail a concern for our releases?
-- Pete
That sounds like one of the nose maintainers uploaded a broken file to PyPI, or someone compromised one of their accounts, or hijacked the upload mechanism they were relying on. I'm not sure it's evidence that PyPI itself is untrustworthy, the same can happen (and has) in other places like NPM... really any artifact registry is susceptible if there are no cryptographic signatures or external checksums to validate the files, or if the compromise happens earlier in automation than where checksums or signatures are generated for that matter.
Was the altered code malicious? Did the maintainers publish a security advisory somewhere with details? The PyPI maintainers are generally willing to help investigate such incidents, and are in the process of pushing stronger authentication mechanisms (2FA for logins, separate upload tokens, TUF for artifact attestation).
Anyway, back to the original topic, I don't think any of us were strongly against hosting copies of the release tarballs/wheels for OpenDev's Python-based utilities, we just hadn't taken the time to set up jobs to upload them anywhere besides PyPI nor decided on any sort of signing/attestation solution (reuse what we're doing for OpenStack with the OpenStack release signing key? Create a separate OpenDev release signing key and use that? Switch OpenStack's releases to an OpenDev signing key too? Do something other than OpenPGP signatures in the wake of the SKS WoT collapse?).
On Tue, 4 Jan 2022 23:11:50 +0000 Jeremy Stanley fungi@yuggoth.org wrote:
On 2022-01-04 16:13:17 -0600 (-0600), Pete Zaitcev wrote:
Was the altered code malicious?
Not at all. It looked like an adaptation to py3 that went wrong.
-- Pete
On Tue, Jan 04, 2022 at 05:56:46PM -0600, Pete Zaitcev wrote:
On Tue, 4 Jan 2022 23:11:50 +0000 Jeremy Stanley fungi@yuggoth.org wrote:
On 2022-01-04 16:13:17 -0600 (-0600), Pete Zaitcev wrote:
Was the altered code malicious?
Not at all. It looked like an adaptation to py3 that went wrong.
So, previously, releases were uploaded to tarballs.opendev.org[3], which is not the case anymore.
To me it looks like publishing releases on tarballs.... would solve the immediate issue Pete pointed out. Also it would provide an alternate download option.
Is that doable?
Matthias
[3] https://tarballs.opendev.org/openstack/git-review/
On 2022-01-05 08:31:51 +0100 (+0100), Matthias Runge wrote: [...]
So, previously, releases were uploaded to tarballs.opendev.org[3], which is not the case anymore.
To me it looks like publishing releases on tarballs.... would solve the immediate issue Pete pointed out. Also it would provide an alternate download option.
Is that doable?
It's definitely doable, as we do it for other projects outside the opendev Zuul tenant, e.g. those in the openstack Zuul tenant. And technically we do upload tarballs for git-review to the tarballs site, but currently only branch tip tarballs not release tarballs. Later this week or early next I hope to move the prior artifacts at that URL and set up a redirect to do open{stack->dev}/git-review there as a first step, since that's where any future tarballs would appear anyway.
Once more of the OpenDev sysadmins are back from winter holidays, I'll make sure there aren't any objections to adding release tarball uploads for our Python-based tools (git-review, bindep, et cetera), and work out the most elegant solution from a job perspective. We can punt on the signing part initially, I expect, and separately decide what sor of solution and signing key we might want for that.
On 2022-01-04 21:20:24 +0000 (+0000), Jeremy Stanley wrote:
On 2022-01-04 14:37:55 -0600 (-0600), Pete Zaitcev wrote:
I noticed that the list of tags for git-review in the website does not include the 2.2.x tags, which exist in git:
[...]
The 2.2.0 is missing from https://opendev.org/opendev/git-review/tags
[...]
For the first URL I suspect that's a bug in Gitea not correctly reflecting mirrored tags. I don't recall noticing this before, but it definitely seems like Gitea is serving that view from its database and not updating whenever a new tag is pushed in. It doesn't seem to only affect the git-review repo. For example, the most recent nova tag displayed is from 2019-06-18. Skimming the OpenDev meeting agendas from around then, it looks like roughly a month later we were rebuilding the Gitea server farm (all the machines currently seem to have creation dates between 2019-07-23 and 2019-07-29), so it's quite probable that recreating the database is the only thing which has been populating the tags table in their DB. We'll have to take a look at it and probably file a bug with the Gitea maintainers.
[...]
Update on this part: After discussing briefly with clarkb in #opendev he reminded me that this is the "bad" tarballs tab we want to hide in Gitea (along with its releases tab). The branch drop-down has its own tags tab which should contain a correct and complete list of them for selecting.
participants (3)
-
Jeremy Stanley
-
Matthias Runge
-
Pete Zaitcev