On Wed, Apr 13, 2022, at 12:11 AM, Ian Wienand wrote:
On Tue, Apr 12, 2022 at 05:05:22PM -0700, Michael Johnson wrote: 65;6602;1c> tldr: All devstack based jobs are going to fail with newer versions of
git - don't bother rechecking
git has released a security fix [1] that is starting to roll out in distributions (Ubuntu focal for example) that will cause pbr to be unable to access the package metadata for packages checked out locally due to the directory ownership used in devstack.
This turns out to be annoyingly complicated.
Since devstack checks out all code as "stack" and then installs globally with "sudo pip install -e ...", pbr will be running in a directory owned by "stack" as root and its git calls will hit this failure.
If we make the code directories owned by root, we now have additional problems. Several places do things in the code repositories -- e.g. setup virtualenvs, run ./tools/*.sh scripts to generate sample config files and run tox as "stack" (tox then tries to install the source tree in it's virtualenv -- if it's owned by root -- again -- failure).
I explored a bunch of these options in
https://review.opendev.org/c/openstack/devstack/+/837636
and anyone feel free to take over that and keep trying.
The other option is to use the new config flag to mark our checkouts as safe. This is obviously simpler, but it seems like a very ugly thing for a nominally generic tool like devstack to do to your global git config. This is done with
https://review.opendev.org/c/openstack/devstack/+/837659
and appears to work; but will need backporting for grenade if we want to take this path.
This ended up being the quickest option to unblocking things so we backported it all the way through to Victoria then landed the changes from Victoria up to master in that order. This means that devstack testing should work again and you can recheck/approve/push changes once again. However, we noticed that these changes don't quite work on Ubuntu Bionic just on Ubuntu Focal. Dan pushed up https://review.opendev.org/c/openstack/devstack/+/837759 to address the Bionic problem and make unstack clean up after ourselves. Once this lands to master we can backport it using our typical backporting process. Finally fungi has been working on https://review.opendev.org/c/openstack/devstack/+/837731 to separate the package creation step from the package installation step. This allows us to build the python package as the stack user and do the install as root avoiding any git concerns about different ownership of repositories. As the commit message in that change notes this effectively means that we cannot have editable installs anymore. If we decide that is a necessary feature of devstack then I think we should look into resurrecting https://review.opendev.org/c/openstack/devstack/+/558930 to have devstack install into a global virtualenv. Then stack can own the virtualenv, and there is no git concern about file ownership. In the past this change sort of died out as it is quite a large change to how devstack operates and will potentially have significant fallout of its own if we land it and there just didn't seem to be a will to go through that. Maybe this situation has changed our opinion on that. Others should feel free to push updates to that change as I'm not sure I'll have time to dedicate to it again.
When this kicked off I sent in a link to HN thinking that thanks to our very upstream focused CI we were likely some of the first to hit this; it's currently the top post so I think that is accurate that this is having wide impact:
https://news.ycombinator.com/item?id=31009675
It is probably worth keeping one eye on upstream for any developments that might change our options.
-i