[all] Devstack jobs are failing due to a git security fix
tldr: All devstack based jobs are going to fail with newer versions of git - don't bother rechecking git has released a security fix [1] that is starting to roll out in distributions (Ubuntu focal for example) that will cause pbr to be unable to access the package metadata for packages checked out locally due to the directory ownership used in devstack. We have opened a devstack bug to track this issue: https://bugs.launchpad.net/devstack/+bug/1968798 Please see the bug for the details and example error. Michael P.S. Thanks to clarkb, fungi, and ianw for helping track down the root cause! [1] https://github.com/git/git/commit/8959555cee7ec045958f9b6dd62e541affb7e7d9
On Tue, Apr 12, 2022 at 05:05:22PM -0700, Michael Johnson wrote: 65;6602;1c> tldr: All devstack based jobs are going to fail with newer versions of
git - don't bother rechecking
git has released a security fix [1] that is starting to roll out in distributions (Ubuntu focal for example) that will cause pbr to be unable to access the package metadata for packages checked out locally due to the directory ownership used in devstack.
This turns out to be annoyingly complicated. Since devstack checks out all code as "stack" and then installs globally with "sudo pip install -e ...", pbr will be running in a directory owned by "stack" as root and its git calls will hit this failure. If we make the code directories owned by root, we now have additional problems. Several places do things in the code repositories -- e.g. setup virtualenvs, run ./tools/*.sh scripts to generate sample config files and run tox as "stack" (tox then tries to install the source tree in it's virtualenv -- if it's owned by root -- again -- failure). I explored a bunch of these options in https://review.opendev.org/c/openstack/devstack/+/837636 and anyone feel free to take over that and keep trying. The other option is to use the new config flag to mark our checkouts as safe. This is obviously simpler, but it seems like a very ugly thing for a nominally generic tool like devstack to do to your global git config. This is done with https://review.opendev.org/c/openstack/devstack/+/837659 and appears to work; but will need backporting for grenade if we want to take this path. When this kicked off I sent in a link to HN thinking that thanks to our very upstream focused CI we were likely some of the first to hit this; it's currently the top post so I think that is accurate that this is having wide impact: https://news.ycombinator.com/item?id=31009675 It is probably worth keeping one eye on upstream for any developments that might change our options. -i
Hey! I actually wonder if the approach with config flag to mark checkouts as safe should be applied more generally, when zuul preps repos for usage, instead of hook in devstack specifically. As it's a more general issue, since zuul repos can't be used as is now for other projects as well (limited to devstack). ср, 13 апр. 2022 г. в 09:14, Ian Wienand <iwienand@redhat.com>:
On Tue, Apr 12, 2022 at 05:05:22PM -0700, Michael Johnson wrote: 65;6602;1c> tldr: All devstack based jobs are going to fail with newer versions of
git - don't bother rechecking
git has released a security fix [1] that is starting to roll out in distributions (Ubuntu focal for example) that will cause pbr to be unable to access the package metadata for packages checked out locally due to the directory ownership used in devstack.
This turns out to be annoyingly complicated.
Since devstack checks out all code as "stack" and then installs globally with "sudo pip install -e ...", pbr will be running in a directory owned by "stack" as root and its git calls will hit this failure.
If we make the code directories owned by root, we now have additional problems. Several places do things in the code repositories -- e.g. setup virtualenvs, run ./tools/*.sh scripts to generate sample config files and run tox as "stack" (tox then tries to install the source tree in it's virtualenv -- if it's owned by root -- again -- failure).
I explored a bunch of these options in
https://review.opendev.org/c/openstack/devstack/+/837636
and anyone feel free to take over that and keep trying.
The other option is to use the new config flag to mark our checkouts as safe. This is obviously simpler, but it seems like a very ugly thing for a nominally generic tool like devstack to do to your global git config. This is done with
https://review.opendev.org/c/openstack/devstack/+/837659
and appears to work; but will need backporting for grenade if we want to take this path.
When this kicked off I sent in a link to HN thinking that thanks to our very upstream focused CI we were likely some of the first to hit this; it's currently the top post so I think that is accurate that this is having wide impact:
https://news.ycombinator.com/item?id=31009675
It is probably worth keeping one eye on upstream for any developments that might change our options.
-i
On 2022-04-13 11:17:06 +0200 (+0200), Dmitriy Rabotyagov wrote:
I actually wonder if the approach with config flag to mark checkouts as safe should be applied more generally, when zuul preps repos for usage, instead of hook in devstack specifically. As it's a more general issue, since zuul repos can't be used as is now for other projects as well (limited to devstack). [...]
I don't follow the logic here. Zuul checkouts are owned by the zuul user which is also the user under which the job payload is executed. This problem only arises if you try to run Git as a different user than zuul. -- Jeremy Stanley
On 2022-04-13 17:11:27 +1000 (+1000), Ian Wienand wrote: [...]
Since devstack checks out all code as "stack" and then installs globally with "sudo pip install -e ...", pbr will be running in a directory owned by "stack" as root and its git calls will hit this failure.
If we make the code directories owned by root, we now have additional problems. Several places do things in the code repositories -- e.g. setup virtualenvs, run ./tools/*.sh scripts to generate sample config files and run tox as "stack" (tox then tries to install the source tree in it's virtualenv -- if it's owned by root -- again -- failure). [...]
Forgive me as caffeine is still finding its way into my veins, but it has occurred to me that the error is occurring because we're calling PBR (and thus Git) while installing the software, when that's not strictly necessary. It happens because we're taking advantage of pip's ability to call out to a build process before installing, but we can always separate building and installing. The former doesn't need root privs, and the latter doesn't need to call PBR/Git. Update the install-from-source routine to build a wheel as stack and then only sudo pip install the resulting wheel. -- Jeremy Stanley
On 2022-04-13 11:59:20 +0000 (+0000), Jeremy Stanley wrote: [...]
Forgive me as caffeine is still finding its way into my veins, but it has occurred to me that the error is occurring because we're calling PBR (and thus Git) while installing the software, when that's not strictly necessary. It happens because we're taking advantage of pip's ability to call out to a build process before installing, but we can always separate building and installing. The former doesn't need root privs, and the latter doesn't need to call PBR/Git.
Update the install-from-source routine to build a wheel as stack and then only sudo pip install the resulting wheel.
I was able to make a successful go of this in https://review.opendev.org/837731 so if there's interest we have evidence it's possible to continue down that path. Unfortunately, it comes at the expense of losing editable mode installation (pip install -e, setup.py develop) as that doesn't use pip's normal package-then-install codepath and instead tightly couples the build and install steps. I've heard from a couple of people so far that editable mode support in DevStack is critical to keep, so it's probably better to resurrect the venv install solution in https://review.opendev.org/558930 since that would allow us to drop the `sudo pip install` nastiness once and for all. -- Jeremy Stanley
On Wed, Apr 13, 2022, at 12:11 AM, Ian Wienand wrote:
On Tue, Apr 12, 2022 at 05:05:22PM -0700, Michael Johnson wrote: 65;6602;1c> tldr: All devstack based jobs are going to fail with newer versions of
git - don't bother rechecking
git has released a security fix [1] that is starting to roll out in distributions (Ubuntu focal for example) that will cause pbr to be unable to access the package metadata for packages checked out locally due to the directory ownership used in devstack.
This turns out to be annoyingly complicated.
Since devstack checks out all code as "stack" and then installs globally with "sudo pip install -e ...", pbr will be running in a directory owned by "stack" as root and its git calls will hit this failure.
If we make the code directories owned by root, we now have additional problems. Several places do things in the code repositories -- e.g. setup virtualenvs, run ./tools/*.sh scripts to generate sample config files and run tox as "stack" (tox then tries to install the source tree in it's virtualenv -- if it's owned by root -- again -- failure).
I explored a bunch of these options in
https://review.opendev.org/c/openstack/devstack/+/837636
and anyone feel free to take over that and keep trying.
The other option is to use the new config flag to mark our checkouts as safe. This is obviously simpler, but it seems like a very ugly thing for a nominally generic tool like devstack to do to your global git config. This is done with
https://review.opendev.org/c/openstack/devstack/+/837659
and appears to work; but will need backporting for grenade if we want to take this path.
This ended up being the quickest option to unblocking things so we backported it all the way through to Victoria then landed the changes from Victoria up to master in that order. This means that devstack testing should work again and you can recheck/approve/push changes once again. However, we noticed that these changes don't quite work on Ubuntu Bionic just on Ubuntu Focal. Dan pushed up https://review.opendev.org/c/openstack/devstack/+/837759 to address the Bionic problem and make unstack clean up after ourselves. Once this lands to master we can backport it using our typical backporting process. Finally fungi has been working on https://review.opendev.org/c/openstack/devstack/+/837731 to separate the package creation step from the package installation step. This allows us to build the python package as the stack user and do the install as root avoiding any git concerns about different ownership of repositories. As the commit message in that change notes this effectively means that we cannot have editable installs anymore. If we decide that is a necessary feature of devstack then I think we should look into resurrecting https://review.opendev.org/c/openstack/devstack/+/558930 to have devstack install into a global virtualenv. Then stack can own the virtualenv, and there is no git concern about file ownership. In the past this change sort of died out as it is quite a large change to how devstack operates and will potentially have significant fallout of its own if we land it and there just didn't seem to be a will to go through that. Maybe this situation has changed our opinion on that. Others should feel free to push updates to that change as I'm not sure I'll have time to dedicate to it again.
When this kicked off I sent in a link to HN thinking that thanks to our very upstream focused CI we were likely some of the first to hit this; it's currently the top post so I think that is accurate that this is having wide impact:
https://news.ycombinator.com/item?id=31009675
It is probably worth keeping one eye on upstream for any developments that might change our options.
-i
On Thu, Apr 14, 2022 at 12:12 AM Clark Boylan <cboylan@sapwetik.org> wrote:
On Wed, Apr 13, 2022, at 12:11 AM, Ian Wienand wrote:
On Tue, Apr 12, 2022 at 05:05:22PM -0700, Michael Johnson wrote: 65;6602;1c> tldr: All devstack based jobs are going to fail with newer versions of
git - don't bother rechecking
git has released a security fix [1] that is starting to roll out in distributions (Ubuntu focal for example) that will cause pbr to be unable to access the package metadata for packages checked out locally due to the directory ownership used in devstack.
This turns out to be annoyingly complicated.
Since devstack checks out all code as "stack" and then installs globally with "sudo pip install -e ...", pbr will be running in a directory owned by "stack" as root and its git calls will hit this failure.
If we make the code directories owned by root, we now have additional problems. Several places do things in the code repositories -- e.g. setup virtualenvs, run ./tools/*.sh scripts to generate sample config files and run tox as "stack" (tox then tries to install the source tree in it's virtualenv -- if it's owned by root -- again -- failure).
I explored a bunch of these options in
https://review.opendev.org/c/openstack/devstack/+/837636
and anyone feel free to take over that and keep trying.
The other option is to use the new config flag to mark our checkouts as safe. This is obviously simpler, but it seems like a very ugly thing for a nominally generic tool like devstack to do to your global git config. This is done with
https://review.opendev.org/c/openstack/devstack/+/837659
and appears to work; but will need backporting for grenade if we want to take this path.
This ended up being the quickest option to unblocking things so we backported it all the way through to Victoria then landed the changes from Victoria up to master in that order. This means that devstack testing should work again and you can recheck/approve/push changes once again.
However, we noticed that these changes don't quite work on Ubuntu Bionic just on Ubuntu Focal. Dan pushed up https://review.opendev.org/c/openstack/devstack/+/837759 to address the Bionic problem and make unstack clean up after ourselves. Once this lands to master we can backport it using our typical backporting process.
Finally fungi has been working on https://review.opendev.org/c/openstack/devstack/+/837731 to separate the package creation step from the package installation step. This allows us to build the python package as the stack user and do the install as root avoiding any git concerns about different ownership of repositories. As the commit message in that change notes this effectively means that we cannot have editable installs anymore.
If we decide that is a necessary feature of devstack then I think we should look into resurrecting https://review.opendev.org/c/openstack/devstack/+/558930 to have devstack install into a global virtualenv. Then stack can own the virtualenv, and there is no git concern about file ownership. In the past this change sort of died out as it is quite a large change to how devstack operates and will potentially have significant fallout of its own if we land it and there just didn't seem to be a will to go through that. Maybe this situation has changed our opinion on that. Others should feel free to push updates to that change as I'm not sure I'll have time to dedicate to it again.
As a data point: maintaining bifrost has become much easier once we did a similar thing and started using a virtualenv. Dmitry
When this kicked off I sent in a link to HN thinking that thanks to our very upstream focused CI we were likely some of the first to hit this; it's currently the top post so I think that is accurate that this is having wide impact:
https://news.ycombinator.com/item?id=31009675
It is probably worth keeping one eye on upstream for any developments that might change our options.
-i
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On Thu, Apr 14, 2022 at 12:12 AM Clark Boylan <cboylan@sapwetik.org> wrote:
On Wed, Apr 13, 2022, at 12:11 AM, Ian Wienand wrote:
On Tue, Apr 12, 2022 at 05:05:22PM -0700, Michael Johnson wrote: 65;6602;1c> tldr: All devstack based jobs are going to fail with newer versions of
git - don't bother rechecking
git has released a security fix [1] that is starting to roll out in distributions (Ubuntu focal for example) that will cause pbr to be unable to access the package metadata for packages checked out locally due to the directory ownership used in devstack.
This turns out to be annoyingly complicated.
Since devstack checks out all code as "stack" and then installs globally with "sudo pip install -e ...", pbr will be running in a directory owned by "stack" as root and its git calls will hit this failure.
If we make the code directories owned by root, we now have additional problems. Several places do things in the code repositories -- e.g. setup virtualenvs, run ./tools/*.sh scripts to generate sample config files and run tox as "stack" (tox then tries to install the source tree in it's virtualenv -- if it's owned by root -- again -- failure).
I explored a bunch of these options in
https://review.opendev.org/c/openstack/devstack/+/837636
and anyone feel free to take over that and keep trying.
The other option is to use the new config flag to mark our checkouts as safe. This is obviously simpler, but it seems like a very ugly thing for a nominally generic tool like devstack to do to your global git config. This is done with
https://review.opendev.org/c/openstack/devstack/+/837659
and appears to work; but will need backporting for grenade if we want to take this path.
This ended up being the quickest option to unblocking things so we backported it all the way through to Victoria then landed the changes from Victoria up to master in that order. This means that devstack testing should work again and you can recheck/approve/push changes once again.
However, we noticed that these changes don't quite work on Ubuntu Bionic just on Ubuntu Focal. Dan pushed up https://review.opendev.org/c/openstack/devstack/+/837759 to address the Bionic problem and make unstack clean up after ourselves. Once this lands to master we can backport it using our typical backporting process.
Finally fungi has been working on https://review.opendev.org/c/openstack/devstack/+/837731 to separate the package creation step from the package installation step. This allows us to build the python package as the stack user and do the install as root avoiding any git concerns about different ownership of repositories. As the commit message in that change notes this effectively means that we cannot have editable installs anymore.
If we decide that is a necessary feature of devstack then I think we should look into resurrecting https://review.opendev.org/c/openstack/devstack/+/558930 to have devstack install into a global virtualenv. Then stack can own the virtualenv, and there is no git concern about file ownership. In the past this change sort of died out as it is quite a large change to how devstack operates and will potentially have significant fallout of its own if we land it and there just didn't seem to be a will to go through that. Maybe this situation has changed our opinion on that. Others should feel free to push updates to that change as I'm not sure I'll have time to dedicate to it again.
As a data point: maintaining bifrost has become much easier once we did a similar thing and started using a virtualenv. apparntly issue with ironic were one of the things that stop this progressing in the past so perhaps your expirnce with virtual env and bifrost could help unblock this if we review
On Thu, 2022-04-14 at 16:27 +0200, Dmitry Tantsur wrote: that effort. usign a virtual evev will isolate use form the distro packages which should allow use to remvoe alot of hte fixups in https://github.com/openstack/devstack/blob/master/tools/fixup_stuff.sh so there are defintly advantages to be had. developers will jsut have to learn to activate the venv on the cli or in there ide but it woudl allow much of the existing workflow to be kept without need to use sudo for python package installs.
Dmitry
When this kicked off I sent in a link to HN thinking that thanks to our very upstream focused CI we were likely some of the first to hit this; it's currently the top post so I think that is accurate that this is having wide impact:
https://news.ycombinator.com/item?id=31009675
It is probably worth keeping one eye on upstream for any developments that might change our options.
-i
On Tue, 2022-04-12 at 17:05 -0700, Michael Johnson wrote:
tldr: All devstack based jobs are going to fail with newer versions of git - don't bother rechecking
git has released a security fix [1] that is starting to roll out in distributions (Ubuntu focal for example) that will cause pbr to be unable to access the package metadata for packages checked out locally due to the directory ownership used in devstack.
We have opened a devstack bug to track this issue: https://bugs.launchpad.net/devstack/+bug/1968798
ok ok i hit this a few months ago but tought this was related to arm i spoke to clark and fungi about it when i was tryign to get devstack running on a vm macbook air this broke the ablity to do "pip install -e" which broke my normal workflow of keeping my workign repos in /opt/repos and the devstack managed ones in /opt/stack i was getting the same pbr issue when i tried to sudo pip install -e /opt/repos/nova that is also when i encounted the issues with unistalling so this has been out in the wild since at least january
Please see the bug for the details and example error.
Michael
P.S. Thanks to clarkb, fungi, and ianw for helping track down the root cause!
[1] https://github.com/git/git/commit/8959555cee7ec045958f9b6dd62e541affb7e7d9
On 2022-04-13 12:06:22 +0100 (+0100), Sean Mooney wrote: [...]
i hit this a few months ago [...] this has been out in the wild since at least january [...]
It's easy to get confused here. You saw the same symptoms, because the error returned by PBR is the same any time there's a problem using Git to query revisions and tags, but as I'm sure you can imagine, there are limitless ways for that to happen. You didn't see this bug though, since it's the result of a security fix for a vulnerability disclosed yesterday. -- Jeremy Stanley
On Wed, 2022-04-13 at 12:05 +0000, Jeremy Stanley wrote:
On 2022-04-13 12:06:22 +0100 (+0100), Sean Mooney wrote: [...]
i hit this a few months ago [...] this has been out in the wild since at least january [...]
It's easy to get confused here. You saw the same symptoms, because the error returned by PBR is the same any time there's a problem using Git to query revisions and tags, but as I'm sure you can imagine, there are limitless ways for that to happen. You didn't see this bug though, since it's the result of a security fix for a vulnerability disclosed yesterday.
ah thanks for context.
participants (7)
-
Clark Boylan
-
Dmitriy Rabotyagov
-
Dmitry Tantsur
-
Ian Wienand
-
Jeremy Stanley
-
Michael Johnson
-
Sean Mooney