On Fri, Sep 24, 2021, at 8:07 AM, Neil Jerram wrote:
> On Fri, Sep 24, 2021 at 3:38 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
>> On 2021-09-24 12:33:14 +0100 (+0100), Neil Jerram wrote:
>> > On Wed, Sep 22, 2021 at 5:07 PM Balazs Gibizer <balazs.gibizer@est.tech>
>> > wrote:
>> >
>> > >
>> > > I agree not to pin this on master. We need to support the latest and
>> > > greatest on master. But on stable I don't see the problem saying we
>> > > only support the tooling version that was the latest when the major
>> > > release was created.
>> > >
>> >
>> > +1, this seems like a straightforward fix to me.
>> > https://review.opendev.org/c/openstack/requirements/+/810859 proposes what
>> > I think is the right change.  Is there any downside?
>>
>> Did that change actually fix the observed problem with tox jobs?
>> Last I recall, the requirements/constraints on SetupTools were only
>> effectively applied by DevStack because that includes a separate
>> step for installing or downgrading it.
>
> I'm afraid I haven't yet tested it, or worked out how I would test it
> prior to merge - any hints for that?
>
> My reasoning is based on reading the stack.sh stdout and underlying
> devstack code (for stable/ussuri):
>
> + tools/install_pip.sh:main:152            :   pip_install_gr setuptools
> + inc/python:pip_install_gr:76             :   local name=setuptools
> + inc/python:pip_install_gr:77             :   local clean_name
> ++ inc/python:pip_install_gr:78             :   
> get_from_global_requirements setuptools
> ++ inc/python:get_from_global_requirements:238 :   local
> package=setuptools
> ++ inc/python:get_from_global_requirements:239 :   local required_pkg
> +++ inc/python:get_from_global_requirements:240 :   grep -i -h
> '^setuptools' /opt/stack/requirements/global-requirements.txt
> +++ inc/python:get_from_global_requirements:240 :   cut -d# -f1
> ++ inc/python:get_from_global_requirements:240 :   
> required_pkg='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\''
> 
> setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\''
>  '
> ++ inc/python:get_from_global_requirements:241 :   [[
> setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='3.5'
> 
> setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7'
>   == '' ]]
> ++ inc/python:get_from_global_requirements:244 :   echo
> 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\'''
> 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\'''
> + inc/python:pip_install_gr:78             :   
> clean_name='setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\''
> setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\'''
> + inc/python:pip_install_gr:79             :   pip_install
> 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0;python_version>='\''3.5'\'''
> 'setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='\''2.7'\'''
> Using python 3.6 to install
> setuptools!=24.0.0,!=34.0.0,!=34.0.1,!=34.0.2,!=34.0.3,!=34.1.0,!=34.1.1,!=34.2.0,!=34.3.0,!=34.3.1,!=34.3.2,!=36.2.0,<45.0.0;python_version<='2.7'
> because python3_enabled=True
>
> It clearly seems to be installing a version of setuptools as
> constrained by global-requirements.txt, and it appears that that
> installed version is the relevant factor, because runs with 58.* fail
> and older runs with 57.* succeeded.

Yup, as Fungi mentions this will work for devstack because devstack manages the setuptools installation. It won't work for tox because requirements and constraints are applied after tox+virtualenv have installed setuptools.

That said there may not be a one size fits all fix here and we'll have to address it several different ways. I think Fungi's concern is that we think a single fix has corrected the issue when that isn't the case.