Open Stack

Hi,

> On 9 Feb 2020, at 01:39, Ghanshyam Mann <gmann at ghanshyammann.com> wrote:
> 
> ---- On Thu, 06 Feb 2020 18:23:51 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
>> ---- On Wed, 05 Feb 2020 14:28:28 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
>>> ---- On Tue, 04 Feb 2020 18:42:47 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
>>>> I am writing at top now for easy ready.
>>>> 
>>>> * Gate status:
>>>> - All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6
>>>> - Stable/stein and stable/train gates are all good as they are on Bionic with py3.6.
>>>> - The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above.
>>>> - I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates
>>>> the tempest tox env with master u-c needs to be fixed[2].
>>> 
>>> While testing the Tempest fix, I realized that it will fix the future Tempest release, not the old tags which are used for
>>> stable branch testing.  I gave many thoughts on this but cannot find the best way to solve this. Only option seems
>>> to cap the upper-constraint like it was proposed by chandan[1]. 
>>> 
>>> NOTE: we need to do those cap for all such dependencies of Tempest & its plugins require py>=3.6. We need to maintain
>>> such cap only for Tempest & its plugins till stable/rocky EOL.
>> 
>> Updates:
>> ======
>> Instead of capping requirement, Tempest role run-tempest fix on master branch work fine because of Zuul pickup the job definition
>> and playbooks from master branch.  But there is another issue occurring here. grenade job fails and blocks the fixes on stable branches 
>> to merge because old branch has to be fixed first. So we need to merge the stable/ocata fix first and then stab;e/pike and so on.
> 
> All the fixes are merged now and good to recheck on stable branch jobs. I am working on bug#1862240 now.
> 
> - https://review.opendev.org/#/q/topic:fix-stable-gate+(status:open+OR+status:merged)

Thx. I see that neutron-tempest-plugin’s rocky jobs now works fine: https://review.opendev.org/#/c/706451/2 finally \o/ 

> 
> -gmann
> 
>> 
>> New bug:
>> =======
>> Another bug on py2 jobs even on stable/train etc. - https://bugs.launchpad.net/tempest/+bug/1862240
>> 
>> Tempest tox env moved to py3 fail the py2 jobs on stable branch jobs who are still using deprecated tox env 'all-plugin'. 
>> plugins are installed on py2 in py2 jobs and tempest create tox env with py3 and so does 'all-plugin' which is sitepackage=True
>> try to find plugins on py3 and fail.
>> The solution is to replace the 'all-plugin' tox env to 'all'. I am fixing it for designate. But due to grenade job nature, we need to
>> reverse the fixes here, first, merge the stable/stein and then stable/train. 
>> 
>> Stay tuned for updates and more bugs.  The stable branches gate  is still not green, I will update the status on ML about gate staus.
>> 
>> -gmann
>> 
>>> 
>>> Background on this issue:
>>> --------------------------------
>>> 
>>> Devstack install the Tempest and its plugins in venv using master u-c from tox.ini[2], With Tempest pin on
>>> stable branch, devstack can use the stable branch u-c[3] which is all set and Tempest is installed in venv with
>>> stable branch u-c. But while running the tests, Tempest roles run-tempest recreate the tox env and using master u-c.
>>> 
>>> I am fixing that in run-tempest roles[4] but that cannot be fixed for Tempest old tags so stable branch testing still broken.
>>> -------------------------------------------
>>> 
>>> [1] https://review.opendev.org/#/c/705685/
>>> [2] https://opendev.org/openstack/tempest/src/commit/bc9fe8eca801f54915ff3eafa418e6e18ac2df63/tox.ini#L14
>>> [3] https://review.opendev.org/#/c/705089
>>> [4] https://review.opendev.org/#/c/705870/22/roles/run-tempest/tasks/main.yaml
>>> 
>>> -gmann
>>> 
>>> 
>>>> - Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata.
>>>> 
>>>> * https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED
>>>> - Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now. You can re-run such jobs.
>>>> 
>>>> [1] https://review.opendev.org/#/c/705089/
>>>> [2] https://review.opendev.org/#/c/705870/
>>>> 
>>>> -gmann
>>>> 
>>>> ---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
>>>>> ---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
>>>>>> Hello Everyone, 
>>>>>> 
>>>>>> This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement
>>>>>> of 'EOLing python2 drama' in subject :).
>>>>>> 
>>>>>> neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0
>>>>>> is py3 only and so does u-c on the master has been updated to 2.0.0. 
>>>>>> 
>>>>>> All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin 
>>>>>> is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying 
>>>>>> to install the latest neutron-lib and failing.
>>>>>> 
>>>>>> This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the
>>>>>> py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop
>>>>>> from master and kepe testing py2 on stable bracnhes.
>>>>>> 
>>>>>> We have two way to fix this:
>>>>>> 
>>>>>> 1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c.
>>>>>> For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and
>>>>>> use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
>>>>>> 
>>>>>> 2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way.
>>>>>> I am trying this first[2] and testing[3].
>>>>>> 
>>>>> 
>>>>> I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
>>>>> 
>>>>> Tried option#2:
>>>>> We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the
>>>>> bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro
>>>>> job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3
>>>>> and more hack for example to set the python alias on such distro.  It can be stable common jobs running on Xenial
>>>>> or distro-specific job like centos7 etc where we have < py3.6.
>>>>> 
>>>>> Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our
>>>>> CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases.
>>>>> Testing your cloud with the latest Tempest is the best possible way.
>>>>> 
>>>>> Going with option#1:
>>>>> IMO, this is a workable option with the current situation.  Below is plan to make Tempest and its plugins working
>>>>> for all possible distro/py version.
>>>>> 
>>>>> 1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). 
>>>>> * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5.
>>>>> * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest
>>>>> on >py3.6 env(venv or separate node).
>>>>> * Patch is up - https://review.opendev.org/#/c/704840/
>>>>> 
>>>>> 2.Modify Tempest tox env basepython to py3 
>>>>> * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6
>>>>> like fedora or future distro
>>>>> *Patch is up- https://review.opendev.org/#/c/704688/2 
>>>>> 
>>>>> 3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs.
>>>>> * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs.
>>>>> * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to 
>>>>> that branch can be used. For example Tempest 19.0.0 for rocky[3].
>>>>> * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
>>>>> 
>>>>> 4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case):
>>>>> We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing. amotoki brought
>>>>> this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky
>>>>> jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which
>>>>> moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using
>>>>> in-tree plugins for their stable branch testing. 
>>>>> We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these
>>>>> stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins
>>>>> in neutron-vpnaas case. This will be easy for future maintenance also.
>>>>> 
>>>>> Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
>>>>> 
>>>>> [1] https://review.opendev.org/#/c/703476/
>>>>> [2] https://review.opendev.org/#/c/703011/
>>>>> [3] https://releases.openstack.org/rocky/#rocky-tempest
>>>>> 
>>>>> -gmann
>>>>> 
>>>>>> [1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/log/job-output.txt#33040
>>>>>> [2] https://review.opendev.org/#/c/703011/
>>>>>> [3] https://review.opendev.org/#/c/703012/
>>>>>> 
>>>>>> 
>>>>>> -gmanne

— 
Slawek Kaplonski
Senior software engineer
Red Hat