Open Stack

 ---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 > Hello Everyone, 
 > 
 > This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement
 > of 'EOLing python2 drama' in subject :).
 > 
 > neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0
 > is py3 only and so does u-c on the master has been updated to 2.0.0. 
 > 
 > All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin 
 > is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying 
 > to install the latest neutron-lib and failing.
 > 
 > This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the
 > py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop
 > from master and kepe testing py2 on stable bracnhes.
 > 
 > We have two way to fix this:
 > 
 > 1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c.
 > For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and
 > use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
 > 
 > 2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way.
 > I am trying this first[2] and testing[3].
 > 

I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.

Tried option#2:
We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the
bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro
job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3
and more hack for example to set the python alias on such distro.  It can be stable common jobs running on Xenial
or distro-specific job like centos7 etc where we have < py3.6.

Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our
CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases.
Testing your cloud with the latest Tempest is the best possible way.

Going with option#1:
IMO, this is a workable option with the current situation.  Below is plan to make Tempest and its plugins working
for all possible distro/py version.

1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). 
* Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5.
* 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest
on >py3.6 env(venv or separate node).
* Patch is up - https://review.opendev.org/#/c/704840/

2.Modify Tempest tox env basepython to py3 
* Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6
like fedora or future distro
*Patch is up- https://review.opendev.org/#/c/704688/2 

3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs.
* Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs.
* If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to 
that branch can be used. For example Tempest 19.0.0 for rocky[3].
* We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/

4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case):
We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing.  amotoki brought
this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky
jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which
moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using
in-tree plugins for their stable branch testing. 
We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these
stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins
in neutron-vpnaas case. This will be easy for future maintenance also.

Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
 
[1] https://review.opendev.org/#/c/703476/
[2] https://review.opendev.org/#/c/703011/
[3] https://releases.openstack.org/rocky/#rocky-tempest

-gmann

 > [1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/log/job-output.txt#33040
 > [2] https://review.opendev.org/#/c/703011/
 > [3] https://review.opendev.org/#/c/703012/
 > 
 > 
 > -gmanne
 > 
 > 
 > 
 > 
 >