Open Stack

I am writing at top now for easy ready.

* Gate status:
- All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6
- Stable/stein and stable/train gates are all good as they are on Bionic with py3.6.
- The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above.
- I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates
the tempest tox env with master u-c needs to be fixed[2].
- Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata.

* https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED
- Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now.  You can re-run such jobs.

 [1] https://review.opendev.org/#/c/705089/
 [2] https://review.opendev.org/#/c/705870/

-gmann

 ---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  ---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  > Hello Everyone, 
 >  > 
 >  > This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement
 >  > of 'EOLing python2 drama' in subject :).
 >  > 
 >  > neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0
 >  > is py3 only and so does u-c on the master has been updated to 2.0.0. 
 >  > 
 >  > All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin 
 >  > is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying 
 >  > to install the latest neutron-lib and failing.
 >  > 
 >  > This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the
 >  > py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop
 >  > from master and kepe testing py2 on stable bracnhes.
 >  > 
 >  > We have two way to fix this:
 >  > 
 >  > 1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c.
 >  > For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and
 >  > use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
 >  > 
 >  > 2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way.
 >  > I am trying this first[2] and testing[3].
 >  > 
 > 
 > I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
 > 
 > Tried option#2:
 > We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the
 > bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro
 > job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3
 > and more hack for example to set the python alias on such distro.  It can be stable common jobs running on Xenial
 > or distro-specific job like centos7 etc where we have < py3.6.
 > 
 > Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our
 > CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases.
 > Testing your cloud with the latest Tempest is the best possible way.
 > 
 > Going with option#1:
 > IMO, this is a workable option with the current situation.  Below is plan to make Tempest and its plugins working
 > for all possible distro/py version.
 > 
 > 1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). 
 > * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5.
 > * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest
 > on >py3.6 env(venv or separate node).
 > * Patch is up - https://review.opendev.org/#/c/704840/
 > 
 > 2.Modify Tempest tox env basepython to py3 
 > * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6
 > like fedora or future distro
 > *Patch is up- https://review.opendev.org/#/c/704688/2 
 > 
 > 3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs.
 > * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs.
 > * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to 
 > that branch can be used. For example Tempest 19.0.0 for rocky[3].
 > * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
 > 
 > 4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case):
 > We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing.  amotoki brought
 > this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky
 > jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which
 > moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using
 > in-tree plugins for their stable branch testing. 
 > We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these
 > stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins
 > in neutron-vpnaas case. This will be easy for future maintenance also.
 > 
 > Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
 >  
 > [1] https://review.opendev.org/#/c/703476/
 > [2] https://review.opendev.org/#/c/703011/
 > [3] https://releases.openstack.org/rocky/#rocky-tempest
 > 
 > -gmann
 > 
 >  > [1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/log/job-output.txt#33040
 >  > [2] https://review.opendev.org/#/c/703011/
 >  > [3] https://review.opendev.org/#/c/703012/
 >  > 
 >  > 
 >  > -gmanne
 >  > 
 >  > 
 >  > 
 >  > 
 >  >
 >