Open Stack

 ---- On Wed, 05 Feb 2020 14:28:28 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  ---- On Tue, 04 Feb 2020 18:42:47 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  > I am writing at top now for easy ready.
 >  > 
 >  > * Gate status:
 >  > - All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6
 >  > - Stable/stein and stable/train gates are all good as they are on Bionic with py3.6.
 >  > - The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above.
 >  > - I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates
 >  > the tempest tox env with master u-c needs to be fixed[2].
 > 
 > While testing the Tempest fix, I realized that it will fix the future Tempest release, not the old tags which are used for
 > stable branch testing.  I gave many thoughts on this but cannot find the best way to solve this. Only option seems
 > to cap the upper-constraint like it was proposed by chandan[1]. 
 > 
 > NOTE: we need to do those cap for all such dependencies of Tempest & its plugins require py>=3.6. We need to maintain
 > such cap only for Tempest & its plugins till stable/rocky EOL.

Updates:
======
Instead of capping requirement, Tempest role run-tempest fix on master branch work fine because of Zuul pickup the job definition
and playbooks from master branch.  But there is another issue occurring here. grenade job fails and blocks the fixes on stable branches 
to merge because old branch has to be fixed first. So we need to merge the stable/ocata fix first and then stab;e/pike and so on.

New bug:
=======
Another bug on py2 jobs even on stable/train etc. - https://bugs.launchpad.net/tempest/+bug/1862240

Tempest tox env moved to py3 fail the py2 jobs on stable branch jobs who are still using deprecated tox env 'all-plugin'. 
plugins are installed on py2 in py2 jobs and tempest create tox env with py3 and so does 'all-plugin' which is sitepackage=True
try to find plugins on py3 and fail.
The solution is to replace the 'all-plugin' tox env to 'all'. I am fixing it for designate. But due to grenade job nature, we need to
reverse the fixes here, first, merge the stable/stein and then stable/train. 

Stay tuned for updates and more bugs.  The stable branches gate  is still not green, I will update the status on ML about gate staus.

-gmann

 > 
 > Background on this issue:
 > --------------------------------
 > 
 > Devstack install the Tempest and its plugins in venv using master u-c from tox.ini[2], With Tempest pin on
 > stable branch, devstack can use the stable branch u-c[3] which is all set and Tempest is installed in venv with
 > stable branch u-c. But while running the tests, Tempest roles run-tempest recreate the tox env and using master u-c.
 > 
 > I am fixing that in run-tempest roles[4] but that cannot be fixed for Tempest old tags so stable branch testing still broken.
 > -------------------------------------------
 > 
 > [1] https://review.opendev.org/#/c/705685/
 > [2] https://opendev.org/openstack/tempest/src/commit/bc9fe8eca801f54915ff3eafa418e6e18ac2df63/tox.ini#L14
 > [3] https://review.opendev.org/#/c/705089
 > [4] https://review.opendev.org/#/c/705870/22/roles/run-tempest/tasks/main.yaml
 > 
 > -gmann
 > 
 > 
 >  > - Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata.
 >  > 
 >  > * https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED
 >  > - Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now.  You can re-run such jobs.
 >  > 
 >  >  [1] https://review.opendev.org/#/c/705089/
 >  >  [2] https://review.opendev.org/#/c/705870/
 >  > 
 >  > -gmann
 >  > 
 >  >  ---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  >  >  ---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  >  >  > Hello Everyone, 
 >  >  >  > 
 >  >  >  > This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement
 >  >  >  > of 'EOLing python2 drama' in subject :).
 >  >  >  > 
 >  >  >  > neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0
 >  >  >  > is py3 only and so does u-c on the master has been updated to 2.0.0. 
 >  >  >  > 
 >  >  >  > All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin 
 >  >  >  > is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying 
 >  >  >  > to install the latest neutron-lib and failing.
 >  >  >  > 
 >  >  >  > This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the
 >  >  >  > py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop
 >  >  >  > from master and kepe testing py2 on stable bracnhes.
 >  >  >  > 
 >  >  >  > We have two way to fix this:
 >  >  >  > 
 >  >  >  > 1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c.
 >  >  >  > For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and
 >  >  >  > use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
 >  >  >  > 
 >  >  >  > 2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way.
 >  >  >  > I am trying this first[2] and testing[3].
 >  >  >  > 
 >  >  > 
 >  >  > I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
 >  >  > 
 >  >  > Tried option#2:
 >  >  > We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the
 >  >  > bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro
 >  >  > job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3
 >  >  > and more hack for example to set the python alias on such distro.  It can be stable common jobs running on Xenial
 >  >  > or distro-specific job like centos7 etc where we have < py3.6.
 >  >  > 
 >  >  > Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our
 >  >  > CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases.
 >  >  > Testing your cloud with the latest Tempest is the best possible way.
 >  >  > 
 >  >  > Going with option#1:
 >  >  > IMO, this is a workable option with the current situation.  Below is plan to make Tempest and its plugins working
 >  >  > for all possible distro/py version.
 >  >  > 
 >  >  > 1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). 
 >  >  > * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5.
 >  >  > * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest
 >  >  > on >py3.6 env(venv or separate node).
 >  >  > * Patch is up - https://review.opendev.org/#/c/704840/
 >  >  > 
 >  >  > 2.Modify Tempest tox env basepython to py3 
 >  >  > * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6
 >  >  > like fedora or future distro
 >  >  > *Patch is up- https://review.opendev.org/#/c/704688/2 
 >  >  > 
 >  >  > 3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs.
 >  >  > * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs.
 >  >  > * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to 
 >  >  > that branch can be used. For example Tempest 19.0.0 for rocky[3].
 >  >  > * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
 >  >  > 
 >  >  > 4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case):
 >  >  > We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing.  amotoki brought
 >  >  > this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky
 >  >  > jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which
 >  >  > moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using
 >  >  > in-tree plugins for their stable branch testing. 
 >  >  > We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these
 >  >  > stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins
 >  >  > in neutron-vpnaas case. This will be easy for future maintenance also.
 >  >  > 
 >  >  > Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
 >  >  >  
 >  >  > [1] https://review.opendev.org/#/c/703476/
 >  >  > [2] https://review.opendev.org/#/c/703011/
 >  >  > [3] https://releases.openstack.org/rocky/#rocky-tempest
 >  >  > 
 >  >  > -gmann
 >  >  > 
 >  >  >  > [1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/log/job-output.txt#33040
 >  >  >  > [2] https://review.opendev.org/#/c/703011/
 >  >  >  > [3] https://review.opendev.org/#/c/703012/
 >  >  >  > 
 >  >  >  > 
 >  >  >  > -gmanne
 >  >  >  > 
 >  >  >  > 
 >  >  >  > 
 >  >  >  > 
 >  >  >  >
 >  >  >
 >  > 
 >  > 
 >  >
 > 
 > 
 >