Open Stack

 ---- On Thu, 06 Feb 2020 18:23:51 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  ---- On Wed, 05 Feb 2020 14:28:28 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  >  ---- On Tue, 04 Feb 2020 18:42:47 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  >  > I am writing at top now for easy ready.
 >  >  > 
 >  >  > * Gate status:
 >  >  > - All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6
 >  >  > - Stable/stein and stable/train gates are all good as they are on Bionic with py3.6.
 >  >  > - The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above.
 >  >  > - I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates
 >  >  > the tempest tox env with master u-c needs to be fixed[2].
 >  > 
 >  > While testing the Tempest fix, I realized that it will fix the future Tempest release, not the old tags which are used for
 >  > stable branch testing.  I gave many thoughts on this but cannot find the best way to solve this. Only option seems
 >  > to cap the upper-constraint like it was proposed by chandan[1]. 
 >  > 
 >  > NOTE: we need to do those cap for all such dependencies of Tempest & its plugins require py>=3.6. We need to maintain
 >  > such cap only for Tempest & its plugins till stable/rocky EOL.
 > 
 > Updates:
 > ======
 > Instead of capping requirement, Tempest role run-tempest fix on master branch work fine because of Zuul pickup the job definition
 > and playbooks from master branch.  But there is another issue occurring here. grenade job fails and blocks the fixes on stable branches 
 > to merge because old branch has to be fixed first. So we need to merge the stable/ocata fix first and then stab;e/pike and so on.

All the fixes are merged now and good to recheck on stable branch jobs. I am working on bug#1862240 now.

- https://review.opendev.org/#/q/topic:fix-stable-gate+(status:open+OR+status:merged)

-gmann

 > 
 > New bug:
 > =======
 > Another bug on py2 jobs even on stable/train etc. - https://bugs.launchpad.net/tempest/+bug/1862240
 > 
 > Tempest tox env moved to py3 fail the py2 jobs on stable branch jobs who are still using deprecated tox env 'all-plugin'. 
 > plugins are installed on py2 in py2 jobs and tempest create tox env with py3 and so does 'all-plugin' which is sitepackage=True
 > try to find plugins on py3 and fail.
 > The solution is to replace the 'all-plugin' tox env to 'all'. I am fixing it for designate. But due to grenade job nature, we need to
 > reverse the fixes here, first, merge the stable/stein and then stable/train. 
 > 
 > Stay tuned for updates and more bugs.  The stable branches gate  is still not green, I will update the status on ML about gate staus.
 > 
 > -gmann
 > 
 >  > 
 >  > Background on this issue:
 >  > --------------------------------
 >  > 
 >  > Devstack install the Tempest and its plugins in venv using master u-c from tox.ini[2], With Tempest pin on
 >  > stable branch, devstack can use the stable branch u-c[3] which is all set and Tempest is installed in venv with
 >  > stable branch u-c. But while running the tests, Tempest roles run-tempest recreate the tox env and using master u-c.
 >  > 
 >  > I am fixing that in run-tempest roles[4] but that cannot be fixed for Tempest old tags so stable branch testing still broken.
 >  > -------------------------------------------
 >  > 
 >  > [1] https://review.opendev.org/#/c/705685/
 >  > [2] https://opendev.org/openstack/tempest/src/commit/bc9fe8eca801f54915ff3eafa418e6e18ac2df63/tox.ini#L14
 >  > [3] https://review.opendev.org/#/c/705089
 >  > [4] https://review.opendev.org/#/c/705870/22/roles/run-tempest/tasks/main.yaml
 >  > 
 >  > -gmann
 >  > 
 >  > 
 >  >  > - Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata.
 >  >  > 
 >  >  > * https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED
 >  >  > - Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now.  You can re-run such jobs.
 >  >  > 
 >  >  >  [1] https://review.opendev.org/#/c/705089/
 >  >  >  [2] https://review.opendev.org/#/c/705870/
 >  >  > 
 >  >  > -gmann
 >  >  > 
 >  >  >  ---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  >  >  >  ---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  >  >  >  > Hello Everyone, 
 >  >  >  >  > 
 >  >  >  >  > This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement
 >  >  >  >  > of 'EOLing python2 drama' in subject :).
 >  >  >  >  > 
 >  >  >  >  > neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0
 >  >  >  >  > is py3 only and so does u-c on the master has been updated to 2.0.0. 
 >  >  >  >  > 
 >  >  >  >  > All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin 
 >  >  >  >  > is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying 
 >  >  >  >  > to install the latest neutron-lib and failing.
 >  >  >  >  > 
 >  >  >  >  > This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the
 >  >  >  >  > py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop
 >  >  >  >  > from master and kepe testing py2 on stable bracnhes.
 >  >  >  >  > 
 >  >  >  >  > We have two way to fix this:
 >  >  >  >  > 
 >  >  >  >  > 1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c.
 >  >  >  >  > For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and
 >  >  >  >  > use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
 >  >  >  >  > 
 >  >  >  >  > 2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way.
 >  >  >  >  > I am trying this first[2] and testing[3].
 >  >  >  >  > 
 >  >  >  > 
 >  >  >  > I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
 >  >  >  > 
 >  >  >  > Tried option#2:
 >  >  >  > We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the
 >  >  >  > bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro
 >  >  >  > job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3
 >  >  >  > and more hack for example to set the python alias on such distro.  It can be stable common jobs running on Xenial
 >  >  >  > or distro-specific job like centos7 etc where we have < py3.6.
 >  >  >  > 
 >  >  >  > Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our
 >  >  >  > CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases.
 >  >  >  > Testing your cloud with the latest Tempest is the best possible way.
 >  >  >  > 
 >  >  >  > Going with option#1:
 >  >  >  > IMO, this is a workable option with the current situation.  Below is plan to make Tempest and its plugins working
 >  >  >  > for all possible distro/py version.
 >  >  >  > 
 >  >  >  > 1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). 
 >  >  >  > * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5.
 >  >  >  > * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest
 >  >  >  > on >py3.6 env(venv or separate node).
 >  >  >  > * Patch is up - https://review.opendev.org/#/c/704840/
 >  >  >  > 
 >  >  >  > 2.Modify Tempest tox env basepython to py3 
 >  >  >  > * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6
 >  >  >  > like fedora or future distro
 >  >  >  > *Patch is up- https://review.opendev.org/#/c/704688/2 
 >  >  >  > 
 >  >  >  > 3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs.
 >  >  >  > * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs.
 >  >  >  > * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to 
 >  >  >  > that branch can be used. For example Tempest 19.0.0 for rocky[3].
 >  >  >  > * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
 >  >  >  > 
 >  >  >  > 4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case):
 >  >  >  > We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing.  amotoki brought
 >  >  >  > this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky
 >  >  >  > jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which
 >  >  >  > moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using
 >  >  >  > in-tree plugins for their stable branch testing. 
 >  >  >  > We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these
 >  >  >  > stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins
 >  >  >  > in neutron-vpnaas case. This will be easy for future maintenance also.
 >  >  >  > 
 >  >  >  > Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
 >  >  >  >  
 >  >  >  > [1] https://review.opendev.org/#/c/703476/
 >  >  >  > [2] https://review.opendev.org/#/c/703011/
 >  >  >  > [3] https://releases.openstack.org/rocky/#rocky-tempest
 >  >  >  > 
 >  >  >  > -gmann
 >  >  >  > 
 >  >  >  >  > [1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/log/job-output.txt#33040
 >  >  >  >  > [2] https://review.opendev.org/#/c/703011/
 >  >  >  >  > [3] https://review.opendev.org/#/c/703012/
 >  >  >  >  > 
 >  >  >  >  > 
 >  >  >  >  > -gmanne
 >  >  >  >  > 
 >  >  >  >  > 
 >  >  >  >  > 
 >  >  >  >  > 
 >  >  >  >  >
 >  >  >  >
 >  >  > 
 >  >  > 
 >  >  >
 >  > 
 >  > 
 >  >
 > 
 > 
 >