[qa][stable][tempest-plugins]: Tempest & plugins py2 jobs failure for stable branches (1860033: the EOLing python2 drama)
Ghanshyam Mann
gmann at ghanshyammann.com
Sun Feb 9 00:39:42 UTC 2020
---- On Thu, 06 Feb 2020 18:23:51 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
> ---- On Wed, 05 Feb 2020 14:28:28 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
> > ---- On Tue, 04 Feb 2020 18:42:47 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
> > > I am writing at top now for easy ready.
> > >
> > > * Gate status:
> > > - All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6
> > > - Stable/stein and stable/train gates are all good as they are on Bionic with py3.6.
> > > - The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above.
> > > - I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates
> > > the tempest tox env with master u-c needs to be fixed[2].
> >
> > While testing the Tempest fix, I realized that it will fix the future Tempest release, not the old tags which are used for
> > stable branch testing. I gave many thoughts on this but cannot find the best way to solve this. Only option seems
> > to cap the upper-constraint like it was proposed by chandan[1].
> >
> > NOTE: we need to do those cap for all such dependencies of Tempest & its plugins require py>=3.6. We need to maintain
> > such cap only for Tempest & its plugins till stable/rocky EOL.
>
> Updates:
> ======
> Instead of capping requirement, Tempest role run-tempest fix on master branch work fine because of Zuul pickup the job definition
> and playbooks from master branch. But there is another issue occurring here. grenade job fails and blocks the fixes on stable branches
> to merge because old branch has to be fixed first. So we need to merge the stable/ocata fix first and then stab;e/pike and so on.
All the fixes are merged now and good to recheck on stable branch jobs. I am working on bug#1862240 now.
- https://review.opendev.org/#/q/topic:fix-stable-gate+(status:open+OR+status:merged)
-gmann
>
> New bug:
> =======
> Another bug on py2 jobs even on stable/train etc. - https://bugs.launchpad.net/tempest/+bug/1862240
>
> Tempest tox env moved to py3 fail the py2 jobs on stable branch jobs who are still using deprecated tox env 'all-plugin'.
> plugins are installed on py2 in py2 jobs and tempest create tox env with py3 and so does 'all-plugin' which is sitepackage=True
> try to find plugins on py3 and fail.
> The solution is to replace the 'all-plugin' tox env to 'all'. I am fixing it for designate. But due to grenade job nature, we need to
> reverse the fixes here, first, merge the stable/stein and then stable/train.
>
> Stay tuned for updates and more bugs. The stable branches gate is still not green, I will update the status on ML about gate staus.
>
> -gmann
>
> >
> > Background on this issue:
> > --------------------------------
> >
> > Devstack install the Tempest and its plugins in venv using master u-c from tox.ini[2], With Tempest pin on
> > stable branch, devstack can use the stable branch u-c[3] which is all set and Tempest is installed in venv with
> > stable branch u-c. But while running the tests, Tempest roles run-tempest recreate the tox env and using master u-c.
> >
> > I am fixing that in run-tempest roles[4] but that cannot be fixed for Tempest old tags so stable branch testing still broken.
> > -------------------------------------------
> >
> > [1] https://review.opendev.org/#/c/705685/
> > [2] https://opendev.org/openstack/tempest/src/commit/bc9fe8eca801f54915ff3eafa418e6e18ac2df63/tox.ini#L14
> > [3] https://review.opendev.org/#/c/705089
> > [4] https://review.opendev.org/#/c/705870/22/roles/run-tempest/tasks/main.yaml
> >
> > -gmann
> >
> >
> > > - Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata.
> > >
> > > * https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED
> > > - Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now. You can re-run such jobs.
> > >
> > > [1] https://review.opendev.org/#/c/705089/
> > > [2] https://review.opendev.org/#/c/705870/
> > >
> > > -gmann
> > >
> > > ---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
> > > > ---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
> > > > > Hello Everyone,
> > > > >
> > > > > This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement
> > > > > of 'EOLing python2 drama' in subject :).
> > > > >
> > > > > neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0
> > > > > is py3 only and so does u-c on the master has been updated to 2.0.0.
> > > > >
> > > > > All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin
> > > > > is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying
> > > > > to install the latest neutron-lib and failing.
> > > > >
> > > > > This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the
> > > > > py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop
> > > > > from master and kepe testing py2 on stable bracnhes.
> > > > >
> > > > > We have two way to fix this:
> > > > >
> > > > > 1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c.
> > > > > For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and
> > > > > use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
> > > > >
> > > > > 2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way.
> > > > > I am trying this first[2] and testing[3].
> > > > >
> > > >
> > > > I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
> > > >
> > > > Tried option#2:
> > > > We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the
> > > > bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro
> > > > job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3
> > > > and more hack for example to set the python alias on such distro. It can be stable common jobs running on Xenial
> > > > or distro-specific job like centos7 etc where we have < py3.6.
> > > >
> > > > Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our
> > > > CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases.
> > > > Testing your cloud with the latest Tempest is the best possible way.
> > > >
> > > > Going with option#1:
> > > > IMO, this is a workable option with the current situation. Below is plan to make Tempest and its plugins working
> > > > for all possible distro/py version.
> > > >
> > > > 1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports).
> > > > * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5.
> > > > * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest
> > > > on >py3.6 env(venv or separate node).
> > > > * Patch is up - https://review.opendev.org/#/c/704840/
> > > >
> > > > 2.Modify Tempest tox env basepython to py3
> > > > * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6
> > > > like fedora or future distro
> > > > *Patch is up- https://review.opendev.org/#/c/704688/2
> > > >
> > > > 3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs.
> > > > * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs.
> > > > * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to
> > > > that branch can be used. For example Tempest 19.0.0 for rocky[3].
> > > > * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
> > > >
> > > > 4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case):
> > > > We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing. amotoki brought
> > > > this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky
> > > > jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which
> > > > moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using
> > > > in-tree plugins for their stable branch testing.
> > > > We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these
> > > > stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins
> > > > in neutron-vpnaas case. This will be easy for future maintenance also.
> > > >
> > > > Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
> > > >
> > > > [1] https://review.opendev.org/#/c/703476/
> > > > [2] https://review.opendev.org/#/c/703011/
> > > > [3] https://releases.openstack.org/rocky/#rocky-tempest
> > > >
> > > > -gmann
> > > >
> > > > > [1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/log/job-output.txt#33040
> > > > > [2] https://review.opendev.org/#/c/703011/
> > > > > [3] https://review.opendev.org/#/c/703012/
> > > > >
> > > > >
> > > > > -gmanne
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
More information about the openstack-discuss
mailing list