[qa][stable][tempest-plugins]: Tempest & plugins py2 jobs failure for stable branches (1860033: the EOLing python2 drama)
Hello Everyone, This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement of 'EOLing python2 drama' in subject :). neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0 is py3 only and so does u-c on the master has been updated to 2.0.0. All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying to install the latest neutron-lib and failing. This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop from master and kepe testing py2 on stable bracnhes. We have two way to fix this: 1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c. For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing. 2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. I am trying this first[2] and testing[3]. [1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/... [2] https://review.opendev.org/#/c/703011/ [3] https://review.opendev.org/#/c/703012/ -gmanne
On 2020-01-16 22:02:05 -0600 (-0600), Ghanshyam Mann wrote: [...]
Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. [...]
This makes more sense anyway. Tempest and its plug-ins are already segregated from the system with a virtualenv due to conflicts with stable branch requirements, so hopefully switching that virtualenv to Python 3.x for all jobs is trivial (but I won't be surprised to learn there are subtle challenges hidden just beneath the surface). -- Jeremy Stanley
+1 for py3 in tempest venv. Makes most sense. Though the test is failing now: 2020-01-17 04:30:06.975801 | controller | ERROR: Could not find a version that satisfies the requirement neutron-lib===2.0.0 (from -c u-c-m.txt (line 79)) (from versions: 0.0.1, 0.0.2, 0.0.3, 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.9.1, 1.9.2, 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.21.0, 1.22.0, 1.23.0, 1.24.0, 1.25.0, 1.26.0, 1.27.0, 1.28.0, 1.29.0, 1.29.1, 1.30.0, 1.31.0) 2020-01-17 04:30:06.993738 | controller | ERROR: No matching distribution found for neutron-lib===2.0.0 (from -c u-c-m.txt (line 79)) and the reason is: pypi: data-requires-python=">=3.6" 3.5 < 3.6 Need some newer python in there. -yoctozepto pt., 17 sty 2020 o 05:15 Jeremy Stanley <fungi@yuggoth.org> napisał(a):
On 2020-01-16 22:02:05 -0600 (-0600), Ghanshyam Mann wrote: [...]
Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. [...]
This makes more sense anyway. Tempest and its plug-ins are already segregated from the system with a virtualenv due to conflicts with stable branch requirements, so hopefully switching that virtualenv to Python 3.x for all jobs is trivial (but I won't be surprised to learn there are subtle challenges hidden just beneath the surface). -- Jeremy Stanley
On 2020-01-17 08:49:32 +0100 (+0100), Radosław Piliszek wrote: [...]
ERROR: No matching distribution found for neutron-lib===2.0.0 (from -c u-c-m.txt (line 79))
and the reason is: pypi: data-requires-python=">=3.6"
3.5 < 3.6
Need some newer python in there. [...]
Or older neutron-lib? -- Jeremy Stanley
On Fri, 17 Jan 2020 at 05:11, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-01-16 22:02:05 -0600 (-0600), Ghanshyam Mann wrote: [...]
Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. [...]
This makes more sense anyway. Tempest and its plug-ins are already segregated from the system with a virtualenv due to conflicts with stable branch requirements, so hopefully switching that virtualenv to Python 3.x for all jobs is trivial (but I won't be surprised to learn there are subtle challenges hidden just beneath the surface).
That sounds good for supported releases. Once we have them back in working order, I wonder how it will turn out for queens. In neutron, there is a recent failure [1] as this EM branch now uses a pinned version of the plugin. The fix there is most likely to also pin tempest - to queens-em [2] but then will also require some fix for the EOLing python2 drama. As tempest is branchless, it looks like if we want to keep neutron-tempest-plugin tests for queens we will rather need solution 1 for this branch? (but let's focus first on getting the supported branches back in working order) [1] https://bugs.launchpad.net/neutron/+bug/1859988 [2] https://review.opendev.org/702868 -- Bernard Cafarelli
Hi,
On 17 Jan 2020, at 11:14, Bernard Cafarelli <bcafarel@redhat.com> wrote:
On Fri, 17 Jan 2020 at 05:11, Jeremy Stanley <fungi@yuggoth.org> wrote: On 2020-01-16 22:02:05 -0600 (-0600), Ghanshyam Mann wrote: [...]
Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. [...]
This makes more sense anyway. Tempest and its plug-ins are already segregated from the system with a virtualenv due to conflicts with stable branch requirements, so hopefully switching that virtualenv to Python 3.x for all jobs is trivial (but I won't be surprised to learn there are subtle challenges hidden just beneath the surface).
That sounds good for supported releases. Once we have them back in working order, I wonder how it will turn out for queens. In neutron, there is a recent failure [1] as this EM branch now uses a pinned version of the plugin. The fix there is most likely to also pin tempest - to queens-em [2] but then will also require some fix for the EOLing python2 drama.
But if we will use for queens branch tempest pinned to queens-em tag, we shouldn’t have any such problems there as all requirements will be also used from queens branch, or am I missing something here?
As tempest is branchless, it looks like if we want to keep neutron-tempest-plugin tests for queens we will rather need solution 1 for this branch? (but let's focus first on getting the supported branches back in working order)
[1] https://bugs.launchpad.net/neutron/+bug/1859988 [2] https://review.opendev.org/702868
-- Bernard Cafarelli
— Slawek Kaplonski Senior software engineer Red Hat
On Fri, 17 Jan 2020 at 12:01, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
On 17 Jan 2020, at 11:14, Bernard Cafarelli <bcafarel@redhat.com> wrote:
On Fri, 17 Jan 2020 at 05:11, Jeremy Stanley <fungi@yuggoth.org> wrote: On 2020-01-16 22:02:05 -0600 (-0600), Ghanshyam Mann wrote: [...]
Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. [...]
This makes more sense anyway. Tempest and its plug-ins are already segregated from the system with a virtualenv due to conflicts with stable branch requirements, so hopefully switching that virtualenv to Python 3.x for all jobs is trivial (but I won't be surprised to learn there are subtle challenges hidden just beneath the surface).
That sounds good for supported releases. Once we have them back in working order, I wonder how it will turn out for queens. In neutron, there is a recent failure [1] as this EM branch now uses a pinned version of the plugin. The fix there is most likely to also pin tempest - to queens-em [2] but then will also require some fix for the EOLing python2 drama.
But if we will use for queens branch tempest pinned to queens-em tag, we shouldn’t have any such problems there as all requirements will be also used from queens branch, or am I missing something here?
Sadly not, from what I read in attempt [1] to limit neutron-lib to "old" version. And I see the same error in a test run with pinned tempest [2]: 2020-01-16 14:44:18.741517 | controller | 2020-01-16 14:44:18.741 | Collecting neutron-lib===2.0.0 (from -c u-c-m.txt (line 79)) 2020-01-16 14:44:19.023699 | controller | 2020-01-16 14:44:19.023 | Could not find a version that satisfies the requirement neutron-lib===2.0.0 (from -c u-c-m.txt (line 79)) (from versions: 0.0.1, 0.0.2, 0.0.3, 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.9.1, 1.9.2, 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.21.0, 1.22.0, 1.23.0, 1.24.0, 1.25.0, 1.26.0, 1.27.0, 1.28.0, 1.29.0, 1.29.1, 1.30.0, 1.31.0) 2020-01-16 14:44:19.042505 | controller | 2020-01-16 14:44:19.042 | No matching distribution found for neutron-lib===2.0.0 (from -c u-c-m.txt (line 79)) [1] https://review.opendev.org/702986/ <https://review.opendev.org/#/c/702986/> [2] https://review.opendev.org/#/c/701900/ https://zuul.opendev.org/t/openstack/build/ee8021c1470a4fb88f55d64cc16ed15e
As tempest is branchless, it looks like if we want to keep
neutron-tempest-plugin tests for queens we will rather need solution 1 for this branch? (but let's focus first on getting the supported branches back in working order)
[1] https://bugs.launchpad.net/neutron/+bug/1859988 [2] https://review.opendev.org/702868
-- Bernard Cafarelli
---- On Fri, 17 Jan 2020 05:51:28 -0600 Bernard Cafarelli <bcafarel@redhat.com> wrote ----
On Fri, 17 Jan 2020 at 12:01, Slawek Kaplonski <skaplons@redhat.com> wrote: Hi,
On 17 Jan 2020, at 11:14, Bernard Cafarelli <bcafarel@redhat.com> wrote:
On Fri, 17 Jan 2020 at 05:11, Jeremy Stanley <fungi@yuggoth.org> wrote: On 2020-01-16 22:02:05 -0600 (-0600), Ghanshyam Mann wrote: [...]
Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. [...]
This makes more sense anyway. Tempest and its plug-ins are already segregated from the system with a virtualenv due to conflicts with stable branch requirements, so hopefully switching that virtualenv to Python 3.x for all jobs is trivial (but I won't be surprised to learn there are subtle challenges hidden just beneath the surface).
That sounds good for supported releases. Once we have them back in working order, I wonder how it will turn out for queens. In neutron, there is a recent failure [1] as this EM branch now uses a pinned version of the plugin. The fix there is most likely to also pin tempest - to queens-em [2] but then will also require some fix for the EOLing python2 drama.
But if we will use for queens branch tempest pinned to queens-em tag, we shouldn’t have any such problems there as all requirements will be also used from queens branch, or am I missing something here? Sadly not, from what I read in attempt [1] to limit neutron-lib to "old" version. And I see the same error in a test run with pinned tempest [2]:2020-01-16 14:44:18.741517 | controller | 2020-01-16 14:44:18.741 | Collecting neutron-lib===2.0.0 (from -c u-c-m.txt (line 79)) 2020-01-16 14:44:19.023699 | controller | 2020-01-16 14:44:19.023 | Could not find a version that satisfies the requirement neutron-lib===2.0.0 (from -c u-c-m.txt (line 79)) (from versions: 0.0.1, 0.0.2, 0.0.3, 0.1.0, 0.2.0, 0.3.0, 0.4.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.9.1, 1.9.2, 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.21.0, 1.22.0, 1.23.0, 1.24.0, 1.25.0, 1.26.0, 1.27.0, 1.28.0, 1.29.0, 1.29.1, 1.30.0, 1.31.0) 2020-01-16 14:44:19.042505 | controller | 2020-01-16 14:44:19.042 | No matching distribution found for neutron-lib===2.0.0 (from -c u-c-m.txt (line 79))
Yes, Temepst venv uses the upper constraint from master always. We need to cap the u-c also accordingly. I did not do for Ocata/Pike case which I will fix as they can start failing any time. For Tempest, it is straight forward where u-c has to be used of the branch corresponding to pin Tempest. But for plugins, it is complex. All tempest plugins are being installed one by one un single logic in devstack so using different constraint for different plugins might not be possible (there should not be much cases like that where job tests more than one plugins tests but there are few). Best possible solution I can think of is to cap all the Tempest plugins together with Tempest and use corresponding stable branch u-c. Or we modify devstack logic with if-else condition for plugins require cap and rest else will be master. Any other thought? -gmann
[1] https://review.opendev.org/702986/[2] https://review.opendev.org/#/c/701900/ https://zuul.opendev.org/t/openstack/build/ee8021c1470a4fb88f55d64cc16ed15e
As tempest is branchless, it looks like if we want to keep neutron-tempest-plugin tests for queens we will rather need solution 1 for this branch? (but let's focus first on getting the supported branches back in working order)
[1] https://bugs.launchpad.net/neutron/+bug/1859988 [2] https://review.opendev.org/702868
-- Bernard Cafarelli
On 2020-01-17 07:31:24 -0600 (-0600), Ghanshyam Mann wrote: [...]
Best possible solution I can think of is to cap all the Tempest plugins together with Tempest and use corresponding stable branch u-c. Or we modify devstack logic with if-else condition for plugins require cap and rest else will be master. Any other thought? [...]
Constraints is going to be at odds with PEP 503 data-requires-python signaling. If we didn't include neutron-lib in the constraints list for Tempest's virtualenv (maybe filter it out with the edit-constraints tool) then pip should select the highest possible version which matches the versionspec in the requirements list and supports the Python interpreter with which that virtualenv was built. -- Jeremy Stanley
---- On Fri, 17 Jan 2020 09:10:03 -0600 Jeremy Stanley <fungi@yuggoth.org> wrote ----
On 2020-01-17 07:31:24 -0600 (-0600), Ghanshyam Mann wrote: [...]
Best possible solution I can think of is to cap all the Tempest plugins together with Tempest and use corresponding stable branch u-c. Or we modify devstack logic with if-else condition for plugins require cap and rest else will be master. Any other thought? [...]
Constraints is going to be at odds with PEP 503 data-requires-python signaling. If we didn't include neutron-lib in the constraints list for Tempest's virtualenv (maybe filter it out with the edit-constraints tool) then pip should select the highest possible version which matches the versionspec in the requirements list and supports the Python interpreter with which that virtualenv was built.
There will be more lib like neutron-lib, basically all dependency of Tempest or plugins that will become py2 incompatible day by day. -gmann
-- Jeremy Stanley
On 2020-01-17 10:30:32 -0600 (-0600), Ghanshyam Mann wrote:
---- On Fri, 17 Jan 2020 09:10:03 -0600 Jeremy Stanley <fungi@yuggoth.org> wrote ----
On 2020-01-17 07:31:24 -0600 (-0600), Ghanshyam Mann wrote: [...]
Best possible solution I can think of is to cap all the Tempest plugins together with Tempest and use corresponding stable branch u-c. Or we modify devstack logic with if-else condition for plugins require cap and rest else will be master. Any other thought? [...]
Constraints is going to be at odds with PEP 503 data-requires-python signaling. If we didn't include neutron-lib in the constraints list for Tempest's virtualenv (maybe filter it out with the edit-constraints tool) then pip should select the highest possible version which matches the versionspec in the requirements list and supports the Python interpreter with which that virtualenv was built.
There will be more lib like neutron-lib, basically all dependency of Tempest or plugins that will become py2 incompatible day by day.
Yes, but the problem here isn't Python 2.7 incompatibility; it's Python 3.5 incompatibility. We can't run current Tempest on Ubuntu 16.04 LTS without installing a custom Python 3.6 interpreter. -- Jeremy Stanley
---- On Fri, 17 Jan 2020 04:14:49 -0600 Bernard Cafarelli <bcafarel@redhat.com> wrote ----
On Fri, 17 Jan 2020 at 05:11, Jeremy Stanley <fungi@yuggoth.org> wrote: On 2020-01-16 22:02:05 -0600 (-0600), Ghanshyam Mann wrote: [...]
Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. [...]
This makes more sense anyway. Tempest and its plug-ins are already segregated from the system with a virtualenv due to conflicts with stable branch requirements, so hopefully switching that virtualenv to Python 3.x for all jobs is trivial (but I won't be surprised to learn there are subtle challenges hidden just beneath the surface).
That sounds good for supported releases. Once we have them back in working order, I wonder how it will turn out for queens.In neutron, there is a recent failure [1] as this EM branch now uses a pinned version of the plugin. The fix there is most likely to also pin tempest - to queens-em [2] but then will also require some fix for the EOLing python2 drama. As tempest is branchless, it looks like if we want to keep neutron-tempest-plugin tests for queens we will rather need solution 1 for this branch? (but let's focus first on getting the supported branches back in working order)
Yes, for EM branch we need to apply the options#1. Tempest does not support EM branches and we will keep using Tempest master as long as it keeps passing. If it fails due to test incompatibility or any code behaviour, then we need to pin the Tempest. We did this for Ocata[1] and Pike[2]. But for phyton 2.7 drop case, we will use py3 env if possible to test the stable branch until failing due to other reasons then cap it. Currently, we support the Tempest pin by TEEMPEST_BRANCH but no way to pin Tempest Plugins which need some logic in devstack side to pick up the plugin tag from job. [1] https://review.opendev.org/#/c/681950/ [2] https://review.opendev.org/#/c/684769/ -gmann
[1] https://bugs.launchpad.net/neutron/+bug/1859988 [2] https://review.opendev.org/702868
-- Bernard Cafarelli
---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement of 'EOLing python2 drama' in subject :).
neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0 is py3 only and so does u-c on the master has been updated to 2.0.0.
All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying to install the latest neutron-lib and failing.
This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop from master and kepe testing py2 on stable bracnhes.
We have two way to fix this:
1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c. For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. I am trying this first[2] and testing[3].
I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues. Tried option#2: We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3 and more hack for example to set the python alias on such distro. It can be stable common jobs running on Xenial or distro-specific job like centos7 etc where we have < py3.6. Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases. Testing your cloud with the latest Tempest is the best possible way. Going with option#1: IMO, this is a workable option with the current situation. Below is plan to make Tempest and its plugins working for all possible distro/py version. 1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5. * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest on >py3.6 env(venv or separate node). * Patch is up - https://review.opendev.org/#/c/704840/ 2.Modify Tempest tox env basepython to py3 * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6 like fedora or future distro *Patch is up- https://review.opendev.org/#/c/704688/2 3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs. * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs. * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to that branch can be used. For example Tempest 19.0.0 for rocky[3]. * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/ 4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case): We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing. amotoki brought this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using in-tree plugins for their stable branch testing. We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins in neutron-vpnaas case. This will be easy for future maintenance also. Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing. [1] https://review.opendev.org/#/c/703476/ [2] https://review.opendev.org/#/c/703011/ [3] https://releases.openstack.org/rocky/#rocky-tempest -gmann
[1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/... [2] https://review.opendev.org/#/c/703011/ [3] https://review.opendev.org/#/c/703012/
-gmanne
I am writing at top now for easy ready. * Gate status: - All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6 - Stable/stein and stable/train gates are all good as they are on Bionic with py3.6. - The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above. - I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates the tempest tox env with master u-c needs to be fixed[2]. - Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata. * https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED - Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now. You can re-run such jobs. [1] https://review.opendev.org/#/c/705089/ [2] https://review.opendev.org/#/c/705870/ -gmann ---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement of 'EOLing python2 drama' in subject :).
neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0 is py3 only and so does u-c on the master has been updated to 2.0.0.
All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying to install the latest neutron-lib and failing.
This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop from master and kepe testing py2 on stable bracnhes.
We have two way to fix this:
1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c. For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. I am trying this first[2] and testing[3].
I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
Tried option#2: We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3 and more hack for example to set the python alias on such distro. It can be stable common jobs running on Xenial or distro-specific job like centos7 etc where we have < py3.6.
Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases. Testing your cloud with the latest Tempest is the best possible way.
Going with option#1: IMO, this is a workable option with the current situation. Below is plan to make Tempest and its plugins working for all possible distro/py version.
1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5. * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest on >py3.6 env(venv or separate node). * Patch is up - https://review.opendev.org/#/c/704840/
2.Modify Tempest tox env basepython to py3 * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6 like fedora or future distro *Patch is up- https://review.opendev.org/#/c/704688/2
3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs. * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs. * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to that branch can be used. For example Tempest 19.0.0 for rocky[3]. * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case): We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing. amotoki brought this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using in-tree plugins for their stable branch testing. We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins in neutron-vpnaas case. This will be easy for future maintenance also.
Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
[1] https://review.opendev.org/#/c/703476/ [2] https://review.opendev.org/#/c/703011/ [3] https://releases.openstack.org/rocky/#rocky-tempest
-gmann
[1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/... [2] https://review.opendev.org/#/c/703011/ [3] https://review.opendev.org/#/c/703012/
-gmanne
---- On Tue, 04 Feb 2020 18:42:47 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
I am writing at top now for easy ready.
* Gate status: - All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6 - Stable/stein and stable/train gates are all good as they are on Bionic with py3.6. - The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above. - I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates the tempest tox env with master u-c needs to be fixed[2].
While testing the Tempest fix, I realized that it will fix the future Tempest release, not the old tags which are used for stable branch testing. I gave many thoughts on this but cannot find the best way to solve this. Only option seems to cap the upper-constraint like it was proposed by chandan[1]. NOTE: we need to do those cap for all such dependencies of Tempest & its plugins require py>=3.6. We need to maintain such cap only for Tempest & its plugins till stable/rocky EOL. Background on this issue: -------------------------------- Devstack install the Tempest and its plugins in venv using master u-c from tox.ini[2], With Tempest pin on stable branch, devstack can use the stable branch u-c[3] which is all set and Tempest is installed in venv with stable branch u-c. But while running the tests, Tempest roles run-tempest recreate the tox env and using master u-c. I am fixing that in run-tempest roles[4] but that cannot be fixed for Tempest old tags so stable branch testing still broken. ------------------------------------------- [1] https://review.opendev.org/#/c/705685/ [2] https://opendev.org/openstack/tempest/src/commit/bc9fe8eca801f54915ff3eafa41... [3] https://review.opendev.org/#/c/705089 [4] https://review.opendev.org/#/c/705870/22/roles/run-tempest/tasks/main.yaml -gmann
- Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata.
* https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED - Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now. You can re-run such jobs.
[1] https://review.opendev.org/#/c/705089/ [2] https://review.opendev.org/#/c/705870/
-gmann
---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement of 'EOLing python2 drama' in subject :).
neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0 is py3 only and so does u-c on the master has been updated to 2.0.0.
All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying to install the latest neutron-lib and failing.
This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop from master and kepe testing py2 on stable bracnhes.
We have two way to fix this:
1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c. For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. I am trying this first[2] and testing[3].
I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
Tried option#2: We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3 and more hack for example to set the python alias on such distro. It can be stable common jobs running on Xenial or distro-specific job like centos7 etc where we have < py3.6.
Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases. Testing your cloud with the latest Tempest is the best possible way.
Going with option#1: IMO, this is a workable option with the current situation. Below is plan to make Tempest and its plugins working for all possible distro/py version.
1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5. * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest on >py3.6 env(venv or separate node). * Patch is up - https://review.opendev.org/#/c/704840/
2.Modify Tempest tox env basepython to py3 * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6 like fedora or future distro *Patch is up- https://review.opendev.org/#/c/704688/2
3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs. * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs. * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to that branch can be used. For example Tempest 19.0.0 for rocky[3]. * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case): We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing. amotoki brought this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using in-tree plugins for their stable branch testing. We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins in neutron-vpnaas case. This will be easy for future maintenance also.
Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
[1] https://review.opendev.org/#/c/703476/ [2] https://review.opendev.org/#/c/703011/ [3] https://releases.openstack.org/rocky/#rocky-tempest
-gmann
[1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/... [2] https://review.opendev.org/#/c/703011/ [3] https://review.opendev.org/#/c/703012/
-gmanne
---- On Wed, 05 Feb 2020 14:28:28 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 04 Feb 2020 18:42:47 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
I am writing at top now for easy ready.
* Gate status: - All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6 - Stable/stein and stable/train gates are all good as they are on Bionic with py3.6. - The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above. - I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates the tempest tox env with master u-c needs to be fixed[2].
While testing the Tempest fix, I realized that it will fix the future Tempest release, not the old tags which are used for stable branch testing. I gave many thoughts on this but cannot find the best way to solve this. Only option seems to cap the upper-constraint like it was proposed by chandan[1].
NOTE: we need to do those cap for all such dependencies of Tempest & its plugins require py>=3.6. We need to maintain such cap only for Tempest & its plugins till stable/rocky EOL.
Updates: ====== Instead of capping requirement, Tempest role run-tempest fix on master branch work fine because of Zuul pickup the job definition and playbooks from master branch. But there is another issue occurring here. grenade job fails and blocks the fixes on stable branches to merge because old branch has to be fixed first. So we need to merge the stable/ocata fix first and then stab;e/pike and so on. New bug: ======= Another bug on py2 jobs even on stable/train etc. - https://bugs.launchpad.net/tempest/+bug/1862240 Tempest tox env moved to py3 fail the py2 jobs on stable branch jobs who are still using deprecated tox env 'all-plugin'. plugins are installed on py2 in py2 jobs and tempest create tox env with py3 and so does 'all-plugin' which is sitepackage=True try to find plugins on py3 and fail. The solution is to replace the 'all-plugin' tox env to 'all'. I am fixing it for designate. But due to grenade job nature, we need to reverse the fixes here, first, merge the stable/stein and then stable/train. Stay tuned for updates and more bugs. The stable branches gate is still not green, I will update the status on ML about gate staus. -gmann
Background on this issue: --------------------------------
Devstack install the Tempest and its plugins in venv using master u-c from tox.ini[2], With Tempest pin on stable branch, devstack can use the stable branch u-c[3] which is all set and Tempest is installed in venv with stable branch u-c. But while running the tests, Tempest roles run-tempest recreate the tox env and using master u-c.
I am fixing that in run-tempest roles[4] but that cannot be fixed for Tempest old tags so stable branch testing still broken. -------------------------------------------
[1] https://review.opendev.org/#/c/705685/ [2] https://opendev.org/openstack/tempest/src/commit/bc9fe8eca801f54915ff3eafa41... [3] https://review.opendev.org/#/c/705089 [4] https://review.opendev.org/#/c/705870/22/roles/run-tempest/tasks/main.yaml
-gmann
- Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata.
* https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED - Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now. You can re-run such jobs.
[1] https://review.opendev.org/#/c/705089/ [2] https://review.opendev.org/#/c/705870/
-gmann
---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement of 'EOLing python2 drama' in subject :).
neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0 is py3 only and so does u-c on the master has been updated to 2.0.0.
All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying to install the latest neutron-lib and failing.
This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop from master and kepe testing py2 on stable bracnhes.
We have two way to fix this:
1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c. For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. I am trying this first[2] and testing[3].
I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
Tried option#2: We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3 and more hack for example to set the python alias on such distro. It can be stable common jobs running on Xenial or distro-specific job like centos7 etc where we have < py3.6.
Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases. Testing your cloud with the latest Tempest is the best possible way.
Going with option#1: IMO, this is a workable option with the current situation. Below is plan to make Tempest and its plugins working for all possible distro/py version.
1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5. * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest on >py3.6 env(venv or separate node). * Patch is up - https://review.opendev.org/#/c/704840/
2.Modify Tempest tox env basepython to py3 * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6 like fedora or future distro *Patch is up- https://review.opendev.org/#/c/704688/2
3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs. * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs. * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to that branch can be used. For example Tempest 19.0.0 for rocky[3]. * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case): We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing. amotoki brought this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using in-tree plugins for their stable branch testing. We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins in neutron-vpnaas case. This will be easy for future maintenance also.
Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
[1] https://review.opendev.org/#/c/703476/ [2] https://review.opendev.org/#/c/703011/ [3] https://releases.openstack.org/rocky/#rocky-tempest
-gmann
[1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/... [2] https://review.opendev.org/#/c/703011/ [3] https://review.opendev.org/#/c/703012/
-gmanne
---- On Thu, 06 Feb 2020 18:23:51 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Wed, 05 Feb 2020 14:28:28 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 04 Feb 2020 18:42:47 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
I am writing at top now for easy ready.
* Gate status: - All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6 - Stable/stein and stable/train gates are all good as they are on Bionic with py3.6. - The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above. - I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates the tempest tox env with master u-c needs to be fixed[2].
While testing the Tempest fix, I realized that it will fix the future Tempest release, not the old tags which are used for stable branch testing. I gave many thoughts on this but cannot find the best way to solve this. Only option seems to cap the upper-constraint like it was proposed by chandan[1].
NOTE: we need to do those cap for all such dependencies of Tempest & its plugins require py>=3.6. We need to maintain such cap only for Tempest & its plugins till stable/rocky EOL.
Updates: ====== Instead of capping requirement, Tempest role run-tempest fix on master branch work fine because of Zuul pickup the job definition and playbooks from master branch. But there is another issue occurring here. grenade job fails and blocks the fixes on stable branches to merge because old branch has to be fixed first. So we need to merge the stable/ocata fix first and then stab;e/pike and so on.
All the fixes are merged now and good to recheck on stable branch jobs. I am working on bug#1862240 now. - https://review.opendev.org/#/q/topic:fix-stable-gate+(status:open+OR+status:...) -gmann
New bug: ======= Another bug on py2 jobs even on stable/train etc. - https://bugs.launchpad.net/tempest/+bug/1862240
Tempest tox env moved to py3 fail the py2 jobs on stable branch jobs who are still using deprecated tox env 'all-plugin'. plugins are installed on py2 in py2 jobs and tempest create tox env with py3 and so does 'all-plugin' which is sitepackage=True try to find plugins on py3 and fail. The solution is to replace the 'all-plugin' tox env to 'all'. I am fixing it for designate. But due to grenade job nature, we need to reverse the fixes here, first, merge the stable/stein and then stable/train.
Stay tuned for updates and more bugs. The stable branches gate is still not green, I will update the status on ML about gate staus.
-gmann
Background on this issue: --------------------------------
Devstack install the Tempest and its plugins in venv using master u-c from tox.ini[2], With Tempest pin on stable branch, devstack can use the stable branch u-c[3] which is all set and Tempest is installed in venv with stable branch u-c. But while running the tests, Tempest roles run-tempest recreate the tox env and using master u-c.
I am fixing that in run-tempest roles[4] but that cannot be fixed for Tempest old tags so stable branch testing still broken. -------------------------------------------
[1] https://review.opendev.org/#/c/705685/ [2] https://opendev.org/openstack/tempest/src/commit/bc9fe8eca801f54915ff3eafa41... [3] https://review.opendev.org/#/c/705089 [4] https://review.opendev.org/#/c/705870/22/roles/run-tempest/tasks/main.yaml
-gmann
- Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata.
* https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED - Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now. You can re-run such jobs.
[1] https://review.opendev.org/#/c/705089/ [2] https://review.opendev.org/#/c/705870/
-gmann
---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement of 'EOLing python2 drama' in subject :).
neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0 is py3 only and so does u-c on the master has been updated to 2.0.0.
All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying to install the latest neutron-lib and failing.
This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop from master and kepe testing py2 on stable bracnhes.
We have two way to fix this:
1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c. For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. I am trying this first[2] and testing[3].
I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
Tried option#2: We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3 and more hack for example to set the python alias on such distro. It can be stable common jobs running on Xenial or distro-specific job like centos7 etc where we have < py3.6.
Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases. Testing your cloud with the latest Tempest is the best possible way.
Going with option#1: IMO, this is a workable option with the current situation. Below is plan to make Tempest and its plugins working for all possible distro/py version.
1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5. * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest on >py3.6 env(venv or separate node). * Patch is up - https://review.opendev.org/#/c/704840/
2.Modify Tempest tox env basepython to py3 * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6 like fedora or future distro *Patch is up- https://review.opendev.org/#/c/704688/2
3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs. * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs. * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to that branch can be used. For example Tempest 19.0.0 for rocky[3]. * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case): We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing. amotoki brought this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using in-tree plugins for their stable branch testing. We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins in neutron-vpnaas case. This will be easy for future maintenance also.
Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
[1] https://review.opendev.org/#/c/703476/ [2] https://review.opendev.org/#/c/703011/ [3] https://releases.openstack.org/rocky/#rocky-tempest
-gmann
[1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/... [2] https://review.opendev.org/#/c/703011/ [3] https://review.opendev.org/#/c/703012/
-gmanne
Hi,
On 9 Feb 2020, at 01:39, Ghanshyam Mann <gmann@ghanshyammann.com> wrote:
---- On Thu, 06 Feb 2020 18:23:51 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Wed, 05 Feb 2020 14:28:28 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 04 Feb 2020 18:42:47 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
I am writing at top now for easy ready.
* Gate status: - All the stable branch gate till stable/rocky is blocked due to Tempest dependency (oslo today) dropping support for <py3.6 - Stable/stein and stable/train gates are all good as they are on Bionic with py3.6. - The tempest master gate is also blocked due to stable/rocky jobs with the same reason mentioned above. - I am working on fixes. Devstack side installation of Tempest is working[1] with stable u-c but "run-tempest" role which recreates the tempest tox env with master u-c needs to be fixed[2].
While testing the Tempest fix, I realized that it will fix the future Tempest release, not the old tags which are used for stable branch testing. I gave many thoughts on this but cannot find the best way to solve this. Only option seems to cap the upper-constraint like it was proposed by chandan[1].
NOTE: we need to do those cap for all such dependencies of Tempest & its plugins require py>=3.6. We need to maintain such cap only for Tempest & its plugins till stable/rocky EOL.
Updates: ====== Instead of capping requirement, Tempest role run-tempest fix on master branch work fine because of Zuul pickup the job definition and playbooks from master branch. But there is another issue occurring here. grenade job fails and blocks the fixes on stable branches to merge because old branch has to be fixed first. So we need to merge the stable/ocata fix first and then stab;e/pike and so on.
All the fixes are merged now and good to recheck on stable branch jobs. I am working on bug#1862240 now.
- https://review.opendev.org/#/q/topic:fix-stable-gate+(status:open+OR+status:...)
Thx. I see that neutron-tempest-plugin’s rocky jobs now works fine: https://review.opendev.org/#/c/706451/2 finally \o/
-gmann
New bug: ======= Another bug on py2 jobs even on stable/train etc. - https://bugs.launchpad.net/tempest/+bug/1862240
Tempest tox env moved to py3 fail the py2 jobs on stable branch jobs who are still using deprecated tox env 'all-plugin'. plugins are installed on py2 in py2 jobs and tempest create tox env with py3 and so does 'all-plugin' which is sitepackage=True try to find plugins on py3 and fail. The solution is to replace the 'all-plugin' tox env to 'all'. I am fixing it for designate. But due to grenade job nature, we need to reverse the fixes here, first, merge the stable/stein and then stable/train.
Stay tuned for updates and more bugs. The stable branches gate is still not green, I will update the status on ML about gate staus.
-gmann
Background on this issue: --------------------------------
Devstack install the Tempest and its plugins in venv using master u-c from tox.ini[2], With Tempest pin on stable branch, devstack can use the stable branch u-c[3] which is all set and Tempest is installed in venv with stable branch u-c. But while running the tests, Tempest roles run-tempest recreate the tox env and using master u-c.
I am fixing that in run-tempest roles[4] but that cannot be fixed for Tempest old tags so stable branch testing still broken. -------------------------------------------
[1] https://review.opendev.org/#/c/705685/ [2] https://opendev.org/openstack/tempest/src/commit/bc9fe8eca801f54915ff3eafa41... [3] https://review.opendev.org/#/c/705089 [4] https://review.opendev.org/#/c/705870/22/roles/run-tempest/tasks/main.yaml
-gmann
- Once stable/rocky is green, I will push the fixes on stable/queens|pike|ocata.
* https://bugs.launchpad.net/tempest/+bug/1861308 - FIXED - Tempest fix for >py3.6 is merged and if your 3rd party CI or a distro with >py3.6 should work fine now. You can re-run such jobs.
[1] https://review.opendev.org/#/c/705089/ [2] https://review.opendev.org/#/c/705870/
-gmann
---- On Wed, 29 Jan 2020 16:57:25 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Thu, 16 Jan 2020 22:02:05 -0600 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
This is regarding bug: https://bugs.launchpad.net/tempest/+bug/1860033. Using Radosław's fancy statement of 'EOLing python2 drama' in subject :).
neutron tempest plugin job on stable/rocky started failing as neutron-lib dropped the py2. neutron-lib 2.0.0 is py3 only and so does u-c on the master has been updated to 2.0.0.
All tempest and its plugin uses the master u-c for stable branch testing which is the valid way because of master Tempest & plugin is being used to test the stable branches which need u-c from master itself. These failed jobs also used master u-c[1] which is trying to install the latest neutron-lib and failing.
This is not just neutron tempest plugin issue but for all Tempest plugins jobs. Any lib used by Tempest or plugins can drop the py2 now and leads to this failure. Its just neutron-lib raised the flag first before I plan to hack on Tempest & plugins jobs for py2 drop from master and kepe testing py2 on stable bracnhes.
We have two way to fix this:
1. Separate out the testing of python2 jobs with python2 supported version of Tempest plugins and with respective u-c. For example, test all python2 job with tempest plugin train version (or any latest version if any which support py2) and use u-c from stable/train. This will cap the Tempest & plugins with respective u-c for stable branches testing.
2. Second option is to install the tempest and plugins in py3 env on py2 jobs also. This should be an easy and preferred way. I am trying this first[2] and testing[3].
I am summarizing what Tempest and its plugins should be doing/done for these incompatible issues.
Tried option#2: We tried to install the py3.6 (from ppa which is not the best solution) in Tempest venv on ubuntu Xenail to fix the bug like 1860033 [1]. This needs Tempest to bump the py version for tox env t 3.6[2]. But that broke the distro job where py > 3.6 was available like fedora (Bug 1861308). This can be fixed by making basepython as pythion3 and more hack for example to set the python alias on such distro. It can be stable common jobs running on Xenial or distro-specific job like centos7 etc where we have < py3.6.
Overall this option did not work well as this need lot of hacks depends on the distro. I am dropping this option for our CI/CD. But you can try this on your production cloud testing where you do not need to handle multiple distro cases. Testing your cloud with the latest Tempest is the best possible way.
Going with option#1: IMO, this is a workable option with the current situation. Below is plan to make Tempest and its plugins working for all possible distro/py version.
1. Drop py3.5 from Tempest (also from its plugins if anyone officially supports). * Tempest and its plugin's dependencies are becoming python-requires >=3.6 so Tempest and plugins itself cannot support py3.5. * 'Tempest cannot support py3.5' means cannot run Tempest/plugins on py3.5 env. But still, you can test py3.5 cloud from Tempest on >py3.6 env(venv or separate node). * Patch is up - https://review.opendev.org/#/c/704840/
2.Modify Tempest tox env basepython to py3 * Let's not pin Tempest for py3.6. Any python version >=py3.6 should be working fine for distro does not have py3.6 like fedora or future distro *Patch is up- https://review.opendev.org/#/c/704688/2
3. Use compatible Tempest & its plugin tag for distro having <py3.6 jobs. This is for stable/rocky or centos7 etc jobs. * Tempest 23.0.0 is the last version to support py2 or py3.5. This tag can be used to test py2 or ppy3.5 jobs. * If 23.0.0 is not compatible with stable branch u-c or any tempest plugins tag then Tempest tag corresponding to that branch can be used. For example Tempest 19.0.0 for rocky[3]. * We have used gerrit style way to pin Tempest in past but we are trying tag name now - https://review.opendev.org/#/c/704899/
4. Stable jobs using in-tree tempest plugins (neutron-vpnaas case): We have few cases like neutron-vpnaas stable/rocky where in-tree plugin is used for stable testing. amotoki brought this yesterday. neutron-vpnaas tempest plugin has been moved to neutron-tempest-plugin now but stable/rocky jobs still use in-tree plugin which is causing issues due to incompatible py version on devstack and Tempest tox env(which moved to py3). These jobs use tox -e all-plugins for in-tree plugins. This issue is not just neutron-vpnaas but any project still using in-tree plugins for their stable branch testing. We can solve this by pinning Tempest also + few more hack (which I am sure will be required). But best and easy way to fix these stable branch jobs are to migrate them to use tox ''all' env with separate-repo plugins. For example neutron-tempest-plugins in neutron-vpnaas case. This will be easy for future maintenance also.
Anything stable/stein onwards is all good till now so we will keep using master Tempest/Plugins for their testing.
[1] https://review.opendev.org/#/c/703476/ [2] https://review.opendev.org/#/c/703011/ [3] https://releases.openstack.org/rocky/#rocky-tempest
-gmann
[1] https://zuul.opendev.org/t/openstack/build/fb8a928ed3614e09a9a3cf4637f2f6c2/... [2] https://review.opendev.org/#/c/703011/ [3] https://review.opendev.org/#/c/703012/
-gmanne
— Slawek Kaplonski Senior software engineer Red Hat
participants (5)
-
Bernard Cafarelli
-
Ghanshyam Mann
-
Jeremy Stanley
-
Radosław Piliszek
-
Slawek Kaplonski