Hi, I started seeing an error yesterday when doing this command: tempest UPPER_CONSTRAINTS_FILE= https://releases.openstack.org/constraints/upper/master tox -eall -- baremetal --concurrency=1 The specified regex doesn't match with anythingError: InvocationError for command /opt/stack/new/tempest/.tox/tempest/bin/tempest run --regex baremetal --concurrency=1 Any ideas how could I resolve this? Thanks!
On Thursday, 23 April 2020 00:04:25 CEST Eric Barrera wrote:
Hi,
I started seeing an error yesterday when doing this command:
tempest UPPER_CONSTRAINTS_FILE= https://releases.openstack.org/constraints/upper/master tox -eall -- baremetal --concurrency=1
The specified regex doesn't match with anythingError: InvocationError for command /opt/stack/new/tempest/.tox/tempest/bin/tempest run --regex baremetal --concurrency=1
Is it a legacy zuul job? I've seen this on legacy jobs, we were discussing about it earlier on #openstack-qa (now on #openstack-infra). IIRC I think it was working until earlier today. I captured the relevant logs where the behavior diverges between the (previously) working and the (now) failing jobs: http://paste.openstack.org/show/792573/ -- Luigi
Yes these are legacy Zuulv2 scripts. Do we have a workaround or fix? Thanks -rajini -----Original Message----- From: Luigi Toscano <ltoscano@redhat.com> Sent: Wednesday, April 22, 2020 5:56 PM To: openstack-discuss@lists.openstack.org Cc: Eric Barrera Subject: Re: [tempest] regex failing [EXTERNAL EMAIL] On Thursday, 23 April 2020 00:04:25 CEST Eric Barrera wrote:
Hi,
I started seeing an error yesterday when doing this command:
tempest UPPER_CONSTRAINTS_FILE= https://releases.openstack.org/constraints/upper/master tox -eall -- baremetal --concurrency=1
The specified regex doesn't match with anythingError: InvocationError for command /opt/stack/new/tempest/.tox/tempest/bin/tempest run --regex baremetal --concurrency=1
Is it a legacy zuul job? I've seen this on legacy jobs, we were discussing about it earlier on #openstack-qa (now on #openstack-infra). IIRC I think it was working until earlier today. I captured the relevant logs where the behavior diverges between the (previously) working and the (now) failing jobs: http://paste.openstack.org/show/792573/ -- Luigi
Yes these are legacy Zuulv2 scripts. Do we have a workaround or fix? Thanks -rajini -----Original Message----- From: Luigi Toscano <ltoscano@redhat.com> Sent: Wednesday, April 22, 2020 5:56 PM To: openstack-discuss@lists.openstack.org Cc: Eric Barrera Subject: Re: [tempest] regex failing [EXTERNAL EMAIL] On Thursday, 23 April 2020 00:04:25 CEST Eric Barrera wrote:
Hi,
I started seeing an error yesterday when doing this command:
tempest UPPER_CONSTRAINTS_FILE= https://releases.openstack.org/constraints/upper/master tox -eall -- baremetal --concurrency=1
The specified regex doesn't match with anythingError: InvocationError for command /opt/stack/new/tempest/.tox/tempest/bin/tempest run --regex baremetal --concurrency=1
Is it a legacy zuul job? I've seen this on legacy jobs, we were discussing about it earlier on #openstack-qa (now on #openstack-infra). IIRC I think it was working until earlier today. I captured the relevant logs where the behavior diverges between the (previously) working and the (now) failing jobs: http://paste.openstack.org/show/792573/ -- Luigi
---- On Wed, 22 Apr 2020 17:04:25 -0500 Eric Barrera <ebarrera1211@gmail.com> wrote ----
Hi, I started seeing an error yesterday when doing this command: tempest UPPER_CONSTRAINTS_FILE=https://releases.openstack.org/constraints/upper/master tox -eall -- baremetal --concurrency=1
The specified regex doesn't match with anythingError: InvocationError for command /opt/stack/new/tempest/.tox/tempest/bin/tempest run --regex baremetal --concurrency=1
Any ideas how could I resolve this?
This is a little tricky and not the issue on plugins side how it looks like from error message. I have been fixing this on various place (devstack, grenade, devstack-gate, tempest and some of tempest plugins ) since py2 drop drama started in mid of Ussuri[1]. Tempest & its plugins are installed on virtual env and supposed to test from there so that Tempest and plugins master can be used for stable jobs or py2 jobs. "venv-tempest" is the venv name we use for Tempest venv so that every place it can be used to install plugins, configure and run the tests. devstack setup the venv and installation/configure the Tempest, its plugins and its all good till now but the problem comes when other place like grenade or plugins scripts or devstack-gate for legacy jobs start running or configure something in that venv and end up *recreating* it. *recreating* the venv is the issue which flushes the installed tempest plugins. so that when jobs try to run the plugins tests it fails with test regex. This is what happening now in the legacy job. devstack-gate running tempest tests end up recreating the venv. Why it started recreating the venv even that was fixed earlier[1] is recent change in Tempest tox.ini to use the stable branch constraint for a new release of Tempest. Once Tempest release the new tag we are supposed to make tox.ini use the master constraint again - https://review.opendev.org/#/c/721724/. The window between tox.ini modified to use stable constraint (temporary) till Tempest new release is done and the revert to modify it back to master constraint caused legacy jobs failing. Window should have been minimal but we faced another issue on nova-next job which has to be fixed before Tempst release [2]. Summary: Its release time and a lot of issues occurring as this cycle is roller coaster of py2 drop drama. The fix is up and we will get this merge asap and bring legacy jobs back by tonight. - https://review.opendev.org/#/c/721724/ [1] https://review.opendev.org/#/q/topic:fix-stable-gate+(status:open+OR+status:...) [2] https://review.opendev.org/#/c/721767/ -gmann
Thanks!
---- On Wed, 22 Apr 2020 20:48:34 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Wed, 22 Apr 2020 17:04:25 -0500 Eric Barrera <ebarrera1211@gmail.com> wrote ----
Hi, I started seeing an error yesterday when doing this command: tempest UPPER_CONSTRAINTS_FILE=https://releases.openstack.org/constraints/upper/master tox -eall -- baremetal --concurrency=1
The specified regex doesn't match with anythingError: InvocationError for command /opt/stack/new/tempest/.tox/tempest/bin/tempest run --regex baremetal --concurrency=1
Any ideas how could I resolve this?
This is a little tricky and not the issue on plugins side how it looks like from error message. I have been fixing this on various place (devstack, grenade, devstack-gate, tempest and some of tempest plugins ) since py2 drop drama started in mid of Ussuri[1].
Tempest & its plugins are installed on virtual env and supposed to test from there so that Tempest and plugins master can be used for stable jobs or py2 jobs. "venv-tempest" is the venv name we use for Tempest venv so that every place it can be used to install plugins, configure and run the tests.
devstack setup the venv and installation/configure the Tempest, its plugins and its all good till now but the problem comes when other place like grenade or plugins scripts or devstack-gate for legacy jobs start running or configure something in that venv and end up *recreating* it. *recreating* the venv is the issue which flushes the installed tempest plugins. so that when jobs try to run the plugins tests it fails with test regex. This is what happening now in the legacy job. devstack-gate running tempest tests end up recreating the venv.
Why it started recreating the venv even that was fixed earlier[1] is recent change in Tempest tox.ini to use the stable branch constraint for a new release of Tempest. Once Tempest release the new tag we are supposed to make tox.ini use the master constraint again - https://review.opendev.org/#/c/721724/.
The window between tox.ini modified to use stable constraint (temporary) till Tempest new release is done and the revert to modify it back to master constraint caused legacy jobs failing. Window should have been minimal but we faced another issue on nova-next job which has to be fixed before Tempst release [2].
Summary: Its release time and a lot of issues occurring as this cycle is roller coaster of py2 drop drama. The fix is up and we will get this merge asap and bring legacy jobs back by tonight.
Fix is merged, please recheck the job or re-run your CI. -gmann
[1] https://review.opendev.org/#/q/topic:fix-stable-gate+(status:open+OR+status:...) [2] https://review.opendev.org/#/c/721767/
-gmann
Thanks!
On Thursday, 23 April 2020 03:48:34 CEST Ghanshyam Mann wrote:
---- On Wed, 22 Apr 2020 17:04:25 -0500 Eric Barrera <ebarrera1211@gmail.com> wrote ----
Hi,
I started seeing an error yesterday when doing this command: tempest UPPER_CONSTRAINTS_FILE=https://releases.openstack.org/constraints/up per/master tox -eall -- baremetal --concurrency=1 > The specified regex doesn't match with anythingError: InvocationError for command /opt/stack/new/tempest/.tox/tempest/bin/tempest run --regex baremetal --concurrency=1
Any ideas how could I resolve this?
This is a little tricky and not the issue on plugins side how it looks like from error message. I have been fixing this on various place (devstack, grenade, devstack-gate, tempest and some of tempest plugins ) since py2 drop drama started in mid of Ussuri[1].
And thanks for fixing all of those issues, including this one. Just one comment:
Tempest & its plugins are installed on virtual env and supposed to test from there so that Tempest and plugins master can be used for stable jobs or py2 jobs. "venv-tempest" is the venv name we use for Tempest venv so that every place it can be used to install plugins, configure and run the tests.
devstack setup the venv and installation/configure the Tempest, its plugins and its all good till now but the problem comes when other place like grenade or plugins scripts or devstack-gate for legacy jobs start running or configure something in that venv and end up *recreating* it. *recreating* the venv is the issue which flushes the installed tempest plugins. so that when jobs try to run the plugins tests it fails with test regex. This is what happening now in the legacy job. devstack-gate running tempest tests end up recreating the venv.
Why it started recreating the venv even that was fixed earlier[1] is recent change in Tempest tox.ini to use the stable branch constraint for a new release of Tempest. Once Tempest release the new tag we are supposed to make tox.ini use the master constraint again - https://review.opendev.org/#/c/721724/.
I've seen that change, but I didn't think that it was the reason for the issue because right now the content of https://releases.openstack.org/constraints/upper/ussuri and https://releases.openstack.org/constraints/upper/master is the same. I guess I need to recheck the workflow. -- Luigi
---- On Thu, 23 Apr 2020 03:42:19 -0500 Luigi Toscano <ltoscano@redhat.com> wrote ----
On Thursday, 23 April 2020 03:48:34 CEST Ghanshyam Mann wrote:
---- On Wed, 22 Apr 2020 17:04:25 -0500 Eric Barrera <ebarrera1211@gmail.com> wrote ----
Hi,
I started seeing an error yesterday when doing this command: tempest UPPER_CONSTRAINTS_FILE=https://releases.openstack.org/constraints/up per/master tox -eall -- baremetal --concurrency=1 > The specified regex doesn't match with anythingError: InvocationError for command /opt/stack/new/tempest/.tox/tempest/bin/tempest run --regex baremetal --concurrency=1
Any ideas how could I resolve this?
This is a little tricky and not the issue on plugins side how it looks like from error message. I have been fixing this on various place (devstack, grenade, devstack-gate, tempest and some of tempest plugins ) since py2 drop drama started in mid of Ussuri[1].
And thanks for fixing all of those issues, including this one. Just one comment:
Tempest & its plugins are installed on virtual env and supposed to test from there so that Tempest and plugins master can be used for stable jobs or py2 jobs. "venv-tempest" is the venv name we use for Tempest venv so that every place it can be used to install plugins, configure and run the tests.
devstack setup the venv and installation/configure the Tempest, its plugins and its all good till now but the problem comes when other place like grenade or plugins scripts or devstack-gate for legacy jobs start running or configure something in that venv and end up *recreating* it. *recreating* the venv is the issue which flushes the installed tempest plugins. so that when jobs try to run the plugins tests it fails with test regex. This is what happening now in the legacy job. devstack-gate running tempest tests end up recreating the venv.
Why it started recreating the venv even that was fixed earlier[1] is recent change in Tempest tox.ini to use the stable branch constraint for a new release of Tempest. Once Tempest release the new tag we are supposed to make tox.ini use the master constraint again - https://review.opendev.org/#/c/721724/.
I've seen that change, but I didn't think that it was the reason for the issue because right now the content of https://releases.openstack.org/constraints/upper/ussuri and https://releases.openstack.org/constraints/upper/master
is the same. I guess I need to recheck the workflow.
Yeah, content is same for both constraint but issue istox env is recreated because there is some change in constraint file being used which remove the plugins installed in original tox env. -gmann
-- Luigi
participants (4)
-
Eric Barrera
-
Ghanshyam Mann
-
Luigi Toscano
-
Rajini.Karthik＠Dell.com