On 19-08-19 10:54:37, Matthew Treinish wrote:

On Sun, Aug 18, 2019 at 11:16:11AM -0500, Matthew Thode wrote:

...

NOVA: lxml===4.4.1 nova tests fail https://bugs.launchpad.net/nova/+bug/1838666 websockify===0.9.0 tempest test failing

KEYSTONE: oauthlib===3.1.0 keystone https://bugs.launchpad.net/keystone/+bug/1839393

NEUTRON: tenacity===5.1.1 https://2c976b5e9e9a7bed9985-82d79a041e998664bd1d0bc4b6e78332.ssl.cf2.rackcd... this could be caused by pytest===5.1.0 as well

KURYR: kubernetes===10.0.1 openshift PINS this, only kuryr-tempest-plugin deps on it https://review.opendev.org/665352

MISC: tornado===5.1.1 salt is cauing this, no eta on fix (same as the last year) stestr===2.5.0 needs merged https://github.com/mtreinish/stestr/pull/265

This actually doesn't fix the underlying issue blocking it here. PR 265 is for fixing a compatibility issue with python 3.4, which we don't officially support in stestr but was a simple fix. The blocker is actually not an stestr issue, it's a testtools bug:

https://github.com/testing-cabal/testtools/issues/272

Where this is coming into play here is that stestr 2.5.0 switched to using an internal test runner built off of stdlib unittest instead of testtools/subunit for python 3. This was done to fix a huge number of compatibility issues people had reported when trying to run stdlib unittest suites using stestr on python >= 3.5 (which were caused by unittest2 and testools). The complication for openstack (more specificially tempest) is that it's built off of testtools not stdlib unittest. So when tempest raises 'self.skipException' as part of it's class level skip checks testtools raises 'unittest2.case.SkipTest' instead of 'unittest.case.SkipTest'. stdlib unittest does not understand what that is and treats it as an unhandled exception which is a test failure, instead of the intended skip result. [1] This is actually a general bug and will come up whenever anyone tries to use stdlib unittest to run tempest. We need to come up with a fix for this problem in testtools [2] or just workaround it in tempest.

[1] skip decorators typically aren't effected by this because they set an attribute that gets checked before the test method is executed instead of relying on an exception, which is why this is mostly only an issue for tempest because it does a lot of run time skips via exceptions.

[2] testtools is mostly unmaintained at this point, I was recently granted merge access but haven't had much free time to actively maintain it

-Matt Treinish

...

jsonschema===3.0.2 see https://review.opendev.org/649789

I'm trying to get this in place as we are getting closer to the requirements freeze (sept 9th-13th). Any help clearing up these bugs would be appreciated.

-- Matthew Thode

Thanks for the clarification, now to get the other projects to pay attention :|

On 8/21/19 3:44 PM, Eric Fried wrote:

...

...

...

NOVA: lxml===4.4.1 nova tests fail https://bugs.launchpad.net/nova/+bug/1838666

Sean Mooney agreed to take a look at this one.

...

...

...

websockify===0.9.0 tempest test failing

This is now known as bug 1840788 [1].

I did some initial investigation and tried to fix it. Turned out my fix was already part of another change. But it also turns out that that fix is only part of the solution. See the bug report for (links to) details.

At this point, it needs someone who understands what test_novnc [2] is actually trying to do. Because I don't.

Anyone?

From the test run on your patch [3], it looks like we're going to need a change in nova as well in nova/console/websocketproxy.py [4]:

AttributeError: module 'websockify' has no attribute 'WebSocketServer'

It looks like the WebSocketServer class got moved from websockify.websocket to websockify.websocketserver in v0.9.0:

https://github.com/novnc/websockify/commit/8a697622495fd319582cd1c604e7eb2cc...

and thus is no longer accessible via the top level 'websockify' module as a result:

https://github.com/novnc/websockify/blob/v0.9.0/websockify/__init__.py

Since this is a change to upper-constraints to allow v0.9.0, we will need a way for it to work with both module layouts, yeah?

Cheers, -melanie

[3] https://review.opendev.org/677798 [4] https://zuul.opendev.org/t/openstack/build/e0a8a19021b64350a0a55cc08f374d02/...

Thanks, efried

[1] https://bugs.launchpad.net/nova/+bug/1840788 [2] https://opendev.org/openstack/tempest/src/branch/master/tempest/api/compute/...

On 8/21/19 4:30 PM, Clark Boylan wrote:

On Wed, Aug 21, 2019, at 4:25 PM, melanie witt wrote:

Snip

...

Since this is a change to upper-constraints to allow v0.9.0, we will need a way for it to work with both module layouts, yeah?

Yup, the constrained version is likely one of many valid versions people can use. The constrained version is just what we test with to prevent issues like these from breaking the gate. Unless you also change the lower bounds of the dep people may still continue to use the old version too. Good idea to support the older code in that case.

Thanks Clark!

I've uploaded a nova change at: https://review.opendev.org/677856

and added it as a Depends-On to: https://review.opendev.org/677798

to see if it solves the problem.

Cheers, -melanie

On 19-08-22 12:45:59, Eric Fried wrote:

{NOVA: websockify 0.9.0 breaks test_novnc [1]}

...

I've uploaded a nova change at: https://review.opendev.org/677856

and added it as a Depends-On to: https://review.opendev.org/677798

to see if it solves the problem.

It does; it's now merging. Once it and [2] (tempest) have merged, we should be able to proceed with the u-c bump.

Thanks Mel!

@prometheanfire, you could now swizzle [3] to un-DNM and Depends-On those two. (Is that how this is done, or do we need to let a bot-generated patch like [4] do it?)

efried

[1] https://bugs.launchpad.net/nova/+bug/1840788 [2] https://review.opendev.org/674364 [3] https://review.opendev.org/677479 [4] https://review.opendev.org/677619

I've un DNM'd it, thanks for the work

---- On Mon, 19 Aug 2019 23:54:37 +0900 Matthew Treinish mtreinish@kortar.org wrote ----

On Sun, Aug 18, 2019 at 11:16:11AM -0500, Matthew Thode wrote:

...

NOVA: lxml===4.4.1 nova tests fail https://bugs.launchpad.net/nova/+bug/1838666 websockify===0.9.0 tempest test failing

KEYSTONE: oauthlib===3.1.0 keystone https://bugs.launchpad.net/keystone/+bug/1839393

NEUTRON: tenacity===5.1.1 https://2c976b5e9e9a7bed9985-82d79a041e998664bd1d0bc4b6e78332.ssl.cf2.rackcd... this could be caused by pytest===5.1.0 as well

KURYR: kubernetes===10.0.1 openshift PINS this, only kuryr-tempest-plugin deps on it https://review.opendev.org/665352

MISC: tornado===5.1.1 salt is cauing this, no eta on fix (same as the last year) stestr===2.5.0 needs merged https://github.com/mtreinish/stestr/pull/265

This actually doesn't fix the underlying issue blocking it here. PR 265 is for fixing a compatibility issue with python 3.4, which we don't officially support in stestr but was a simple fix. The blocker is actually not an stestr issue, it's a testtools bug:

https://github.com/testing-cabal/testtools/issues/272

Where this is coming into play here is that stestr 2.5.0 switched to using an internal test runner built off of stdlib unittest instead of testtools/subunit for python 3. This was done to fix a huge number of compatibility issues people had reported when trying to run stdlib unittest suites using stestr on python >= 3.5 (which were caused by unittest2 and testools). The complication for openstack (more specificially tempest) is that it's built off of testtools not stdlib unittest. So when tempest raises 'self.skipException' as part of it's class level skip checks testtools raises 'unittest2.case.SkipTest' instead of 'unittest.case.SkipTest'. stdlib unittest does not understand what that is and treats it as an unhandled exception which is a test failure, instead of the intended skip result. [1] This is actually a general bug and will come up whenever anyone tries to use stdlib unittest to run tempest. We need to come up with a fix for this problem in testtools [2] or just workaround it in tempest.

[1] skip decorators typically aren't effected by this because they set an attribute that gets checked before the test method is executed instead of relying on an exception, which is why this is mostly only an issue for tempest because it does a lot of run time skips via exceptions.

[2] testtools is mostly unmaintained at this point, I was recently granted merge access but haven't had much free time to actively maintain it

Thanks matt for details. As you know, for Tempest where we need to support py2.7 (including unitest2 use) for stable branches, we are going to use the specific stetsr version/branch(<when you will remove the py27 support) so fixing it on testtools side is good option to me. I think your PR to remove the unittest2 use form testtools make sense to me [1]. A workaround in Tempest can be last option for us.

Till we fix it and to avoid gate break, can we cap stestr in g-r - stestr<2.5.0 ? I know that is not the options you like.

[1] https://github.com/mtreinish/testtools/commit/38fc9a9e302f68d471d7b097c7327b...

-gmann

-Matt Treinish

...

jsonschema===3.0.2 see https://review.opendev.org/649789

I'm trying to get this in place as we are getting closer to the requirements freeze (sept 9th-13th). Any help clearing up these bugs would be appreciated.

-- Matthew Thode

---- On Thu, 22 Aug 2019 04:41:44 +0900 Matthew Treinish mtreinish@kortar.org wrote ----

On Wed, Aug 21, 2019 at 07:21:41PM +0900, Ghanshyam Mann wrote:

...

---- On Mon, 19 Aug 2019 23:54:37 +0900 Matthew Treinish mtreinish@kortar.org wrote ----

...

On Sun, Aug 18, 2019 at 11:16:11AM -0500, Matthew Thode wrote:

...

NOVA: lxml===4.4.1 nova tests fail https://bugs.launchpad.net/nova/+bug/1838666 websockify===0.9.0 tempest test failing

KEYSTONE: oauthlib===3.1.0 keystone https://bugs.launchpad.net/keystone/+bug/1839393

NEUTRON: tenacity===5.1.1 https://2c976b5e9e9a7bed9985-82d79a041e998664bd1d0bc4b6e78332.ssl.cf2.rackcd... this could be caused by pytest===5.1.0 as well

KURYR: kubernetes===10.0.1 openshift PINS this, only kuryr-tempest-plugin deps on it https://review.opendev.org/665352

MISC: tornado===5.1.1 salt is cauing this, no eta on fix (same as the last year) stestr===2.5.0 needs merged https://github.com/mtreinish/stestr/pull/265

This actually doesn't fix the underlying issue blocking it here. PR 265 is for fixing a compatibility issue with python 3.4, which we don't officially support in stestr but was a simple fix. The blocker is actually not an stestr issue, it's a testtools bug:

https://github.com/testing-cabal/testtools/issues/272

Where this is coming into play here is that stestr 2.5.0 switched to using an internal test runner built off of stdlib unittest instead of testtools/subunit for python 3. This was done to fix a huge number of compatibility issues people had reported when trying to run stdlib unittest suites using stestr on python >= 3.5 (which were caused by unittest2 and testools). The complication for openstack (more specificially tempest) is that it's built off of testtools not stdlib unittest. So when tempest raises 'self.skipException' as part of it's class level skip checks testtools raises 'unittest2.case.SkipTest' instead of 'unittest.case.SkipTest'. stdlib unittest does not understand what that is and treats it as an unhandled exception which is a test failure, instead of the intended skip result. [1] This is actually a general bug and will come up whenever anyone tries to use stdlib unittest to run tempest. We need to come up with a fix for this problem in testtools [2] or just workaround it in tempest.

[1] skip decorators typically aren't effected by this because they set an attribute that gets checked before the test method is executed instead of relying on an exception, which is why this is mostly only an issue for tempest because it does a lot of run time skips via exceptions.

[2] testtools is mostly unmaintained at this point, I was recently granted merge access but haven't had much free time to actively maintain it

Thanks matt for details. As you know, for Tempest where we need to support py2.7 (including unitest2 use) for stable branches, we are going to use the specific stetsr version/branch(<when you will remove the py27 support) so fixing it on testtools side is good option to me. I think your PR to remove the unittest2 use form testtools make sense to me [1]. A workaround in Tempest can be last option for us.

https://github.com/testing-cabal/testtools/pull/277 isn't a short term solution, unittest2 is still needed for python < 3.5 in testtools and testtools has not deprecated support for python 2.7 or 3.4 yet. I probably can rework that PR so that it's conditional and always uses stdlib unittest for python >= 3.5 but then testtools ends up maintaining two separate paths depending on python version. I'd like to continue thinking about that is as a long term solution because I don't know when I'll have the time to keep pushing that PR forward.

Thanks for more details. I understand that might take time. I am in OpenInfra event and after that on vacation till 29th Aug. I will be able to check the workaround on testtools or tempest side after that only. I will check with Matthew about when is the plan to move the stestr to 2.5.0.

-gmann

...

Till we fix it and to avoid gate break, can we cap stestr in g-r - stestr<2.5.0 ? I know that is not the options you like.

[1] https://github.com/mtreinish/testtools/commit/38fc9a9e302f68d471d7b097c7327b...

-gmann

...

-Matt Treinish

...

jsonschema===3.0.2 see https://review.opendev.org/649789

I'm trying to get this in place as we are getting closer to the requirements freeze (sept 9th-13th). Any help clearing up these bugs would be appreciated.

-- Matthew Thode

---- On Wed, 11 Sep 2019 00:23:08 +0900 Matthew Thode mthode@mthode.org wrote ----

On 19-08-23 20:09:31, Ghanshyam Mann wrote:

...

---- On Thu, 22 Aug 2019 04:41:44 +0900 Matthew Treinish mtreinish@kortar.org wrote ----

...

On Wed, Aug 21, 2019 at 07:21:41PM +0900, Ghanshyam Mann wrote:

...

---- On Mon, 19 Aug 2019 23:54:37 +0900 Matthew Treinish mtreinish@kortar.org wrote ----

...

On Sun, Aug 18, 2019 at 11:16:11AM -0500, Matthew Thode wrote:

...

NOVA: lxml===4.4.1 nova tests fail https://bugs.launchpad.net/nova/+bug/1838666 websockify===0.9.0 tempest test failing

KEYSTONE: oauthlib===3.1.0 keystone https://bugs.launchpad.net/keystone/+bug/1839393

NEUTRON: tenacity===5.1.1 https://2c976b5e9e9a7bed9985-82d79a041e998664bd1d0bc4b6e78332.ssl.cf2.rackcd... this could be caused by pytest===5.1.0 as well

KURYR: kubernetes===10.0.1 openshift PINS this, only kuryr-tempest-plugin deps on it https://review.opendev.org/665352

MISC: tornado===5.1.1 salt is cauing this, no eta on fix (same as the last year) stestr===2.5.0 needs merged https://github.com/mtreinish/stestr/pull/265

This actually doesn't fix the underlying issue blocking it here. PR 265 is for fixing a compatibility issue with python 3.4, which we don't officially support in stestr but was a simple fix. The blocker is actually not an stestr issue, it's a testtools bug:

https://github.com/testing-cabal/testtools/issues/272

Where this is coming into play here is that stestr 2.5.0 switched to using an internal test runner built off of stdlib unittest instead of testtools/subunit for python 3. This was done to fix a huge number of compatibility issues people had reported when trying to run stdlib unittest suites using stestr on python >= 3.5 (which were caused by unittest2 and testools). The complication for openstack (more specificially tempest) is that it's built off of testtools not stdlib unittest. So when tempest raises 'self.skipException' as part of it's class level skip checks testtools raises 'unittest2.case.SkipTest' instead of 'unittest.case.SkipTest'. stdlib unittest does not understand what that is and treats it as an unhandled exception which is a test failure, instead of the intended skip result. [1] This is actually a general bug and will come up whenever anyone tries to use stdlib unittest to run tempest. We need to come up with a fix for this problem in testtools [2] or just workaround it in tempest.

[1] skip decorators typically aren't effected by this because they set an attribute that gets checked before the test method is executed instead of relying on an exception, which is why this is mostly only an issue for tempest because it does a lot of run time skips via exceptions.

[2] testtools is mostly unmaintained at this point, I was recently granted merge access but haven't had much free time to actively maintain it

Thanks matt for details. As you know, for Tempest where we need to support py2.7 (including unitest2 use) for stable branches, we are going to use the specific stetsr version/branch(<when you will remove the py27 support) so fixing it on testtools side is good option to me. I think your PR to remove the unittest2 use form testtools make sense to me [1]. A workaround in Tempest can be last option for us.

https://github.com/testing-cabal/testtools/pull/277 isn't a short term solution, unittest2 is still needed for python < 3.5 in testtools and testtools has not deprecated support for python 2.7 or 3.4 yet. I probably can rework that PR so that it's conditional and always uses stdlib unittest for python >= 3.5 but then testtools ends up maintaining two separate paths depending on python version. I'd like to continue thinking about that is as a long term solution because I don't know when I'll have the time to keep pushing that PR forward.

Thanks for more details. I understand that might take time. I am in OpenInfra event and after that on vacation till 29th Aug. I will be able to check the workaround on testtools or tempest side after that only. I will check with Matthew about when is the plan to move the stestr to 2.5.0.

-gmann

...

...

Till we fix it and to avoid gate break, can we cap stestr in g-r - stestr<2.5.0 ? I know that is not the options you like.

[1] https://github.com/mtreinish/testtools/commit/38fc9a9e302f68d471d7b097c7327b...

-gmann

...

-Matt Treinish

...

jsonschema===3.0.2 see https://review.opendev.org/649789

I'm trying to get this in place as we are getting closer to the requirements freeze (sept 9th-13th). Any help clearing up these bugs would be appreciated.

-- Matthew Thode

Any progress on this, at the moment only stestr-2.5.1 is being held back.

https://review.opendev.org/680914

There is no progress on this yet. As unittest2 cannot be dropped from testtools, we need to get some workaround in Tempest. I need more time to try the failure and fix.

-gmann

-- Matthew Thode

On Thu, 2019-08-22 at 08:17 -0500, Matthew Thode wrote:

On 19-08-22 09:53:30, Michał Dulko wrote:

...

On Sun, 2019-08-18 at 11:16 -0500, Matthew Thode wrote:

...

NOVA: lxml===4.4.1 nova tests fail https://bugs.launchpad.net/nova/+bug/1838666 websockify===0.9.0 tempest test failing

KEYSTONE: oauthlib===3.1.0 keystone https://bugs.launchpad.net/keystone/+bug/1839393

NEUTRON: tenacity===5.1.1 https://2c976b5e9e9a7bed9985-82d79a041e998664bd1d0bc4b6e78332.ssl.cf2.rackcd... this could be caused by pytest===5.1.0 as well

KURYR: kubernetes===10.0.1 openshift PINS this, only kuryr-tempest-plugin deps on it https://review.opendev.org/665352

Alright, I tested and fixed the commit above and I should be able to get it merged today. Thanks for following up on this.

Yep, saw it merge. Now all it needs is a release so the version on pypy / upper-constraints does not hold back the version of kubernetes.

Alright, let's see how my release commit [1] will work then.

[1] https://review.opendev.org/#/c/678542/

...

...

MISC: tornado===5.1.1 salt is cauing this, no eta on fix (same as the last year) stestr===2.5.0 needs merged https://github.com/mtreinish/stestr/pull/265 jsonschema===3.0.2 see https://review.opendev.org/649789

I'm trying to get this in place as we are getting closer to the requirements freeze (sept 9th-13th). Any help clearing up these bugs would be appreciated.