[all] POST_FAILURE in openstack-tox-py310
I feel like this week is one of the most exciting week we have had recently :-P We started seeing consistent POST_FAILURE in openstack-tox-py310 (and other jobs using py310). https://github.com/testing-cabal/testtools/issues/569 The issue was caused by testtools and was fixed in testools 2.8.4, but we likely need to rebuild the image used in CI to pull the fixed version. Please avoid mass recheck or merging any new changes now before you check the CI status carefully. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit
On Thu, 2026-02-19 at 21:30 +0900, Takashi Kajinami wrote:
I feel like this week is one of the most exciting week we have had recently :-P
Tell me about it 🙄😅
We started seeing consistent POST_FAILURE in openstack-tox-py310 (and other jobs using py310).
https://github.com/testing-cabal/testtools/issues/569
The issue was caused by testtools and was fixed in testools 2.8.4, but we likely need to rebuild the image used in CI to pull the fixed version.
Do we know how this got in? upper-constraints has us capped at 2.7.2 [1]. Stephen [1] https://opendev.org/openstack/requirements/src/commit/b25e902b3f92b796e8c452...
Please avoid mass recheck or merging any new changes now before you check the CI status carefully.
Thank you, Takashi
On 2/19/26 10:05 PM, Stephen Finucane wrote:
On Thu, 2026-02-19 at 21:30 +0900, Takashi Kajinami wrote:
I feel like this week is one of the most exciting week we have had recently :-P
Tell me about it 🙄😅
We started seeing consistent POST_FAILURE in openstack-tox-py310 (and other jobs using py310).
https://github.com/testing-cabal/testtools/issues/569
The issue was caused by testtools and was fixed in testools 2.8.4, but we likely need to rebuild the image used in CI to pull the fixed version.
Do we know how this got in? upper-constraints has us capped at 2.7.2 [1].
IIUC the failing task does not use the tox, and uses the libraries installed during image build. https://opendev.org/opendev/zuul-providers/src/branch/master/dib-elements/no... (Thanks frickler++ for pointing it I'm now talking with fungi++ to get the required images rebuilt to unblock the failing job at least in master sooner, instead of waiting for the next scheduled rebuild.
Stephen
[1] https://opendev.org/openstack/requirements/src/commit/b25e902b3f92b796e8c452...
Please avoid mass recheck or merging any new changes now before you check the CI status carefully.
Thank you, Takashi
I think the issue was resolved by reverting the CI images to an old ones with testtools 2.8.2 . On 2/19/26 10:34 PM, Takashi Kajinami wrote:
On 2/19/26 10:05 PM, Stephen Finucane wrote:
On Thu, 2026-02-19 at 21:30 +0900, Takashi Kajinami wrote:
I feel like this week is one of the most exciting week we have had recently :-P
Tell me about it 🙄😅
We started seeing consistent POST_FAILURE in openstack-tox-py310 (and other jobs using py310).
https://github.com/testing-cabal/testtools/issues/569
The issue was caused by testtools and was fixed in testools 2.8.4, but we likely need to rebuild the image used in CI to pull the fixed version.
Do we know how this got in? upper-constraints has us capped at 2.7.2 [1].
IIUC the failing task does not use the tox, and uses the libraries installed during image build. https://opendev.org/opendev/zuul-providers/src/branch/master/dib-elements/no... (Thanks frickler++ for pointing it
I'm now talking with fungi++ to get the required images rebuilt to unblock the failing job at least in master sooner, instead of waiting for the next scheduled rebuild.
Stephen
[1] https://opendev.org/openstack/requirements/src/commit/b25e902b3f92b796e8c452...
Please avoid mass recheck or merging any new changes now before you check the CI status carefully.
Thank you, Takashi
participants (2)
-
Stephen Finucane
-
Takashi Kajinami