[nova][release][tooz] Hold your rechecks
Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4] There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail. Thanks, -Sylvain (who will send an email once the job is back up) [1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova [2] https://review.opendev.org/c/openstack/tooz/+/891355 [3] https://review.opendev.org/c/openstack/releases/+/899966 [4] https://review.opendev.org/c/openstack/requirements/+/901131
On Fri, 17 Nov 2023 at 04:33, Sylvain Bauza <sbauza@redhat.com> wrote:
Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4]
I see you created a revert[5] for the upper-constraints bump. I have single core approved this. There may be issues with the upcoming gerrit upgrade given how long the requirements jobs take to run. Yours Tony. [5] https://review.opendev.org/c/openstack/requirements/+/901079
Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com> a écrit :
Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4]
There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail.
The short-term fix, reverting the upper-constraints change, was merged on Friday evening : https://review.opendev.org/c/openstack/requirements/+/901079 Now, the gate job is back \o/ That said, now tooz and the cinder community should find a solution for the long-term.
Thanks, -Sylvain (who will send an email once the job is back up)
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova [2] https://review.opendev.org/c/openstack/tooz/+/891355 [3] https://review.opendev.org/c/openstack/releases/+/899966 [4] https://review.opendev.org/c/openstack/requirements/+/901131
On Mon, Nov 20, 2023 at 12:47 AM Sylvain Bauza <sbauza@redhat.com> wrote:
Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com> a écrit :
Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4]
There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail.
The short-term fix, reverting the upper-constraints change, was merged on Friday evening : https://review.opendev.org/c/openstack/requirements/+/901079
Now, the gate job is back \o/ That said, now tooz and the cinder community should find a solution for the long-term.
I am not familiar with the grenade job, but here's what I observe. The latest tooz release (4.3.0) seems to be paired with a corresponding devstack patch that bumps the etcd version from 3.3.12 to 3.4.27. See [5] for details. The grenade log I happened to examine [6] does show the correct version of devstack being checked out: 2023-11-17 05:32:01.136212 | controller | opendev.org/openstack/devstack checked out to: 2023-11-17 05:32:01.136273 | controller | bb0c273697bf54dd569ad38e459cd161b62f96cb Option for SQLAlchemy and alembic git source But I gather the job retains a separate "old" and "new" devstack, and apparently the "old" one is being used to prepare things: 2023-11-17 05:34:30.934847 | controller | ++ stackrc:source:724 : ETCD_VERSION=v3.3.12 ... 2023-11-17 05:37:46.023835 | controller | + lib/cinder:configure_cinder:409 : iniset /etc/cinder/cinder.conf coordination backend_url etcd3+http://158.69.68.66:2379 I suspect cinder is being configured to use the old interface compatible with etcd v3.3.12, but tooz assumes it's working with etcd v3.4.27. I don't know if this info is helpful, but hopefully someone else knows where to go next. [5] https://review.opendev.org/q/topic:%22etcd-3.4%22 [6] https://252c20d223888c59399f-4adf110539aed9252479dc6b548d2884.ssl.cf1.rackcd... Alan Thanks,
-Sylvain (who will send an email once the job is back up)
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova [2] https://review.opendev.org/c/openstack/tooz/+/891355 [3] https://review.opendev.org/c/openstack/releases/+/899966 [4] https://review.opendev.org/c/openstack/requirements/+/901131
On 11/23/23 07:40, Alan Bishop wrote:
On Mon, Nov 20, 2023 at 12:47 AM Sylvain Bauza <sbauza@redhat.com> wrote:
Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com> a écrit :
Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4]
There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail.
The short-term fix, reverting the upper-constraints change, was merged on Friday evening : https://review.opendev.org/c/openstack/requirements/+/901079
Now, the gate job is back \o/ That said, now tooz and the cinder community should find a solution for the long-term.
I am not familiar with the grenade job, but here's what I observe. The latest tooz release (4.3.0) seems to be paired with a corresponding devstack patch that bumps the etcd version from 3.3.12 to 3.4.27. See [5] for details. The grenade log I happened to examine [6] does show the correct version of devstack being checked out:
2023-11-17 05:32:01.136212 | controller |opendev.org/openstack/devstack <http://opendev.org/openstack/devstack> checked out to: 2023-11-17 05:32:01.136273 | controller | bb0c273697bf54dd569ad38e459cd161b62f96cb Option for SQLAlchemy and alembic git source But I gather the job retains a separate "old" and "new" devstack, and apparently the "old" one is being used to prepare things: 2023-11-17 05:34:30.934847 | controller | ++stackrc:source:724 : ETCD_VERSION=v3.3.12 ... 2023-11-17 05:37:46.023835 | controller | + lib/cinder:configure_cinder:409 : iniset /etc/cinder/cinder.conf coordination backend_url etcd3+http://158.69.68.66:2379
I suspect cinder is being configured to use the old interface compatible with etcd v3.3.12, but tooz assumes it's working with etcd v3.4.27. I don't know if this info is helpful, but hopefully someone else knows where to go next.
It looked into this issue when it was reported in the begging and dumped my summary in the bug. https://bugs.launchpad.net/python-tooz/+bug/2043810 In short, the issue was caused by the facts below. - etcd is not updated during grenade so has only old API (v3alpha/beta) - the configuration is created by stable/2023.1 devestack and is not updated during grenade, so users the default api version in tooz - tooz is updated during grenadea and the default api has been changed to v3, assuming that users would set the available api version according to etcd3 server version. So there are a few potential fixes. 1. Hard-code api version in stable/2023.1 to use v1beta to use that version consistently during upgrade, as expected by current tooz . 2. Update tooz to detect available API version instead of relying on the hard-coded default I proposed the solution 1 ( https://review.opendev.org/c/openstack/devstack/+/901221 ) earlier but some concern was raised about it, mainly because operators might not notice the requirement and face a same problem during upgrade. So I started implementing the solution 2 in https://review.opendev.org/q/topic:%22bug/2043810%22 .
[5] https://review.opendev.org/q/topic:%22etcd-3.4%22 [6] https://252c20d223888c59399f-4adf110539aed9252479dc6b548d2884.ssl.cf1.rackcd...
Alan
Thanks, -Sylvain (who will send an email once the job is back up)
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova <https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova> [2] https://review.opendev.org/c/openstack/tooz/+/891355 [3] https://review.opendev.org/c/openstack/releases/+/899966 [4] https://review.opendev.org/c/openstack/requirements/+/901131
On 11/23/23 17:17, Takashi Kajinami wrote:
On 11/23/23 07:40, Alan Bishop wrote:
On Mon, Nov 20, 2023 at 12:47 AM Sylvain Bauza <sbauza@redhat.com> wrote:
Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com> a écrit :
Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4]
There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail.
The short-term fix, reverting the upper-constraints change, was merged on Friday evening : https://review.opendev.org/c/openstack/requirements/+/901079
Now, the gate job is back \o/ That said, now tooz and the cinder community should find a solution for the long-term.
I am not familiar with the grenade job, but here's what I observe. The latest tooz release (4.3.0) seems to be paired with a corresponding devstack patch that bumps the etcd version from 3.3.12 to 3.4.27. See [5] for details. The grenade log I happened to examine [6] does show the correct version of devstack being checked out:
2023-11-17 05:32:01.136212 | controller |opendev.org/openstack/devstack <http://opendev.org/openstack/devstack> checked out to: 2023-11-17 05:32:01.136273 | controller | bb0c273697bf54dd569ad38e459cd161b62f96cb Option for SQLAlchemy and alembic git source But I gather the job retains a separate "old" and "new" devstack, and apparently the "old" one is being used to prepare things: 2023-11-17 05:34:30.934847 | controller | ++stackrc:source:724 : ETCD_VERSION=v3.3.12 ... 2023-11-17 05:37:46.023835 | controller | + lib/cinder:configure_cinder:409 : iniset /etc/cinder/cinder.conf coordination backend_url etcd3+http://158.69.68.66:2379
I suspect cinder is being configured to use the old interface compatible with etcd v3.3.12, but tooz assumes it's working with etcd v3.4.27. I don't know if this info is helpful, but hopefully someone else knows where to go next.
It looked into this issue when it was reported in the begging and dumped my summary in the bug.
in the *beginning* Wrong spell-fix caused by typo X-(
https://bugs.launchpad.net/python-tooz/+bug/2043810
In short, the issue was caused by the facts below.
- etcd is not updated during grenade so has only old API (v3alpha/beta)
- the configuration is created by stable/2023.1 devestack and is not updated during grenade, so users the default api version in tooz
- tooz is updated during grenadea and the default api has been changed to v3, assuming that users would set the available api version according to etcd3 server version.
So there are a few potential fixes.
1. Hard-code api version in stable/2023.1 to use v1beta to use that version consistently during upgrade, as expected by current tooz .
2. Update tooz to detect available API version instead of relying on the hard-coded default
I proposed the solution 1 ( https://review.opendev.org/c/openstack/devstack/+/901221 ) earlier but some concern was raised about it, mainly because operators might not notice the requirement and face a same problem during upgrade.
So I started implementing the solution 2 in https://review.opendev.org/q/topic:%22bug/2043810%22 .
[5] https://review.opendev.org/q/topic:%22etcd-3.4%22 [6] https://252c20d223888c59399f-4adf110539aed9252479dc6b548d2884.ssl.cf1.rackcd...
Alan
Thanks, -Sylvain (who will send an email once the job is back up)
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova <https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova> [2] https://review.opendev.org/c/openstack/tooz/+/891355 [3] https://review.opendev.org/c/openstack/releases/+/899966 [4] https://review.opendev.org/c/openstack/requirements/+/901131
We just released tooz 5.0.0, which contains the capability to detect the available etcd api version automatically and I believe this should address the problem earlier found in the skip level job. The new release will be used in CI once the version in upper-constraint is bumped by [1]. Please let me know if you face the same problem (or a different one) after the bump is merged. [1] https://review.opendev.org/c/openstack/requirements/+/905223 On 11/23/23 17:31, Takashi Kajinami wrote:
On 11/23/23 17:17, Takashi Kajinami wrote:
On 11/23/23 07:40, Alan Bishop wrote:
On Mon, Nov 20, 2023 at 12:47 AM Sylvain Bauza <sbauza@redhat.com> wrote:
Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com> a écrit :
Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4]
There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail.
The short-term fix, reverting the upper-constraints change, was merged on Friday evening : https://review.opendev.org/c/openstack/requirements/+/901079
Now, the gate job is back \o/ That said, now tooz and the cinder community should find a solution for the long-term.
I am not familiar with the grenade job, but here's what I observe. The latest tooz release (4.3.0) seems to be paired with a corresponding devstack patch that bumps the etcd version from 3.3.12 to 3.4.27. See [5] for details. The grenade log I happened to examine [6] does show the correct version of devstack being checked out:
2023-11-17 05:32:01.136212 | controller |opendev.org/openstack/devstack <http://opendev.org/openstack/devstack> checked out to: 2023-11-17 05:32:01.136273 | controller | bb0c273697bf54dd569ad38e459cd161b62f96cb Option for SQLAlchemy and alembic git source But I gather the job retains a separate "old" and "new" devstack, and apparently the "old" one is being used to prepare things: 2023-11-17 05:34:30.934847 | controller | ++stackrc:source:724 : ETCD_VERSION=v3.3.12 ... 2023-11-17 05:37:46.023835 | controller | + lib/cinder:configure_cinder:409 : iniset /etc/cinder/cinder.conf coordination backend_url etcd3+http://158.69.68.66:2379
I suspect cinder is being configured to use the old interface compatible with etcd v3.3.12, but tooz assumes it's working with etcd v3.4.27. I don't know if this info is helpful, but hopefully someone else knows where to go next.
It looked into this issue when it was reported in the begging and dumped my summary in the bug.
in the *beginning*
Wrong spell-fix caused by typo X-(
https://bugs.launchpad.net/python-tooz/+bug/2043810
In short, the issue was caused by the facts below.
- etcd is not updated during grenade so has only old API (v3alpha/beta)
- the configuration is created by stable/2023.1 devestack and is not updated during grenade, so users the default api version in tooz
- tooz is updated during grenadea and the default api has been changed to v3, assuming that users would set the available api version according to etcd3 server version.
So there are a few potential fixes.
1. Hard-code api version in stable/2023.1 to use v1beta to use that version consistently during upgrade, as expected by current tooz .
2. Update tooz to detect available API version instead of relying on the hard-coded default
I proposed the solution 1 ( https://review.opendev.org/c/openstack/devstack/+/901221 ) earlier but some concern was raised about it, mainly because operators might not notice the requirement and face a same problem during upgrade.
So I started implementing the solution 2 in https://review.opendev.org/q/topic:%22bug/2043810%22 .
[5] https://review.opendev.org/q/topic:%22etcd-3.4%22 [6] https://252c20d223888c59399f-4adf110539aed9252479dc6b548d2884.ssl.cf1.rackcd...
Alan
Thanks, -Sylvain (who will send an email once the job is back up)
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova <https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova> [2] https://review.opendev.org/c/openstack/tooz/+/891355 [3] https://review.opendev.org/c/openstack/releases/+/899966 [4] https://review.opendev.org/c/openstack/requirements/+/901131
participants (4)
-
Alan Bishop
-
Sylvain Bauza
-
Takashi Kajinami
-
Tony Breeds