On 11/23/23 07:40, Alan Bishop wrote:

On Mon, Nov 20, 2023 at 12:47 AM Sylvain Bauza sbauza@redhat.com wrote:

Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com> a
écrit :

    Hey folks,
    FWIW, our grenade check on SLURP upgrades is 100% failing now
    [1] due to a behavioural change in the tooz library [2] that's
    pulled by Cinder and which was released [3] and delivered to
    upper-contraints yesterday [4]

    There are different possibilities in order to unblock the
    gate, but until one of them is agreed and merged, please avoid
    wasting our CI resources by asking to recheck for something
    which will fail.


The short-term fix, reverting the upper-constraints change, was
merged on Friday evening :
https://review.opendev.org/c/openstack/requirements/+/901079

Now, the gate job is back \o/
That said, now tooz and the cinder community should find a
solution for the long-term.

I am not familiar with the grenade job, but here's what I observe. The latest tooz release (4.3.0) seems to be paired with a corresponding devstack patch that bumps the etcd version from 3.3.12 to 3.4.27. See [5] for details. The grenade log I happened to examine [6] does show the correct version of devstack being checked out:

2023-11-17 05:32:01.136212 | controller |opendev.org/openstack/devstack http://opendev.org/openstack/devstack checked out to: 2023-11-17 05:32:01.136273 | controller | bb0c273697bf54dd569ad38e459cd161b62f96cb Option for SQLAlchemy and alembic git source But I gather the job retains a separate "old" and "new" devstack, and apparently the "old" one is being used to prepare things: 2023-11-17 05:34:30.934847 | controller | ++stackrc:source:724 : ETCD_VERSION=v3.3.12 ... 2023-11-17 05:37:46.023835 | controller | + lib/cinder:configure_cinder:409 : iniset /etc/cinder/cinder.conf coordination backend_url etcd3+http://158.69.68.66:2379

I suspect cinder is being configured to use the old interface compatible with etcd v3.3.12, but tooz assumes it's working with etcd v3.4.27. I don't know if this info is helpful, but hopefully someone else knows where to go next.

It looked into this issue when it was reported in the begging and dumped my summary in the bug.

https://bugs.launchpad.net/python-tooz/+bug/2043810

In short, the issue was caused by the facts below.

- etcd is not updated during grenade so has only old API (v3alpha/beta)

- the configuration is created by stable/2023.1 devestack and is not updated during grenade,    so users the default api version in tooz

- tooz is updated during grenadea and the default api has been changed to v3, assuming that    users would set the available api version according to etcd3 server version.

So there are a few potential fixes.

1. Hard-code api version in stable/2023.1 to use v1beta to use that version consistently during upgrade,      as expected by current tooz .

2. Update tooz to detect available API version instead of relying on the hard-coded default

I proposed the solution 1 ( https://review.opendev.org/c/openstack/devstack/+/901221 ) earlier but some concern was raised about it, mainly because operators might not notice the requirement and face a same problem during upgrade.

So I started implementing the solution 2 in https://review.opendev.org/q/topic:%22bug/2043810%22 .

[5] https://review.opendev.org/q/topic:%22etcd-3.4%22 [6] https://252c20d223888c59399f-4adf110539aed9252479dc6b548d2884.ssl.cf1.rackcd...

Alan

    Thanks,
    -Sylvain (who will send an email once the job is back up)

    [1]
    https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova
    <https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova>
    [2] https://review.opendev.org/c/openstack/tooz/+/891355
    [3] https://review.opendev.org/c/openstack/releases/+/899966
    [4] https://review.opendev.org/c/openstack/requirements/+/901131

We just released tooz 5.0.0, which contains the capability to detect the available etcd api version automatically and I believe this should address the problem earlier found in the skip level job.

The new release will be used in CI once the version in upper-constraint is bumped by [1]. Please let me know if you face the same problem (or a different one) after the bump is merged.

[1] https://review.opendev.org/c/openstack/requirements/+/905223

On 11/23/23 17:31, Takashi Kajinami wrote:

On 11/23/23 17:17, Takashi Kajinami wrote:

...

On 11/23/23 07:40, Alan Bishop wrote:

...

On Mon, Nov 20, 2023 at 12:47 AM Sylvain Bauza sbauza@redhat.com wrote:

Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com>
a écrit :

    Hey folks,
    FWIW, our grenade check on SLURP upgrades is 100% failing
    now [1] due to a behavioural change in the tooz library [2]
    that's pulled by Cinder and which was released [3] and
    delivered to upper-contraints yesterday [4]

    There are different possibilities in order to unblock the
    gate, but until one of them is agreed and merged, please
    avoid wasting our CI resources by asking to recheck for
    something which will fail.


The short-term fix, reverting the upper-constraints change, was
merged on Friday evening :
https://review.opendev.org/c/openstack/requirements/+/901079

Now, the gate job is back \o/
That said, now tooz and the cinder community should find a
solution for the long-term.

I am not familiar with the grenade job, but here's what I observe. The latest tooz release (4.3.0) seems to be paired with a corresponding devstack patch that bumps the etcd version from 3.3.12 to 3.4.27. See [5] for details. The grenade log I happened to examine [6] does show the correct version of devstack being checked out:

2023-11-17 05:32:01.136212 | controller |opendev.org/openstack/devstack http://opendev.org/openstack/devstack checked out to: 2023-11-17 05:32:01.136273 | controller | bb0c273697bf54dd569ad38e459cd161b62f96cb Option for SQLAlchemy and alembic git source But I gather the job retains a separate "old" and "new" devstack, and apparently the "old" one is being used to prepare things: 2023-11-17 05:34:30.934847 | controller | ++stackrc:source:724 : ETCD_VERSION=v3.3.12 ... 2023-11-17 05:37:46.023835 | controller | + lib/cinder:configure_cinder:409 : iniset /etc/cinder/cinder.conf coordination backend_url etcd3+http://158.69.68.66:2379

I suspect cinder is being configured to use the old interface compatible with etcd v3.3.12, but tooz assumes it's working with etcd v3.4.27. I don't know if this info is helpful, but hopefully someone else knows where to go next.

It looked into this issue when it was reported in the begging and dumped my summary in the bug.

in the *beginning*

Wrong spell-fix caused by typo X-(

...

https://bugs.launchpad.net/python-tooz/+bug/2043810

In short, the issue was caused by the facts below.

- etcd is not updated during grenade so has only old API (v3alpha/beta)

- the configuration is created by stable/2023.1 devestack and is not updated during grenade,    so users the default api version in tooz

- tooz is updated during grenadea and the default api has been changed to v3, assuming that    users would set the available api version according to etcd3 server version.

So there are a few potential fixes.

1. Hard-code api version in stable/2023.1 to use v1beta to use that version consistently during upgrade,      as expected by current tooz .

2. Update tooz to detect available API version instead of relying on the hard-coded default

I proposed the solution 1 ( https://review.opendev.org/c/openstack/devstack/+/901221 ) earlier but some concern was raised about it, mainly because operators might not notice the requirement and face a same problem during upgrade.

So I started implementing the solution 2 in https://review.opendev.org/q/topic:%22bug/2043810%22 .

...

[5] https://review.opendev.org/q/topic:%22etcd-3.4%22 [6] https://252c20d223888c59399f-4adf110539aed9252479dc6b548d2884.ssl.cf1.rackcd...

Alan

    Thanks,
    -Sylvain (who will send an email once the job is back up)

    [1]
    https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova
    <https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova>
    [2] https://review.opendev.org/c/openstack/tooz/+/891355
    [3] https://review.opendev.org/c/openstack/releases/+/899966
    [4] https://review.opendev.org/c/openstack/requirements/+/901131