We just released tooz 5.0.0, which contains the capability to
detect the available etcd api version
automatically and I believe this should address the problem
earlier found in the skip level job.
The new release will be used in CI once the version in
upper-constraint is bumped by [1].
Please let me know if you face the same problem (or a different
one) after the bump is
merged.
[1] https://review.opendev.org/c/openstack/requirements/+/905223
On 11/23/23 17:17, Takashi Kajinami wrote:
On 11/23/23 07:40, Alan Bishop wrote:
On Mon, Nov 20, 2023 at 12:47 AM Sylvain Bauza <sbauza@redhat.com> wrote:
Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com> a écrit :
Hey folks,FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4]
There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail.
The short-term fix, reverting the upper-constraints change, was merged on Friday evening : https://review.opendev.org/c/openstack/requirements/+/901079
Now, the gate job is back \o/That said, now tooz and the cinder community should find a solution for the long-term.
I am not familiar with the grenade job, but here's what I observe. The latest tooz release (4.3.0) seems to be paired with a corresponding devstack patch that bumps the etcd version from 3.3.12 to 3.4.27. See [5] for details. The grenade log I happened to examine [6] does show the correct version of devstack being checked out:
2023-11-17 05:32:01.136212 | controller | opendev.org/openstack/devstack checked out to: 2023-11-17 05:32:01.136273 | controller | bb0c273697bf54dd569ad38e459cd161b62f96cb Option for SQLAlchemy and alembic git sourceBut I gather the job retains a separate "old" and "new" devstack, and apparently the "old" one is being used to prepare things:2023-11-17 05:34:30.934847 | controller | ++ stackrc:source:724 : ETCD_VERSION=v3.3.12 ... 2023-11-17 05:37:46.023835 | controller | + lib/cinder:configure_cinder:409 : iniset /etc/cinder/cinder.conf coordination backend_url etcd3+http://158.69.68.66:2379
I suspect cinder is being configured to use the old interface compatible with etcd v3.3.12, but tooz assumes it's working with etcd v3.4.27. I don't know if this info is helpful, but hopefully someone else knows where to go next.
It looked into this issue when it was reported in the begging and dumped my summary in the bug.
in the *beginning*
Wrong spell-fix caused by typo X-(
https://bugs.launchpad.net/python-tooz/+bug/2043810
In short, the issue was caused by the facts below.
- etcd is not updated during grenade so has only old API (v3alpha/beta)
- the configuration is created by stable/2023.1 devestack and is not updated during grenade,
so users the default api version in tooz
- tooz is updated during grenadea and the default api has been changed to v3, assuming that
users would set the available api version according to etcd3 server version.
So there are a few potential fixes.
1. Hard-code api version in stable/2023.1 to use v1beta to use that version consistently during upgrade,
as expected by current tooz .
2. Update tooz to detect available API version instead of relying on the hard-coded default
I proposed the solution 1 ( https://review.opendev.org/c/openstack/devstack/+/901221 ) earlier but
some concern was raised about it, mainly because operators might not notice the requirement and
face a same problem during upgrade.So I started implementing the solution 2 in https://review.opendev.org/q/topic:%22bug/2043810%22 .
Alan
Thanks,-Sylvain (who will send an email once the job is back up)