On Mon, Nov 20, 2023 at 12:47 AM Sylvain Bauza <sbauza@redhat.com> wrote:
Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com> a écrit :
Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4]
There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail.
The short-term fix, reverting the upper-constraints change, was merged on Friday evening : https://review.opendev.org/c/openstack/requirements/+/901079
Now, the gate job is back \o/ That said, now tooz and the cinder community should find a solution for the long-term.
I am not familiar with the grenade job, but here's what I observe. The latest tooz release (4.3.0) seems to be paired with a corresponding devstack patch that bumps the etcd version from 3.3.12 to 3.4.27. See [5] for details. The grenade log I happened to examine [6] does show the correct version of devstack being checked out: 2023-11-17 05:32:01.136212 | controller | opendev.org/openstack/devstack checked out to: 2023-11-17 05:32:01.136273 | controller | bb0c273697bf54dd569ad38e459cd161b62f96cb Option for SQLAlchemy and alembic git source But I gather the job retains a separate "old" and "new" devstack, and apparently the "old" one is being used to prepare things: 2023-11-17 05:34:30.934847 | controller | ++ stackrc:source:724 : ETCD_VERSION=v3.3.12 ... 2023-11-17 05:37:46.023835 | controller | + lib/cinder:configure_cinder:409 : iniset /etc/cinder/cinder.conf coordination backend_url etcd3+http://158.69.68.66:2379 I suspect cinder is being configured to use the old interface compatible with etcd v3.3.12, but tooz assumes it's working with etcd v3.4.27. I don't know if this info is helpful, but hopefully someone else knows where to go next. [5] https://review.opendev.org/q/topic:%22etcd-3.4%22 [6] https://252c20d223888c59399f-4adf110539aed9252479dc6b548d2884.ssl.cf1.rackcd... Alan Thanks,
-Sylvain (who will send an email once the job is back up)
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova [2] https://review.opendev.org/c/openstack/tooz/+/891355 [3] https://review.opendev.org/c/openstack/releases/+/899966 [4] https://review.opendev.org/c/openstack/requirements/+/901131