On 11/23/23 07:40, Alan Bishop wrote:
On Mon, Nov 20, 2023 at 12:47 AM Sylvain Bauza <sbauza@redhat.com> wrote:
Le ven. 17 nov. 2023 à 11:24, Sylvain Bauza <sbauza@redhat.com> a écrit :
Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4]
There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail.
The short-term fix, reverting the upper-constraints change, was merged on Friday evening : https://review.opendev.org/c/openstack/requirements/+/901079
Now, the gate job is back \o/ That said, now tooz and the cinder community should find a solution for the long-term.
I am not familiar with the grenade job, but here's what I observe. The latest tooz release (4.3.0) seems to be paired with a corresponding devstack patch that bumps the etcd version from 3.3.12 to 3.4.27. See [5] for details. The grenade log I happened to examine [6] does show the correct version of devstack being checked out:
2023-11-17 05:32:01.136212 | controller |opendev.org/openstack/devstack <http://opendev.org/openstack/devstack> checked out to: 2023-11-17 05:32:01.136273 | controller | bb0c273697bf54dd569ad38e459cd161b62f96cb Option for SQLAlchemy and alembic git source But I gather the job retains a separate "old" and "new" devstack, and apparently the "old" one is being used to prepare things: 2023-11-17 05:34:30.934847 | controller | ++stackrc:source:724 : ETCD_VERSION=v3.3.12 ... 2023-11-17 05:37:46.023835 | controller | + lib/cinder:configure_cinder:409 : iniset /etc/cinder/cinder.conf coordination backend_url etcd3+http://158.69.68.66:2379
I suspect cinder is being configured to use the old interface compatible with etcd v3.3.12, but tooz assumes it's working with etcd v3.4.27. I don't know if this info is helpful, but hopefully someone else knows where to go next.
It looked into this issue when it was reported in the begging and dumped my summary in the bug. https://bugs.launchpad.net/python-tooz/+bug/2043810 In short, the issue was caused by the facts below. - etcd is not updated during grenade so has only old API (v3alpha/beta) - the configuration is created by stable/2023.1 devestack and is not updated during grenade, so users the default api version in tooz - tooz is updated during grenadea and the default api has been changed to v3, assuming that users would set the available api version according to etcd3 server version. So there are a few potential fixes. 1. Hard-code api version in stable/2023.1 to use v1beta to use that version consistently during upgrade, as expected by current tooz . 2. Update tooz to detect available API version instead of relying on the hard-coded default I proposed the solution 1 ( https://review.opendev.org/c/openstack/devstack/+/901221 ) earlier but some concern was raised about it, mainly because operators might not notice the requirement and face a same problem during upgrade. So I started implementing the solution 2 in https://review.opendev.org/q/topic:%22bug/2043810%22 .
[5] https://review.opendev.org/q/topic:%22etcd-3.4%22 [6] https://252c20d223888c59399f-4adf110539aed9252479dc6b548d2884.ssl.cf1.rackcd...
Alan
Thanks, -Sylvain (who will send an email once the job is back up)
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova <https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova> [2] https://review.opendev.org/c/openstack/tooz/+/891355 [3] https://review.opendev.org/c/openstack/releases/+/899966 [4] https://review.opendev.org/c/openstack/requirements/+/901131