On 11/9/23 22:30, Stephen Finucane wrote:
Just FYI, for projects that have sqlalchemy-tips jobs that have been failing recently, they should now be passing again. The root cause was neither sqlalchemy nor alembic but rather oslo.utils [1]. It was fixed in oslo.utils 6.3.0 which was released [2] roughly an hour ago. A recheck should be sufficient to clear errors and as these tips jobs only run on master there should be no issues with stable branches.
It later turned out unit tests of nova and ironic are broken by the oslo.utils fix, which is preventing us from bumping oslo.utils version in u-c[1]. I've reported a bug[2] and submitted fixes to these projects[3][4]. It'd be nice if someone from nova/ironic core team can pay attention to these patches. # I had to disable the sqlalchemy-2x jobs to merge these unit test fixes, # and such change in zuul config triggered full set of tests... so it may take # a while until all triggered jobs complete. [1] https://review.opendev.org/c/openstack/requirements/+/900517 [2] https://bugs.launchpad.net/nova/+bug/2043116 [3] https://review.opendev.org/c/openstack/nova/+/900532 [4] https://review.opendev.org/c/openstack/ironic/+/900533
A further point to add: while I appreciate that broken gates are never a good thing, I would be careful making these jobs non-voting. I've already seen a few instances where SQLAlchemy 2.0 support has regressed and I for one have no desire to re-herd cats as we push to get SQLAlchemy uncapped in the coming weeks. If this happens again then we can re-examine things, naturally.
I agree that making a job non-voting is not ideal, and for sqlalchemy-2x we have to prioritize keeping it because we are committed to use that version in a near future. However I'm wondering if we have to keep the job with the latest code voting in normal queue or we can replace it by a job with the latest release of sqlalchemy. Because we may require updates in multiple projects, I tend to leave the current usage of master for now, but if we face problems more frequently because of unstable master code then we may want to consider this switch.
Cheers, Stephen
[1]https://review.opendev.org/c/openstack/oslo.utils/+/900343 [2]https://review.opendev.org/c/openstack/releases/+/900382
Thank you, Takashi Kajinami irc: tkajinam