On 11/9/23 22:30, Stephen Finucane wrote:

Just FYI, for projects that have sqlalchemy-tips jobs that have been failing
recently, they should now be passing again. The root cause was neither
sqlalchemy nor alembic but rather oslo.utils [1]. It was fixed in oslo.utils
6.3.0 which was released [2] roughly an hour ago. A recheck should be sufficient
to clear errors and as these tips jobs only run on master there should be no
issues with stable branches.

It later turned out unit tests of nova and ironic are broken by the oslo.utils fix,
which is preventing us from bumping oslo.utils version in u-c[1].

I've reported a bug[2] and submitted fixes to these projects[3][4]. It'd be nice if
someone from nova/ironic core team can pay attention to these patches.

# I had to disable the sqlalchemy-2x jobs to merge these unit test fixes,
# and such change in zuul config triggered full set of tests... so it may take
# a while until all triggered jobs complete.

 [1] https://review.opendev.org/c/openstack/requirements/+/900517
 [2] https://bugs.launchpad.net/nova/+bug/2043116
 [3] https://review.opendev.org/c/openstack/nova/+/900532
 [4] https://review.opendev.org/c/openstack/ironic/+/900533

These unit test fixes were merged, and the requirement bump patch was just merged.
So all jobs using u-c should start consuming oslo.utils 6.3.0 which contains the fix for
the issue.

Please check status of sqlalchemy-2x job in your project, and revert the job definition
ASAP if you have made the job non-voting.

A further point to add: while I appreciate that broken gates are never a good
thing, I would be careful making these jobs non-voting. I've already seen a few
instances where SQLAlchemy 2.0 support has regressed and I for one have no
desire to re-herd cats as we push to get SQLAlchemy uncapped in the coming
weeks. If this happens again then we can re-examine things, naturally.

I agree that making a job non-voting is not ideal, and for sqlalchemy-2x we have
to prioritize keeping it because we are committed to use that version in a near future.
However I'm wondering if we have to keep the job with the latest code voting in
normal queue or we can replace it by a job with the latest release of sqlalchemy.

Because we may require updates in multiple projects, I tend to leave the current
usage of master for now, but if we face problems more frequently because of
unstable master code then we may want to consider this switch.

Cheers,
Stephen

[1] https://review.opendev.org/c/openstack/oslo.utils/+/900343
[2] https://review.opendev.org/c/openstack/releases/+/900382

Thank you,

Takashi Kajinami
irc: tkajinam