[all] Failing sqlalchemy-tips jobs should now be passing again
Just FYI, for projects that have sqlalchemy-tips jobs that have been failing recently, they should now be passing again. The root cause was neither sqlalchemy nor alembic but rather oslo.utils [1]. It was fixed in oslo.utils 6.3.0 which was released [2] roughly an hour ago. A recheck should be sufficient to clear errors and as these tips jobs only run on master there should be no issues with stable branches. A further point to add: while I appreciate that broken gates are never a good thing, I would be careful making these jobs non-voting. I've already seen a few instances where SQLAlchemy 2.0 support has regressed and I for one have no desire to re-herd cats as we push to get SQLAlchemy uncapped in the coming weeks. If this happens again then we can re-examine things, naturally. Cheers, Stephen [1] https://review.opendev.org/c/openstack/oslo.utils/+/900343 [2] https://review.opendev.org/c/openstack/releases/+/900382
On 11/9/23 22:30, Stephen Finucane wrote:
Just FYI, for projects that have sqlalchemy-tips jobs that have been failing recently, they should now be passing again. The root cause was neither sqlalchemy nor alembic but rather oslo.utils [1]. It was fixed in oslo.utils 6.3.0 which was released [2] roughly an hour ago. A recheck should be sufficient to clear errors and as these tips jobs only run on master there should be no issues with stable branches.
It later turned out unit tests of nova and ironic are broken by the oslo.utils fix, which is preventing us from bumping oslo.utils version in u-c[1]. I've reported a bug[2] and submitted fixes to these projects[3][4]. It'd be nice if someone from nova/ironic core team can pay attention to these patches. # I had to disable the sqlalchemy-2x jobs to merge these unit test fixes, # and such change in zuul config triggered full set of tests... so it may take # a while until all triggered jobs complete. [1] https://review.opendev.org/c/openstack/requirements/+/900517 [2] https://bugs.launchpad.net/nova/+bug/2043116 [3] https://review.opendev.org/c/openstack/nova/+/900532 [4] https://review.opendev.org/c/openstack/ironic/+/900533
A further point to add: while I appreciate that broken gates are never a good thing, I would be careful making these jobs non-voting. I've already seen a few instances where SQLAlchemy 2.0 support has regressed and I for one have no desire to re-herd cats as we push to get SQLAlchemy uncapped in the coming weeks. If this happens again then we can re-examine things, naturally.
I agree that making a job non-voting is not ideal, and for sqlalchemy-2x we have to prioritize keeping it because we are committed to use that version in a near future. However I'm wondering if we have to keep the job with the latest code voting in normal queue or we can replace it by a job with the latest release of sqlalchemy. Because we may require updates in multiple projects, I tend to leave the current usage of master for now, but if we face problems more frequently because of unstable master code then we may want to consider this switch.
Cheers, Stephen
[1]https://review.opendev.org/c/openstack/oslo.utils/+/900343 [2]https://review.opendev.org/c/openstack/releases/+/900382
Thank you, Takashi Kajinami irc: tkajinam
On 11/10/23 01:02, Takashi Kajinami wrote:
On 11/9/23 22:30, Stephen Finucane wrote:
Just FYI, for projects that have sqlalchemy-tips jobs that have been failing recently, they should now be passing again. The root cause was neither sqlalchemy nor alembic but rather oslo.utils [1]. It was fixed in oslo.utils 6.3.0 which was released [2] roughly an hour ago. A recheck should be sufficient to clear errors and as these tips jobs only run on master there should be no issues with stable branches.
It later turned out unit tests of nova and ironic are broken by the oslo.utils fix, which is preventing us from bumping oslo.utils version in u-c[1].
I've reported a bug[2] and submitted fixes to these projects[3][4]. It'd be nice if someone from nova/ironic core team can pay attention to these patches.
# I had to disable the sqlalchemy-2x jobs to merge these unit test fixes, # and such change in zuul config triggered full set of tests... so it may take # a while until all triggered jobs complete.
[1] https://review.opendev.org/c/openstack/requirements/+/900517 [2] https://bugs.launchpad.net/nova/+bug/2043116 [3] https://review.opendev.org/c/openstack/nova/+/900532 [4] https://review.opendev.org/c/openstack/ironic/+/900533
These unit test fixes were merged, and the requirement bump patch was just merged. So all jobs using u-c should start consuming oslo.utils 6.3.0 which contains the fix for the issue. Please check status of sqlalchemy-2x job in your project, and revert the job definition ASAP if you have made the job non-voting.
A further point to add: while I appreciate that broken gates are never a good thing, I would be careful making these jobs non-voting. I've already seen a few instances where SQLAlchemy 2.0 support has regressed and I for one have no desire to re-herd cats as we push to get SQLAlchemy uncapped in the coming weeks. If this happens again then we can re-examine things, naturally.
I agree that making a job non-voting is not ideal, and for sqlalchemy-2x we have to prioritize keeping it because we are committed to use that version in a near future. However I'm wondering if we have to keep the job with the latest code voting in normal queue or we can replace it by a job with the latest release of sqlalchemy.
Because we may require updates in multiple projects, I tend to leave the current usage of master for now, but if we face problems more frequently because of unstable master code then we may want to consider this switch.
Cheers, Stephen
[1]https://review.opendev.org/c/openstack/oslo.utils/+/900343 [2]https://review.opendev.org/c/openstack/releases/+/900382
Thank you,
Takashi Kajinami irc: tkajinam
participants (2)
-
Stephen Finucane
-
Takashi Kajinami