[all][ptl][release] Zed release critical issue with oslo.db 12.1.0
Hi,
With the latest oslo.db 12.1.0 release it seems multiple repositories are broken (see the corresponding requirements patch [1] for details, and probably there are more, that are not tested within requirements repository). The patch that introduces the break [2] "is necessary for oslo.db and openstack applications in general to be able to run on SQLAlchemy 2.0" according to patch author.
Since we are after the 3rd milestone of Zed, we have very limited time to figure out what to do. The 2 options are:
1) fix every broken repository to work with oslo.db 12.1.0 as far as we know this is quite impossible as for some repositories the already existing fixes took almost a cycle to be implemented and there are still undiscovered bugs around this. (we don't know if this is really true, or probably there are cases where it's easy to fix)
2) revert the breaking change [2] and do another release in oslo.db now without it (and add it back after Zed release & release oslo.db with it as soon as possible in Antelope cycle) as far as we understand this does not cause any issue for those who already adapted to oslo.db 12.1.0, so this should not be painful
Based on the above Release team is opting toward the 2nd option.
Please let us know your opinion!
[1] https://review.opendev.org/c/openstack/requirements/+/855153/ [2] https://review.opendev.org/c/openstack/oslo.db/+/804775
Thanks,
Előd
Le lun. 5 sept. 2022 à 16:37, Előd Illés elod.illes@est.tech a écrit :
Hi,
With the latest oslo.db 12.1.0 release it seems multiple repositories are broken (see the corresponding requirements patch [1] for details, and probably there are more, that are not tested within requirements repository). The patch that introduces the break [2] "is necessary for oslo.db and openstack applications in general to be able to run on SQLAlchemy 2.0" according to patch author.
Since we are after the 3rd milestone of Zed, we have very limited time to figure out what to do. The 2 options are:
- fix every broken repository to work with oslo.db 12.1.0 as far as we know this is quite impossible as for some repositories
the already existing fixes took almost a cycle to be implemented and there are still undiscovered bugs around this. (we don't know if this is really true, or probably there are cases where it's easy to fix)
- revert the breaking change [2] and do another release in oslo.db now
without it (and add it back after Zed release & release oslo.db with it as soon as possible in Antelope cycle) as far as we understand this does not cause any issue for those who already adapted to oslo.db 12.1.0, so this should not be painful
From a personal point of view, I'm on board with option 2, which closes the
race as soon as we revert and gives us more time to find the root cause and fix it correctly.
From a Nova/Placement PoV, they're not impacted, as we only raise warnings,
so feel free to pick any other project to discuss about option 1.
-Sylvain
Based on the above Release team is opting toward the 2nd option.
Please let us know your opinion!
[1] https://review.opendev.org/c/openstack/requirements/+/855153/ [2] https://review.opendev.org/c/openstack/oslo.db/+/804775
Thanks,
Előd
On Mon, 2022-09-05 at 16:19 +0200, Előd Illés wrote:
Hi, With the latest oslo.db 12.1.0 release it seems multiple repositories are broken (see the corresponding requirements patch [1] for details, and probably there are more, that are not tested within requirements repository). The patch that introduces the break [2] "is necessary for oslo.db and openstack applications in general to be able to run on SQLAlchemy 2.0" according to patch author. Since we are after the 3rd milestone of Zed, we have very limited time to figure out what to do. The 2 options are:
- fix every broken repository to work with oslo.db 12.1.0
as far as we know this is quite impossible as for some repositories the already existing fixes took almost a cycle to be implemented and there are still undiscovered bugs around this. (we don't know if this is really true, or probably there are cases where it's easy to fix)
I don't think this is True. To fully prepare for SQLAlchemy 2.0 you would need to do this work, yes, however the issue here is that oslo.db no longer *defaults* to enabling one of the SQLAlchemy features that's being removed in 2.0, autocommit, and not that it no longer supports it. There's nothing to stop the affected projects from manually enabling autocommit behavior and to prove this I just put together a patch for aodh to do just that [1]. Manually enabling this feature would be my preferred approach since it means we're kicking a slightly smaller can down the road and we'll have a better understanding of the projects that still need to do work for SQLAlchemy 2.0, but I don't know how likely it is that we can land all of these in the next week.
- revert the breaking change [2] and do another release in oslo.db now
without it (and add it back after Zed release & release oslo.db with it as soon as possible in Antelope cycle) as far as we understand this does not cause any issue for those who already adapted to oslo.db 12.1.0, so this should not be painful
This is all true, but it doesn't actually fix anything and delays the necessary work to get us moving in the right direction. I've been harping on about the fact that SQLAlchemy 2.0 is coming and will affect many projects for over a year now [2] so perhaps a small but firm nudge like this would be beneficial and get things rolling moving into Antelope?
Based on the above Release team is opting toward the 2nd option. Please let us know your opinion! [1] https://review.opendev.org/c/openstack/requirements/+/855153/ [2] https://review.opendev.org/c/openstack/oslo.db/+/804775 Thanks,
Előd
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/aodh/+/855962 [2] https://lists.openstack.org/pipermail/openstack-discuss/2021-August/024122.h...
Stephen Finucane wrote:
On Mon, 2022-09-05 at 16:19 +0200, Előd Illés wrote:
[...] The 2 options are:
- fix every broken repository to work with oslo.db 12.1.0
as far as we know this is quite impossible as for some repositories the already existing fixes took almost a cycle to be implemented and there are still undiscovered bugs around this. (we don't know if this is really true, or probably there are cases where it's easy to fix)
I don't think this is True. To fully prepare for SQLAlchemy 2.0 you would need to do this work, yes, however the issue here is that oslo.db no longer *defaults* to enabling one of the SQLAlchemy features that's being removed in 2.0, autocommit, and not that it no longer supports it. There's nothing to stop the affected projects from manually enabling autocommit behavior and to prove this I just put together a patch for aodh to do just that [1]. Manually enabling this feature would be my preferred approach since it means we're kicking a slightly smaller can down the road and we'll have a better understanding of the projects that still need to do work for SQLAlchemy 2.0, but I don't know how likely it is that we can land all of these in the next week.
I agree this is the preferable option... and if we had a couple extra weeks to make it happen I would pick it without hesitation...
Here I feel to make it happen in time for the Zed release we'll need to get someone to post all the autocommit option patches and potentially the TC stepping in to authorize the TaCT team to force-approve the ones that would not get attention from their core teams in time.
- revert the breaking change [2] and do another release in oslo.db now
without it (and add it back after Zed release & release oslo.db with it as soon as possible in Antelope cycle) as far as we understand this does not cause any issue for those who already adapted to oslo.db 12.1.0, so this should not be painful
This is all true, but it doesn't actually fix anything and delays the necessary work to get us moving in the right direction. I've been harping on about the fact that SQLAlchemy 2.0 is coming and will affect many projects for over a year now [2] so perhaps a small but firm nudge like this would be beneficial and get things rolling moving into Antelope?
I agree it's a step back without guarantee that things would go a lot better in the Antelope redo... but it is a more controlled path.
I'd say we should see if we can get critical attention on this in the next couple of days, and if not we can use option (2) as a plan B ?
On Mon, 2022-09-05 at 19:00 +0200, Thierry Carrez wrote:
Stephen Finucane wrote:
On Mon, 2022-09-05 at 16:19 +0200, Előd Illés wrote:
[...] The 2 options are:
- fix every broken repository to work with oslo.db 12.1.0
as far as we know this is quite impossible as for some repositories the already existing fixes took almost a cycle to be implemented and there are still undiscovered bugs around this. (we don't know if this is really true, or probably there are cases where it's easy to fix)
I don't think this is True. To fully prepare for SQLAlchemy 2.0 you would need to do this work, yes, however the issue here is that oslo.db no longer *defaults* to enabling one of the SQLAlchemy features that's being removed in 2.0, autocommit, and not that it no longer supports it. There's nothing to stop the affected projects from manually enabling autocommit behavior and to prove this I just put together a patch for aodh to do just that [1]. Manually enabling this feature would be my preferred approach since it means we're kicking a slightly smaller can down the road and we'll have a better understanding of the projects that still need to do work for SQLAlchemy 2.0, but I don't know how likely it is that we can land all of these in the next week.
I agree this is the preferable option... and if we had a couple extra weeks to make it happen I would pick it without hesitation...
Here I feel to make it happen in time for the Zed release we'll need to get someone to post all the autocommit option patches and potentially the TC stepping in to authorize the TaCT team to force-approve the ones that would not get attention from their core teams in time.
I did those yesterday.
Aodh: https://review.opendev.org/c/openstack/aodh/+/855962 Designate: https://review.opendev.org/c/openstack/designate/+/855965 Manila: https://review.opendev.org/c/openstack/manila/+/855967 Ironic: https://review.opendev.org/c/openstack/ironic/+/855969 Heat: https://review.opendev.org/c/openstack/heat/+/855970 Barbican: https://review.opendev.org/c/openstack/barbican/+/855972
The barbican one isn't correct and I need to respin it. The rest are working as expected though, as proven at [1].
Stephen
[1] https://review.opendev.org/c/openstack/requirements/+/855973/
- revert the breaking change [2] and do another release in oslo.db now
without it (and add it back after Zed release & release oslo.db with it as soon as possible in Antelope cycle) as far as we understand this does not cause any issue for those who already adapted to oslo.db 12.1.0, so this should not be painful
This is all true, but it doesn't actually fix anything and delays the necessary work to get us moving in the right direction. I've been harping on about the fact that SQLAlchemy 2.0 is coming and will affect many projects for over a year now [2] so perhaps a small but firm nudge like this would be beneficial and get things rolling moving into Antelope?
I agree it's a step back without guarantee that things would go a lot better in the Antelope redo... but it is a more controlled path.
I'd say we should see if we can get critical attention on this in the next couple of days, and if not we can use option (2) as a plan B ?
On Tue, 2022-09-06 at 11:22 +0100, Stephen Finucane wrote:
On Mon, 2022-09-05 at 19:00 +0200, Thierry Carrez wrote:
Stephen Finucane wrote:
On Mon, 2022-09-05 at 16:19 +0200, Előd Illés wrote:
[...] The 2 options are:
- fix every broken repository to work with oslo.db 12.1.0
as far as we know this is quite impossible as for some repositories the already existing fixes took almost a cycle to be implemented and there are still undiscovered bugs around this. (we don't know if this is really true, or probably there are cases where it's easy to fix)
I don't think this is True. To fully prepare for SQLAlchemy 2.0 you would need to do this work, yes, however the issue here is that oslo.db no longer *defaults* to enabling one of the SQLAlchemy features that's being removed in 2.0, autocommit, and not that it no longer supports it. There's nothing to stop the affected projects from manually enabling autocommit behavior and to prove this I just put together a patch for aodh to do just that [1]. Manually enabling this feature would be my preferred approach since it means we're kicking a slightly smaller can down the road and we'll have a better understanding of the projects that still need to do work for SQLAlchemy 2.0, but I don't know how likely it is that we can land all of these in the next week.
I agree this is the preferable option... and if we had a couple extra weeks to make it happen I would pick it without hesitation...
Here I feel to make it happen in time for the Zed release we'll need to get someone to post all the autocommit option patches and potentially the TC stepping in to authorize the TaCT team to force-approve the ones that would not get attention from their core teams in time.
I did those yesterday.
Aodh: https://review.opendev.org/c/openstack/aodh/+/855962 Designate: https://review.opendev.org/c/openstack/designate/+/855965 Manila: https://review.opendev.org/c/openstack/manila/+/855967 Ironic: https://review.opendev.org/c/openstack/ironic/+/855969 Heat: https://review.opendev.org/c/openstack/heat/+/855970 Barbican: https://review.opendev.org/c/openstack/barbican/+/855972
The barbican one isn't correct and I need to respin it. The rest are working as expected though, as proven at [1].
Actually, the barbican change isn't working because it isn't needed: Barbican (helpfully) doesn't rely on autocommit. If we can get those other 5 patches merged + the placement patch from gibi, we should have no issues with oslo.db 12.1.0. They're not long-term fixes and more bandaid solutions but they'll tide us over until these projects can rework their DB layers to be SQLAlchemy 2.0 compatible in Antelope.
Stephen
Stephen
[1] https://review.opendev.org/c/openstack/requirements/+/855973/
- revert the breaking change [2] and do another release in oslo.db now
without it (and add it back after Zed release & release oslo.db with it as soon as possible in Antelope cycle) as far as we understand this does not cause any issue for those who already adapted to oslo.db 12.1.0, so this should not be painful
This is all true, but it doesn't actually fix anything and delays the necessary work to get us moving in the right direction. I've been harping on about the fact that SQLAlchemy 2.0 is coming and will affect many projects for over a year now [2] so perhaps a small but firm nudge like this would be beneficial and get things rolling moving into Antelope?
I agree it's a step back without guarantee that things would go a lot better in the Antelope redo... but it is a more controlled path.
I'd say we should see if we can get critical attention on this in the next couple of days, and if not we can use option (2) as a plan B ?
Thanks Stephen for working on this! If those patches get merged *and* nothing else is needed, then Option 1 seems to be the better solution.
Anyone aware of other place where a fix is needed, or have uncertainties about Option 1?
Thanks,
Előd
On 2022. 09. 06. 12:38, Stephen Finucane wrote:
On Tue, 2022-09-06 at 11:22 +0100, Stephen Finucane wrote:
On Mon, 2022-09-05 at 19:00 +0200, Thierry Carrez wrote:
Stephen Finucane wrote:
On Mon, 2022-09-05 at 16:19 +0200, Előd Illés wrote:
[...] The 2 options are:
- fix every broken repository to work with oslo.db 12.1.0
as far as we know this is quite impossible as for some repositories the already existing fixes took almost a cycle to be implemented and there are still undiscovered bugs around this. (we don't know if this is really true, or probably there are cases where it's easy to fix)
I don't think this is True. To fully prepare for SQLAlchemy 2.0 you would need to do this work, yes, however the issue here is that oslo.db no longer *defaults* to enabling one of the SQLAlchemy features that's being removed in 2.0, autocommit, and not that it no longer supports it. There's nothing to stop the affected projects from manually enabling autocommit behavior and to prove this I just put together a patch for aodh to do just that [1]. Manually enabling this feature would be my preferred approach since it means we're kicking a slightly smaller can down the road and we'll have a better understanding of the projects that still need to do work for SQLAlchemy 2.0, but I don't know how likely it is that we can land all of these in the next week.
I agree this is the preferable option... and if we had a couple extra weeks to make it happen I would pick it without hesitation...
Here I feel to make it happen in time for the Zed release we'll need to get someone to post all the autocommit option patches and potentially the TC stepping in to authorize the TaCT team to force-approve the ones that would not get attention from their core teams in time.
I did those yesterday.
Aodh: https://review.opendev.org/c/openstack/aodh/+/855962 Designate: https://review.opendev.org/c/openstack/designate/+/855965 Manila: https://review.opendev.org/c/openstack/manila/+/855967 Ironic: https://review.opendev.org/c/openstack/ironic/+/855969 Heat: https://review.opendev.org/c/openstack/heat/+/855970 Barbican: https://review.opendev.org/c/openstack/barbican/+/855972
The barbican one isn't correct and I need to respin it. The rest are working as expected though, as proven at [1].
Actually, the barbican change isn't working because it isn't needed: Barbican (helpfully) doesn't rely on autocommit. If we can get those other 5 patches merged + the placement patch from gibi, we should have no issues with oslo.db 12.1.0. They're not long-term fixes and more bandaid solutions but they'll tide us over until these projects can rework their DB layers to be SQLAlchemy 2.0 compatible in Antelope.
Stephen
Stephen
[1] https://review.opendev.org/c/openstack/requirements/+/855973/
- revert the breaking change [2] and do another release in oslo.db now
without it (and add it back after Zed release & release oslo.db with it as soon as possible in Antelope cycle) as far as we understand this does not cause any issue for those who already adapted to oslo.db 12.1.0, so this should not be painful
This is all true, but it doesn't actually fix anything and delays the necessary work to get us moving in the right direction. I've been harping on about the fact that SQLAlchemy 2.0 is coming and will affect many projects for over a year now [2] so perhaps a small but firm nudge like this would be beneficial and get things rolling moving into Antelope?
I agree it's a step back without guarantee that things would go a lot better in the Antelope redo... but it is a more controlled path.
I'd say we should see if we can get critical attention on this in the next couple of days, and if not we can use option (2) as a plan B ?
Hi, Thanks for working on this, we discussed this topic yesterday on Neutron CI meeting (see [0]). For the flapping oslo-master job in Neutron we have a fix thanks to Miguel, and we are waiting for the experimental results (we plan a few runs to see if it is stable enough). So I hope we are close to saying a definite answer from Neutron perspective.
Lajos
[0]: https://meetings.opendev.org/meetings/neutron_ci/2022/neutron_ci.2022-09-06-...
Thierry Carrez thierry@openstack.org ezt írta (időpont: 2022. szept. 6., K, 17:56):
Előd Illés wrote:
Thanks Stephen for working on this! If those patches get merged *and* nothing else is needed, then Option 1 seems to be the better solution.
It looks like they are all getting good review traction, so this is looking good.
-- Thierry Carrez (ttx)
On Tue, 2022-09-06 at 17:31 +0200, Thierry Carrez wrote:
Előd Illés wrote:
Thanks Stephen for working on this! If those patches get merged *and* nothing else is needed, then Option 1 seems to be the better solution.
It looks like they are all getting good review traction, so this is looking good.
Yeah, these are all merged now. I just rechecked the requirements patch. That should be green soon.
Stephen
On Wed, 2022-09-07 at 13:13 +0100, Stephen Finucane wrote:
On Tue, 2022-09-06 at 17:31 +0200, Thierry Carrez wrote:
Előd Illés wrote:
Thanks Stephen for working on this! If those patches get merged *and* nothing else is needed, then Option 1 seems to be the better solution.
It looks like they are all getting good review traction, so this is looking good.
Yeah, these are all merged now. I just rechecked the requirements patch. That should be green soon.
This is now green.
https://review.opendev.org/c/openstack/requirements/+/855153
Stephen
Hi, For Neutron we still have 2 hanging patches to fix failing jobs: https://review.opendev.org/c/openstack/neutron/+/856444 https://review.opendev.org/c/openstack/neutron/+/855703
Lajos Katona (lajoskatona)
Stephen Finucane stephenfin@redhat.com ezt írta (időpont: 2022. szept. 7., Sze, 17:00):
On Wed, 2022-09-07 at 13:13 +0100, Stephen Finucane wrote:
On Tue, 2022-09-06 at 17:31 +0200, Thierry Carrez wrote:
Előd Illés wrote:
Thanks Stephen for working on this! If those patches get merged
*and*
nothing else is needed, then Option 1 seems to be the better
solution.
It looks like they are all getting good review traction, so this is looking good.
Yeah, these are all merged now. I just rechecked the requirements patch.
That
should be green soon.
This is now green.
https://review.opendev.org/c/openstack/requirements/+/855153
Stephen
On Tue, 6 Sept 2022 at 12:40, Stephen Finucane stephenfin@redhat.com wrote:
I did those yesterday.
Aodh: https://review.opendev.org/c/openstack/aodh/+/855962 Designate: https://review.opendev.org/c/openstack/designate/+/855965 Manila: https://review.opendev.org/c/openstack/manila/+/855967 Ironic: https://review.opendev.org/c/openstack/ironic/+/855969 Heat: https://review.opendev.org/c/openstack/heat/+/855970 Barbican: https://review.opendev.org/c/openstack/barbican/+/855972
The barbican one isn't correct and I need to respin it. The rest are working as expected though, as proven at [1].
Stephen
[1] https://review.opendev.org/c/openstack/requirements/+/855973/
Hi,
Radosław Piliszek (yoctozepto) and I have discovered that Blazar is also broken by the release of oslo.db 12.1.0: CI jobs are failing CI jobs in [1]. I will try to fix it like Stephen did for other projects, but I would like to flag that it is possible other projects are impacted: in particular we should check any smaller project that doesn't run CI jobs regularly.
This reminds me of the breakage we had at the end of the last cycle with oslo.context 4.0.0 [2].
Cheers, Pierre Riteau (priteau)
[1] https://review.opendev.org/c/openstack/blazar/+/856527 [2] https://lists.openstack.org/pipermail/openstack-discuss/2022-March/027864.ht...
On Fri, 9 Sept 2022 at 16:14, Pierre Riteau pierre@stackhpc.com wrote:
On Tue, 6 Sept 2022 at 12:40, Stephen Finucane stephenfin@redhat.com wrote:
I did those yesterday.
Aodh: https://review.opendev.org/c/openstack/aodh/+/855962 Designate: https://review.opendev.org/c/openstack/designate/+/855965 Manila: https://review.opendev.org/c/openstack/manila/+/855967 Ironic: https://review.opendev.org/c/openstack/ironic/+/855969 Heat: https://review.opendev.org/c/openstack/heat/+/855970 Barbican: https://review.opendev.org/c/openstack/barbican/+/855972
The barbican one isn't correct and I need to respin it. The rest are working as expected though, as proven at [1].
Stephen
[1] https://review.opendev.org/c/openstack/requirements/+/855973/
Hi,
Radosław Piliszek (yoctozepto) and I have discovered that Blazar is also broken by the release of oslo.db 12.1.0: CI jobs are failing CI jobs in [1]. I will try to fix it like Stephen did for other projects, but I would like to flag that it is possible other projects are impacted: in particular we should check any smaller project that doesn't run CI jobs regularly.
This reminds me of the breakage we had at the end of the last cycle with oslo.context 4.0.0 [2].
Cheers, Pierre Riteau (priteau)
[1] https://review.opendev.org/c/openstack/blazar/+/856527 [2] https://lists.openstack.org/pipermail/openstack-discuss/2022-March/027864.ht...
Thank you Stephen for your code, this made it an easy fix [1].
Following the tagging of RC1 for CloudKitty, I have discovered that it is also impacted by this issue. I have prepared a patch [1]. We will need to tag RC2.
[1] https://review.opendev.org/c/openstack/cloudkitty/+/857579
On Fri, 9 Sept 2022 at 16:34, Pierre Riteau pierre@stackhpc.com wrote:
On Fri, 9 Sept 2022 at 16:14, Pierre Riteau pierre@stackhpc.com wrote:
On Tue, 6 Sept 2022 at 12:40, Stephen Finucane stephenfin@redhat.com wrote:
I did those yesterday.
Aodh: https://review.opendev.org/c/openstack/aodh/+/855962 Designate: https://review.opendev.org/c/openstack/designate/+/855965 Manila: https://review.opendev.org/c/openstack/manila/+/855967 Ironic: https://review.opendev.org/c/openstack/ironic/+/855969 Heat: https://review.opendev.org/c/openstack/heat/+/855970 Barbican: https://review.opendev.org/c/openstack/barbican/+/855972
The barbican one isn't correct and I need to respin it. The rest are working as expected though, as proven at [1].
Stephen
[1] https://review.opendev.org/c/openstack/requirements/+/855973/
Hi,
Radosław Piliszek (yoctozepto) and I have discovered that Blazar is also broken by the release of oslo.db 12.1.0: CI jobs are failing CI jobs in [1]. I will try to fix it like Stephen did for other projects, but I would like to flag that it is possible other projects are impacted: in particular we should check any smaller project that doesn't run CI jobs regularly.
This reminds me of the breakage we had at the end of the last cycle with oslo.context 4.0.0 [2].
Cheers, Pierre Riteau (priteau)
[1] https://review.opendev.org/c/openstack/blazar/+/856527 [2] https://lists.openstack.org/pipermail/openstack-discuss/2022-March/027864.ht...
Thank you Stephen for your code, this made it an easy fix [1].
On Fri, Sep 9, 2022 at 4:27 PM Pierre Riteau pierre@stackhpc.com wrote:
On Tue, 6 Sept 2022 at 12:40, Stephen Finucane stephenfin@redhat.com wrote:
I did those yesterday.
Aodh: https://review.opendev.org/c/openstack/aodh/+/855962 Designate: https://review.opendev.org/c/openstack/designate/+/855965 Manila: https://review.opendev.org/c/openstack/manila/+/855967 Ironic: https://review.opendev.org/c/openstack/ironic/+/855969 Heat: https://review.opendev.org/c/openstack/heat/+/855970 Barbican: https://review.opendev.org/c/openstack/barbican/+/855972
The barbican one isn't correct and I need to respin it. The rest are working as expected though, as proven at [1].
Stephen
[1] https://review.opendev.org/c/openstack/requirements/+/855973/
Hi,
Radosław Piliszek (yoctozepto) and I have discovered that Blazar is also broken by the release of oslo.db 12.1.0: CI jobs are failing CI jobs in [1]. I will try to fix it like Stephen did for other projects, but I would like to flag that it is possible other projects are impacted: in particular we should check any smaller project that doesn't run CI jobs regularly.
According to some functional jobs run in RDO, watcher and ec2-api projects are also affected.
I've sent https://review.opendev.org/c/openstack/ec2-api/+/857880 but the gate seems blocked by some other issue.
Regards,
Alfredo
This reminds me of the breakage we had at the end of the last cycle with oslo.context 4.0.0 [2].
Cheers, Pierre Riteau (priteau)
[1] https://review.opendev.org/c/openstack/blazar/+/856527 [2] https://lists.openstack.org/pipermail/openstack-discuss/2022-March/027864.ht...
On Thu, Sep 15, 2022 at 3:55 PM Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Fri, Sep 9, 2022 at 4:27 PM Pierre Riteau pierre@stackhpc.com wrote:
On Tue, 6 Sept 2022 at 12:40, Stephen Finucane stephenfin@redhat.com wrote:
I did those yesterday.
Aodh: https://review.opendev.org/c/openstack/aodh/+/855962 Designate: https://review.opendev.org/c/openstack/designate/+/855965 Manila: https://review.opendev.org/c/openstack/manila/+/855967 Ironic: https://review.opendev.org/c/openstack/ironic/+/855969 Heat: https://review.opendev.org/c/openstack/heat/+/855970 Barbican: https://review.opendev.org/c/openstack/barbican/+/855972
The barbican one isn't correct and I need to respin it. The rest are working as expected though, as proven at [1].
Stephen
[1] https://review.opendev.org/c/openstack/requirements/+/855973/
Hi,
Radosław Piliszek (yoctozepto) and I have discovered that Blazar is also broken by the release of oslo.db 12.1.0: CI jobs are failing CI jobs in [1]. I will try to fix it like Stephen did for other projects, but I would like to flag that it is possible other projects are impacted: in particular we should check any smaller project that doesn't run CI jobs regularly.
According to some functional jobs run in RDO, watcher and ec2-api projects are also affected.
I've sent https://review.opendev.org/c/openstack/ec2-api/+/857880 but the gate seems blocked by some other issue.
Also for watcher https://review.opendev.org/c/openstack/watcher/+/857887
Regards,
Alfredo
This reminds me of the breakage we had at the end of the last cycle with oslo.context 4.0.0 [2].
Cheers, Pierre Riteau (priteau)
[1] https://review.opendev.org/c/openstack/blazar/+/856527 [2] https://lists.openstack.org/pipermail/openstack-discuss/2022-March/027864.ht...
Hi,
Dnia poniedziałek, 5 września 2022 18:14:27 CEST Stephen Finucane pisze:
On Mon, 2022-09-05 at 16:19 +0200, Előd Illés wrote:
Hi, With the latest oslo.db 12.1.0 release it seems multiple repositories are broken (see the corresponding requirements patch [1] for details, and probably there are more, that are not tested within requirements repository). The patch that introduces the break [2] "is necessary for oslo.db and openstack applications in general to be able to run on SQLAlchemy 2.0" according to patch author. Since we are after the 3rd milestone of Zed, we have very limited time to figure out what to do. The 2 options are:
- fix every broken repository to work with oslo.db 12.1.0 as far as we know this is quite impossible as for some repositories the
already existing fixes took almost a cycle to be implemented and there are still undiscovered bugs around this. (we don't know if this is really true, or probably there are cases where it's easy to fix)
I don't think this is True. To fully prepare for SQLAlchemy 2.0 you would need to do this work, yes, however the issue here is that oslo.db no longer *defaults* to enabling one of the SQLAlchemy features that's being removed in 2.0, autocommit, and not that it no longer supports it. There's nothing to stop the affected projects from manually enabling autocommit behavior and to prove this I just put together a patch for aodh to do just that [1]. Manually enabling this feature would be my preferred approach since it means we're kicking a slightly smaller can down the road and we'll have a better understanding of the projects that still need to do work for SQLAlchemy 2.0, but I don't know how likely it is that we can land all of these in the next week.
- revert the breaking change [2] and do another release in oslo.db now
without it (and add it back after Zed release & release oslo.db with it as soon as possible in Antelope cycle) as far as we understand this does not cause any issue for those who already adapted to oslo.db 12.1.0, so this should not be painful
This is all true, but it doesn't actually fix anything and delays the necessary work to get us moving in the right direction. I've been harping on about the fact that SQLAlchemy 2.0 is coming and will affect many projects for over a year now [2] so perhaps a small but firm nudge like this would be beneficial and get things rolling moving into Antelope?
I would vote for reverting it now as we are close to the Zed release deadline and then reverting this revert very early in the Antelope cycle to give projects chance to adopt to the change or enable autocommit feature on their own and start slower adoption later.
Based on the above Release team is opting toward the 2nd option. Please let us know your opinion! [1] https://review.opendev.org/c/openstack/requirements/+/855153/ [2] https://review.opendev.org/c/openstack/oslo.db/+/804775 Thanks,
Előd
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/aodh/+/855962 [2] https://lists.openstack.org/pipermail/openstack-discuss/2021-August/024122.h...
Hi, Revert sounds the rational solution, even if it is a hard decision as a lot of people/projects already put a lot of efforts to work on it. As I see if we revert it now and rever the revert and do a quick release of oslo.db in 2023.1 (aka Antelope) we can force projects to act, and give time to TC to force-approve it for understaffed projects.
Lajos
Slawek Kaplonski skaplons@redhat.com ezt írta (időpont: 2022. szept. 6., K, 9:35):
Hi,
Dnia poniedziałek, 5 września 2022 18:14:27 CEST Stephen Finucane pisze:
On Mon, 2022-09-05 at 16:19 +0200, Előd Illés wrote:
Hi, With the latest oslo.db 12.1.0 release it seems multiple repositories
are
broken (see the corresponding requirements patch [1] for details, and
probably
there are more, that are not tested within requirements repository).
The patch
that introduces the break [2] "is necessary for oslo.db and openstack applications in general to be able to run on SQLAlchemy 2.0" according
to
patch author. Since we are after the 3rd milestone of Zed, we have very limited time
to
figure out what to do. The 2 options are:
- fix every broken repository to work with oslo.db 12.1.0 as far as we know this is quite impossible as for some
repositories the
already existing fixes took almost a cycle to be implemented and there
are
still undiscovered bugs around this. (we don't know if this is really
true, or
probably there are cases where it's easy to fix)
I don't think this is True. To fully prepare for SQLAlchemy 2.0 you
would need
to do this work, yes, however the issue here is that oslo.db no longer *defaults* to enabling one of the SQLAlchemy features that's being
removed in
2.0, autocommit, and not that it no longer supports it. There's nothing
to stop
the affected projects from manually enabling autocommit behavior and to
prove
this I just put together a patch for aodh to do just that [1]. Manually
enabling
this feature would be my preferred approach since it means we're kicking
a
slightly smaller can down the road and we'll have a better understanding
of the
projects that still need to do work for SQLAlchemy 2.0, but I don't know
how
likely it is that we can land all of these in the next week.
- revert the breaking change [2] and do another release in oslo.db now
without it (and add it back after Zed release & release oslo.db with
it as
soon as possible in Antelope cycle) as far as we understand this does not cause any issue for those
who
already adapted to oslo.db 12.1.0, so this should not be painful
This is all true, but it doesn't actually fix anything and delays the
necessary
work to get us moving in the right direction. I've been harping on about
the
fact that SQLAlchemy 2.0 is coming and will affect many projects for
over a year
now [2] so perhaps a small but firm nudge like this would be beneficial
and get
things rolling moving into Antelope?
I would vote for reverting it now as we are close to the Zed release deadline and then reverting this revert very early in the Antelope cycle to give projects chance to adopt to the change or enable autocommit feature on their own and start slower adoption later.
Based on the above Release team is opting toward the 2nd option. Please let us know your opinion! [1] https://review.opendev.org/c/openstack/requirements/+/855153/ [2] https://review.opendev.org/c/openstack/oslo.db/+/804775 Thanks,
Előd
Cheers, Stephen
[1] https://review.opendev.org/c/openstack/aodh/+/855962 [2]
https://lists.openstack.org/pipermail/openstack-discuss/2021-August/024122.h...
-- Slawek Kaplonski Principal Software Engineer Red Hat
participants (8)
-
Alfredo Moralejo Alonso
-
Előd Illés
-
Lajos Katona
-
Pierre Riteau
-
Slawek Kaplonski
-
Stephen Finucane
-
Sylvain Bauza
-
Thierry Carrez