[oslo][kolla][requirements][release][infra] Hit by an old, fixed bug
Hi Folks, as the subject goes, my installation has been hit by an old bug: https://bugs.launchpad.net/oslo.messaging/+bug/1828841 (bug details not important, linked here for background) I am using Stein, deployed with recent Kolla-built source-based images (with only slight modifications compared to vanilla ones). Kolla's procedure for building source-based images considers upper constraints, which, unfortunately, turned out to be lagging behind a few releases w.r.t. oslo.messaging at least. The fix was in 9.7.0 released on May 21, u-c still point to 9.5.0 from Feb 26 and the latest of Stein is 9.8.0 from Jul 18. It seems oslo.messaging is missing from the automatic updates that bot proposes: https://review.opendev.org/#/q/owner:%22OpenStack+Proposal+Bot%22+project:op... Per: https://opendev.org/openstack/releases/src/branch/master/doc/source/referenc... this upper-constraint proposal should be happening for all releases. I would be glad if someone investigated why it happens(/ed) and audited whether other OpenStack projects don't need updating as well to avoid running on old deps when new are awaiting for months. :-) Please note this might apply to other branches as well. PS: for some reason oslo.messaging Stein release notes ( https://docs.openstack.org/releasenotes/oslo.messaging/stein.html ) are stuck at 9.5.0 as well, this could be right (I did not inspect the sources) but I am adding this in PS so you have more things to correlate if they need be. -yoctozepto
On Sun, Dec 29, 2019 at 09:41:45PM +0100, Radosław Piliszek wrote:
Hi Folks,
as the subject goes, my installation has been hit by an old bug: https://bugs.launchpad.net/oslo.messaging/+bug/1828841 (bug details not important, linked here for background)
I am using Stein, deployed with recent Kolla-built source-based images (with only slight modifications compared to vanilla ones). Kolla's procedure for building source-based images considers upper constraints, which, unfortunately, turned out to be lagging behind a few releases w.r.t. oslo.messaging at least. The fix was in 9.7.0 released on May 21, u-c still point to 9.5.0 from Feb 26 and the latest of Stein is 9.8.0 from Jul 18.
It seems oslo.messaging is missing from the automatic updates that bot proposes: https://review.opendev.org/#/q/owner:%22OpenStack+Proposal+Bot%22+project:op...
Per: https://opendev.org/openstack/releases/src/branch/master/doc/source/referenc... this upper-constraint proposal should be happening for all releases.
This is normal and what is expected. Requirements are only updated for the branch in which those releases happen. So if there is a release of oslo.messaging for stable/train, only the stable/train upper constraints are updated for that new release. The stable/stein branch will not be affected because that shows what the tested upper constraints were for that branch. The last stable/stein release for oslo.messaging was 9.5.0: https://opendev.org/openstack/releases/src/branch/master/deliverables/stein/... And 9.5.0 is what is set in the stable/stein upper-constraints: https://opendev.org/openstack/requirements/src/branch/stable/stein/upper-con... To get that raised, whatever necessary bugfixes that are required in oslo.messaging would need to be backported per-cycle until stable/stein (as in, if it was in current master, it would need to be backported and merged to stable/train first, then stable/stein), and once merged a stable release would need to be proposed for that branch's version of the library. Once that stable release is done, that will propose the update to the upper constraint for the given branch.
I would be glad if someone investigated why it happens(/ed) and audited whether other OpenStack projects don't need updating as well to avoid running on old deps when new are awaiting for months. :-) Please note this might apply to other branches as well.
PS: for some reason oslo.messaging Stein release notes ( https://docs.openstack.org/releasenotes/oslo.messaging/stein.html ) are stuck at 9.5.0 as well, this could be right (I did not inspect the sources) but I am adding this in PS so you have more things to correlate if they need be.
Again, as expected. The last stable/stein release was 9.5.0, so that is correct that the release notes for stein only show up to that point.
Thanks, Sean! I knew I was missing something really basic! I was under the impression that 9.x is Stein, like it happens with main projects (major=branch). I could not find any doc explaining oslo.messaging versioning, perhaps Oslo could release 9.5.1 off the stein branch? The issue remains that, even though oslo backports bugfixes into their stable branches, kolla (and very possibly other deployment solutions) no longer benefit from them. -yoctozepto pon., 30 gru 2019 o 16:01 Sean McGinnis <sean.mcginnis@gmx.com> napisał(a):
On Sun, Dec 29, 2019 at 09:41:45PM +0100, Radosław Piliszek wrote:
Hi Folks,
as the subject goes, my installation has been hit by an old bug: https://bugs.launchpad.net/oslo.messaging/+bug/1828841 (bug details not important, linked here for background)
I am using Stein, deployed with recent Kolla-built source-based images (with only slight modifications compared to vanilla ones). Kolla's procedure for building source-based images considers upper constraints, which, unfortunately, turned out to be lagging behind a few releases w.r.t. oslo.messaging at least. The fix was in 9.7.0 released on May 21, u-c still point to 9.5.0 from Feb 26 and the latest of Stein is 9.8.0 from Jul 18.
It seems oslo.messaging is missing from the automatic updates that bot proposes: https://review.opendev.org/#/q/owner:%22OpenStack+Proposal+Bot%22+project:op...
Per: https://opendev.org/openstack/releases/src/branch/master/doc/source/referenc... this upper-constraint proposal should be happening for all releases.
This is normal and what is expected.
Requirements are only updated for the branch in which those releases happen. So if there is a release of oslo.messaging for stable/train, only the stable/train upper constraints are updated for that new release. The stable/stein branch will not be affected because that shows what the tested upper constraints were for that branch.
The last stable/stein release for oslo.messaging was 9.5.0:
https://opendev.org/openstack/releases/src/branch/master/deliverables/stein/...
And 9.5.0 is what is set in the stable/stein upper-constraints:
https://opendev.org/openstack/requirements/src/branch/stable/stein/upper-con...
To get that raised, whatever necessary bugfixes that are required in oslo.messaging would need to be backported per-cycle until stable/stein (as in, if it was in current master, it would need to be backported and merged to stable/train first, then stable/stein), and once merged a stable release would need to be proposed for that branch's version of the library.
Once that stable release is done, that will propose the update to the upper constraint for the given branch.
I would be glad if someone investigated why it happens(/ed) and audited whether other OpenStack projects don't need updating as well to avoid running on old deps when new are awaiting for months. :-) Please note this might apply to other branches as well.
PS: for some reason oslo.messaging Stein release notes ( https://docs.openstack.org/releasenotes/oslo.messaging/stein.html ) are stuck at 9.5.0 as well, this could be right (I did not inspect the sources) but I am adding this in PS so you have more things to correlate if they need be.
Again, as expected. The last stable/stein release was 9.5.0, so that is correct that the release notes for stein only show up to that point.
On 2019-12-30 16:52:49 +0100 (+0100), Radosław Piliszek wrote: [...]
I could not find any doc explaining oslo.messaging versioning [...]
The releases site can be used to identify the most recent releases for each series: https://releases.openstack.org/stein/index.html#library-projects As for how the project is versioned, most libraries (including basically all of Oslo) follow this model: https://releases.openstack.org/reference/release_models.html#cycle-with-inte... Not sure if that's what you're looking for in regards to documentation explaining versioning for that project, but hopefully it's a start. -- Jeremy Stanley
Thanks, already familiar with these. I was looking for something more specific actually, like for when to propose a bugfix release after the cycle deliverable has been released. Sorry for the abundance of tags (incl. [infra]), at first I thought the stagnation was due to some procedure failing, yet it seems simply nobody proposed to rerelease oslo.messaging for Stein. Waiting for someone from Oslo to answer in this thread with more insights. :-) This is important for Kolla as we might be better off sourcing oslo.messaging from git rather than relying on releases which may never happen after the freeze... -yoctozepto wt., 31 gru 2019 o 20:24 Jeremy Stanley <fungi@yuggoth.org> napisał(a):
On 2019-12-30 16:52:49 +0100 (+0100), Radosław Piliszek wrote: [...]
I could not find any doc explaining oslo.messaging versioning [...]
The releases site can be used to identify the most recent releases for each series:
https://releases.openstack.org/stein/index.html#library-projects
As for how the project is versioned, most libraries (including basically all of Oslo) follow this model:
https://releases.openstack.org/reference/release_models.html#cycle-with-inte...
Not sure if that's what you're looking for in regards to documentation explaining versioning for that project, but hopefully it's a start. -- Jeremy Stanley
On 12/30/19 9:52 AM, Radosław Piliszek wrote:
Thanks, Sean! I knew I was missing something really basic! I was under the impression that 9.x is Stein, like it happens with main projects (major=branch). I could not find any doc explaining oslo.messaging versioning, perhaps Oslo could release 9.5.1 off the stein branch?
Oslo for the most part follows semver, so we only bump major versions when there is a breaking change. We bump minor versions each release so we can do bugfix releases on the previous stable branch without stepping on master releases. The underlying cause of this is likely that I'm way behind on releasing the Oslo stable branches. It's high on my todo list now that most people are back from holidays and will be around to help out if a release breaks something. However, anyone can propose a release[0][1] (contrary to what [0] suggests), so if the necessary fix is already on stable/stein and just hasn't been released yet please feel free to do that. You'll just need a +1 from either myself or hberaud (the Oslo release liaison) before the release team will approve it. 0: https://releases.openstack.org/reference/using.html#requesting-a-release 1: https://releases.openstack.org/reference/using.html#using-new-release-comman...
The issue remains that, even though oslo backports bugfixes into their stable branches, kolla (and very possibly other deployment solutions) no longer benefit from them.
-yoctozepto
pon., 30 gru 2019 o 16:01 Sean McGinnis <sean.mcginnis@gmx.com> napisał(a):
On Sun, Dec 29, 2019 at 09:41:45PM +0100, Radosław Piliszek wrote:
Hi Folks,
as the subject goes, my installation has been hit by an old bug: https://bugs.launchpad.net/oslo.messaging/+bug/1828841 (bug details not important, linked here for background)
I am using Stein, deployed with recent Kolla-built source-based images (with only slight modifications compared to vanilla ones). Kolla's procedure for building source-based images considers upper constraints, which, unfortunately, turned out to be lagging behind a few releases w.r.t. oslo.messaging at least. The fix was in 9.7.0 released on May 21, u-c still point to 9.5.0 from Feb 26 and the latest of Stein is 9.8.0 from Jul 18.
It seems oslo.messaging is missing from the automatic updates that bot proposes: https://review.opendev.org/#/q/owner:%22OpenStack+Proposal+Bot%22+project:op...
Per: https://opendev.org/openstack/releases/src/branch/master/doc/source/referenc... this upper-constraint proposal should be happening for all releases.
This is normal and what is expected.
Requirements are only updated for the branch in which those releases happen. So if there is a release of oslo.messaging for stable/train, only the stable/train upper constraints are updated for that new release. The stable/stein branch will not be affected because that shows what the tested upper constraints were for that branch.
The last stable/stein release for oslo.messaging was 9.5.0:
https://opendev.org/openstack/releases/src/branch/master/deliverables/stein/...
And 9.5.0 is what is set in the stable/stein upper-constraints:
https://opendev.org/openstack/requirements/src/branch/stable/stein/upper-con...
To get that raised, whatever necessary bugfixes that are required in oslo.messaging would need to be backported per-cycle until stable/stein (as in, if it was in current master, it would need to be backported and merged to stable/train first, then stable/stein), and once merged a stable release would need to be proposed for that branch's version of the library.
Once that stable release is done, that will propose the update to the upper constraint for the given branch.
I would be glad if someone investigated why it happens(/ed) and audited whether other OpenStack projects don't need updating as well to avoid running on old deps when new are awaiting for months. :-) Please note this might apply to other branches as well.
PS: for some reason oslo.messaging Stein release notes ( https://docs.openstack.org/releasenotes/oslo.messaging/stein.html ) are stuck at 9.5.0 as well, this could be right (I did not inspect the sources) but I am adding this in PS so you have more things to correlate if they need be.
Again, as expected. The last stable/stein release was 9.5.0, so that is correct that the release notes for stein only show up to that point.
Thanks, Ben. That doc preamble really made me think not to cross the holy ground of release proposals. :-) I proposed release [1] and added you and Hervé as reviewers. [1] https://review.opendev.org/701080 -yoctozepto czw., 2 sty 2020 o 21:20 Ben Nemec <openstack@nemebean.com> napisał(a):
On 12/30/19 9:52 AM, Radosław Piliszek wrote:
Thanks, Sean! I knew I was missing something really basic! I was under the impression that 9.x is Stein, like it happens with main projects (major=branch). I could not find any doc explaining oslo.messaging versioning, perhaps Oslo could release 9.5.1 off the stein branch?
Oslo for the most part follows semver, so we only bump major versions when there is a breaking change. We bump minor versions each release so we can do bugfix releases on the previous stable branch without stepping on master releases.
The underlying cause of this is likely that I'm way behind on releasing the Oslo stable branches. It's high on my todo list now that most people are back from holidays and will be around to help out if a release breaks something.
However, anyone can propose a release[0][1] (contrary to what [0] suggests), so if the necessary fix is already on stable/stein and just hasn't been released yet please feel free to do that. You'll just need a +1 from either myself or hberaud (the Oslo release liaison) before the release team will approve it.
0: https://releases.openstack.org/reference/using.html#requesting-a-release 1: https://releases.openstack.org/reference/using.html#using-new-release-comman...
The issue remains that, even though oslo backports bugfixes into their stable branches, kolla (and very possibly other deployment solutions) no longer benefit from them.
-yoctozepto
pon., 30 gru 2019 o 16:01 Sean McGinnis <sean.mcginnis@gmx.com> napisał(a):
On Sun, Dec 29, 2019 at 09:41:45PM +0100, Radosław Piliszek wrote:
Hi Folks,
as the subject goes, my installation has been hit by an old bug: https://bugs.launchpad.net/oslo.messaging/+bug/1828841 (bug details not important, linked here for background)
I am using Stein, deployed with recent Kolla-built source-based images (with only slight modifications compared to vanilla ones). Kolla's procedure for building source-based images considers upper constraints, which, unfortunately, turned out to be lagging behind a few releases w.r.t. oslo.messaging at least. The fix was in 9.7.0 released on May 21, u-c still point to 9.5.0 from Feb 26 and the latest of Stein is 9.8.0 from Jul 18.
It seems oslo.messaging is missing from the automatic updates that bot proposes: https://review.opendev.org/#/q/owner:%22OpenStack+Proposal+Bot%22+project:op...
Per: https://opendev.org/openstack/releases/src/branch/master/doc/source/referenc... this upper-constraint proposal should be happening for all releases.
This is normal and what is expected.
Requirements are only updated for the branch in which those releases happen. So if there is a release of oslo.messaging for stable/train, only the stable/train upper constraints are updated for that new release. The stable/stein branch will not be affected because that shows what the tested upper constraints were for that branch.
The last stable/stein release for oslo.messaging was 9.5.0:
https://opendev.org/openstack/releases/src/branch/master/deliverables/stein/...
And 9.5.0 is what is set in the stable/stein upper-constraints:
https://opendev.org/openstack/requirements/src/branch/stable/stein/upper-con...
To get that raised, whatever necessary bugfixes that are required in oslo.messaging would need to be backported per-cycle until stable/stein (as in, if it was in current master, it would need to be backported and merged to stable/train first, then stable/stein), and once merged a stable release would need to be proposed for that branch's version of the library.
Once that stable release is done, that will propose the update to the upper constraint for the given branch.
I would be glad if someone investigated why it happens(/ed) and audited whether other OpenStack projects don't need updating as well to avoid running on old deps when new are awaiting for months. :-) Please note this might apply to other branches as well.
PS: for some reason oslo.messaging Stein release notes ( https://docs.openstack.org/releasenotes/oslo.messaging/stein.html ) are stuck at 9.5.0 as well, this could be right (I did not inspect the sources) but I am adding this in PS so you have more things to correlate if they need be.
Again, as expected. The last stable/stein release was 9.5.0, so that is correct that the release notes for stein only show up to that point.
Thanks Radosław for the heads up, I validated the new release. Le sam. 4 janv. 2020 à 10:39, Radosław Piliszek <radoslaw.piliszek@gmail.com> a écrit :
Thanks, Ben. That doc preamble really made me think not to cross the holy ground of release proposals. :-)
I proposed release [1] and added you and Hervé as reviewers.
[1] https://review.opendev.org/701080
-yoctozepto
czw., 2 sty 2020 o 21:20 Ben Nemec <openstack@nemebean.com> napisał(a):
On 12/30/19 9:52 AM, Radosław Piliszek wrote:
Thanks, Sean! I knew I was missing something really basic! I was under the impression that 9.x is Stein, like it happens with main projects (major=branch). I could not find any doc explaining oslo.messaging versioning, perhaps Oslo could release 9.5.1 off the stein branch?
Oslo for the most part follows semver, so we only bump major versions when there is a breaking change. We bump minor versions each release so we can do bugfix releases on the previous stable branch without stepping on master releases.
The underlying cause of this is likely that I'm way behind on releasing the Oslo stable branches. It's high on my todo list now that most people are back from holidays and will be around to help out if a release breaks something.
However, anyone can propose a release[0][1] (contrary to what [0] suggests), so if the necessary fix is already on stable/stein and just hasn't been released yet please feel free to do that. You'll just need a +1 from either myself or hberaud (the Oslo release liaison) before the release team will approve it.
0:
https://releases.openstack.org/reference/using.html#requesting-a-release
1:
https://releases.openstack.org/reference/using.html#using-new-release-comman...
The issue remains that, even though oslo backports bugfixes into their stable branches, kolla (and very possibly other deployment solutions) no longer benefit from them.
-yoctozepto
pon., 30 gru 2019 o 16:01 Sean McGinnis <sean.mcginnis@gmx.com>
On Sun, Dec 29, 2019 at 09:41:45PM +0100, Radosław Piliszek wrote:
Hi Folks,
as the subject goes, my installation has been hit by an old bug: https://bugs.launchpad.net/oslo.messaging/+bug/1828841 (bug details not important, linked here for background)
I am using Stein, deployed with recent Kolla-built source-based
images
(with only slight modifications compared to vanilla ones). Kolla's procedure for building source-based images considers upper constraints, which, unfortunately, turned out to be lagging behind a few releases w.r.t. oslo.messaging at least. The fix was in 9.7.0 released on May 21, u-c still point to 9.5.0 from Feb 26 and the latest of Stein is 9.8.0 from Jul 18.
It seems oslo.messaging is missing from the automatic updates that bot proposes:
https://review.opendev.org/#/q/owner:%22OpenStack+Proposal+Bot%22+project:op...
Per:
https://opendev.org/openstack/releases/src/branch/master/doc/source/referenc...
this upper-constraint proposal should be happening for all releases.
This is normal and what is expected.
Requirements are only updated for the branch in which those releases happen. So if there is a release of oslo.messaging for stable/train, only the stable/train upper constraints are updated for that new release. The stable/stein branch will not be affected because that shows what the tested upper constraints were for that branch.
The last stable/stein release for oslo.messaging was 9.5.0:
https://opendev.org/openstack/releases/src/branch/master/deliverables/stein/...
And 9.5.0 is what is set in the stable/stein upper-constraints:
https://opendev.org/openstack/requirements/src/branch/stable/stein/upper-con...
To get that raised, whatever necessary bugfixes that are required in oslo.messaging would need to be backported per-cycle until
stable/stein (as in,
if it was in current master, it would need to be backported and merged to stable/train first, then stable/stein), and once merged a stable release would need to be proposed for that branch's version of the library.
Once that stable release is done, that will propose the update to the upper constraint for the given branch.
I would be glad if someone investigated why it happens(/ed) and audited whether other OpenStack projects don't need updating as well to avoid running on old deps when new are awaiting for months. :-) Please note this might apply to other branches as well.
PS: for some reason oslo.messaging Stein release notes ( https://docs.openstack.org/releasenotes/oslo.messaging/stein.html ) are stuck at 9.5.0 as well, this could be right (I did not inspect
napisał(a): the
sources) but I am adding this in PS so you have more things to correlate if they need be.
Again, as expected. The last stable/stein release was 9.5.0, so that is correct that the release notes for stein only show up to that point.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
participants (5)
-
Ben Nemec
-
Herve Beraud
-
Jeremy Stanley
-
Radosław Piliszek
-
Sean McGinnis