Re: [Heat][requirements] Possible feature backport to stable branches
On Mon, Nov 13, 2023 at 4:50 PM Martin Magr <mmagr@redhat.com> wrote:
Adding more folks from requirements core to CC.
And also correct mailing list. I'm sorry for the noise.
On Mon, Nov 13, 2023 at 4:32 PM Martin Magr <mmagr@redhat.com> wrote:
Greetings,
we have introduced new feature to Aodh [1][2] (which introduced new dependency [3]) and Heat [4] recently. The changes to both projects are relatively small, it just added new respective resources and did not change existing logic. Also the new dependency is providing functionality to those new resources.
We are building our product on stable/antelope and so ideal state for us would be to have those changes included in this branch also on upstream side. I'm aware that Heat follows stable branch policy and it's written in requirements documentation, that new dependency cannot be added in stable ranches, but before we are going to start building workarounds on this I'd like to ask if there is any chance of getting exception (given the fact that changes are pretty safe) on backporting the Heat patch down to stable/antelope and(or at least) including the new dependency in stable/antelope.
Thanks for answer in advance, Martin
[1] https://review.opendev.org/c/openstack/aodh/+/890529 [2] https://review.opendev.org/c/openstack/python-aodhclient/+/893319 [3] https://review.opendev.org/c/openstack/requirements/+/891291 [4] https://review.opendev.org/c/openstack/heat/+/893673
-- Martin Mágr Principal Software Engineer - Service Telemetry Framework Red Hat Czech
-- Martin
Hi, As I mentioned in the heat patch, I have some concerns about this, and now I disagree with the exception request as a heat core. 1. The proposed change is a new feature, not a bug fix. Heat have asserted that we follow the stable branch policy.[1] IMO we should not accept its backport unless - TC updates the stable policy - Heat decides to abandon following the global stable policy and use own one I can't speak for TC, but from my view as a heat core, I'd prefer not changing our project maintenance policy after we've followed the global stable maintenance policy for long period. If we accept this exception then there can be others, and we may eventually accept all feature backports. This changes the expectation for stable branches and can confuse users. [1] https://docs.openstack.org/project-team-guide/stable-branches.html 2. Heat exposes the version a specific resource was added, deprecated or removed. However this interface does not explain a backported feature. The feature was merged after Bobcat release so the support information should look like version >= 21.1.0 or (version < 21.0.0 and version >= 20.1.0( but this does not fit the schema we currently use. 3. This is similar to 2, but heat would be broken in case a user attemptes upgrade from 20.1.0 (antelope with the feature backported) to 21.0.0 (bobcat without the feature), after the new resource is created. We need a proper block to prohibit this upgrade path. 4. The feature was added quite recently, and has never been tested in upstream due to lack of jobs with prometheus. Even if we backport the feature, we should have a proper test coverage in CI or at least manual testing done, to avoid backportimg multiple bug fixes later. I'm open for feedback from the other cores. Thank you, Takashi On 11/14/23 00:58, Martin Magr wrote:
On Mon, Nov 13, 2023 at 4:50 PM Martin Magr <mmagr@redhat.com> wrote:
Adding more folks from requirements core to CC.
And also correct mailing list. I'm sorry for the noise.
On Mon, Nov 13, 2023 at 4:32 PM Martin Magr <mmagr@redhat.com> wrote:
Greetings,
we have introduced new feature to Aodh [1][2] (which introduced new dependency [3]) and Heat [4] recently. The changes to both projects are relatively small, it just added new respective resources and did not change existing logic. Also the new dependency is providing functionality to those new resources.
We are building our product on stable/antelope and so ideal state for us would be to have those changes included in this branch also on upstream side. I'm aware that Heat follows stable branch policy and it's written in requirements documentation, that new dependency cannot be added in stable ranches, but before we are going to start building workarounds on this I'd like to ask if there is any chance of getting exception (given the fact that changes are pretty safe) on backporting the Heat patch down to stable/antelope and(or at least) including the new dependency in stable/antelope.
Thanks for answer in advance, Martin
[1] https://review.opendev.org/c/openstack/aodh/+/890529 [2] https://review.opendev.org/c/openstack/python-aodhclient/+/893319 [3] https://review.opendev.org/c/openstack/requirements/+/891291 [4] https://review.opendev.org/c/openstack/heat/+/893673
-- Martin Mágr Principal Software Engineer - Service Telemetry Framework Red Hat Czech
-- Martin
On Fri, 2023-11-17 at 00:35 +0900, Takashi Kajinami wrote:
Hi,
As I mentioned in the heat patch, I have some concerns about this, and now I disagree with the exception request as a heat core.
1. The proposed change is a new feature, not a bug fix. Heat have asserted that we follow the stable branch policy.[1] IMO we should not accept its backport unless - TC updates the stable policy - Heat decides to abandon following the global stable policy and use own one
I can't speak for TC, but from my view as a heat core, I'd prefer not changing our project maintenance policy after we've followed the global stable maintenance policy for long period. If we accept this exception then there can be others, and we may eventually accept all feature backports. This changes the expectation for stable branches and can confuse users.
[1] https://docs.openstack.org/project-team-guide/stable-branches.html
we are allowed to do feature backports in very limited cases under the stable proicy such execption included operator tooling require to mitigate operational issue like adding a cli option to <service>-manage command to adresss a probelm that woudl have previously required manual db sql commands to fix. the specific exmaple of that i was tinking of in nova were not actully backported but we have said that they would be approcate to backport if there was a need. we have done this in the past at least once in nova but its very very rare and generally only for non end user facing changes. this happens so rarely that i have been unable to find an example of it in nova at all. the closes i have found is we added the ablity to hostmount cpeh volume (which was a feature) as a workaround for a performance regression Ied3732042738a6194b635c55e0304d71a6fb66e3 in an external depndecy. we consider taht to be more of a bug rather a feature as it was a bug just not in nova. The bug was in a c libary shipped in the oeprating systems that nova was installed on. As such we added a tempory feature to nova to easy the live of opearotrs. minor feature like this are accpable as a bugfix and therefor can be backported if they meet the other critia noted in https://docs.openstack.org/project-team-guide/stable-branches.html#appropria.... i dont think https://review.opendev.org/c/openstack/heat/+/893673 falls in that catagory so with my stable maintainer hat on i would also not backport this as i dont think its in line with the intent of the appropriate-fixes section.
2. Heat exposes the version a specific resource was added, deprecated or removed. However this interface does not explain a backported feature. The feature was merged after Bobcat release so the support information should look like version >= 21.1.0 or (version < 21.0.0 and version >= 20.1.0( but this does not fit the schema we currently use.
3. This is similar to 2, but heat would be broken in case a user attemptes upgrade from 20.1.0 (antelope with the feature backported) to 21.0.0 (bobcat without the feature), after the new resource is created. We need a proper block to prohibit this upgrade path.
4. The feature was added quite recently, and has never been tested in upstream due to lack of jobs with prometheus. Even if we backport the feature, we should have a proper test coverage in CI or at least manual testing done, to avoid backportimg multiple bug fixes later.
I'm open for feedback from the other cores.
Thank you, Takashi
On 11/14/23 00:58, Martin Magr wrote:
On Mon, Nov 13, 2023 at 4:50 PM Martin Magr <mmagr@redhat.com> wrote:
Adding more folks from requirements core to CC.
And also correct mailing list. I'm sorry for the noise.
On Mon, Nov 13, 2023 at 4:32 PM Martin Magr <mmagr@redhat.com> wrote:
Greetings,
we have introduced new feature to Aodh [1][2] (which introduced new dependency [3]) and Heat [4] recently. The changes to both projects are relatively small, it just added new respective resources and did not change existing logic. Also the new dependency is providing functionality to those new resources.
We are building our product on stable/antelope and so ideal state for us would be to have those changes included in this branch also on upstream side. I'm aware that Heat follows stable branch policy and it's written in requirements documentation, that new dependency cannot be added in stable ranches, but before we are going to start building workarounds on this I'd like to ask if there is any chance of getting exception (given the fact that changes are pretty safe) on backporting the Heat patch down to stable/antelope and(or at least) including the new dependency in stable/antelope.
i don't belive this is approate for an upstream backport this is something that could be considers in a downstream backport for our downstream product. with my downstream hat on i woudl generaly point out that any feature that require an api change (microversion) is not allowed to be backported downstream eitehr. so the https://review.opendev.org/c/openstack/aodh/+/890529 chagne the heat one depend on is questionable it also add a new depency https://review.opendev.org/c/openstack/aodh/+/890529/13/requirements.txt which is not somethign you are allowed to do in stable branch backports. if you want to prepose the patches to the upstream gerrit so that others can apply the patches to there own openstack distibution i think that is ok but i don't think its correct to ask for stabel branch exception in any of the projects. aodh, requirements, heat or aodhclient.
Thanks for answer in advance, Martin
[1] https://review.opendev.org/c/openstack/aodh/+/890529 [2] https://review.opendev.org/c/openstack/python-aodhclient/+/893319 [3] https://review.opendev.org/c/openstack/requirements/+/891291 [4] https://review.opendev.org/c/openstack/heat/+/893673
-- Martin Mágr Principal Software Engineer - Service Telemetry Framework Red Hat Czech
-- Martin
On 16/11/2023 16:35, Takashi Kajinami wrote:
Hi,
As I mentioned in the heat patch, I have some concerns about this, and now I disagree with the exception request as a heat core.
1. The proposed change is a new feature, not a bug fix. Heat have asserted that we follow the stable branch policy.[1] IMO we should not accept its backport unless - TC updates the stable policy - Heat decides to abandon following the global stable policy and use own one
Thank you Takashi! The argument here is, that it is a feature backport that would not change existing behavior, thus is safe for current users of Bobcat or Antelope.
3. This is similar to 2, but heat would be broken in case a user attemptes upgrade from 20.1.0 (antelope with the feature backported) to 21.0.0 (bobcat without the feature), after the new resource is created. We need a proper block to prohibit this upgrade path.
To me, the scenario of a user using the latest antelope branch that would update to the earliest bobcat release seems artificial. Backporting patches or features requires to backport in this case from Caracal to Bobcat and then to Antelope. For a user to land in a "broken" state or regression would then require: - use the latest Antelope version AND the new prometheus feature - upgrade to Bobcat in a version where the feature does not exist (i.e not including latest backports) For users of distributions, no regressions should be guaranteed by the distribution.
4. The feature was added quite recently, and has never been tested in upstream due to lack of jobs with prometheus. Even if we backport the feature, we should have a proper test coverage in CI or at least manual testing done, to avoid backportimg multiple bug fixes later.
We are working on jobs that involve prometheus, see https://review.opendev.org/c/openstack/ceilometer/+/898087 https://review.opendev.org/c/openstack/ceilometer/+/900509 and following (functional tests for observabilityclient) https://review.opendev.org/c/openstack/python-observabilityclient/+/899924 Ultimately, prometheus should be a drop-in replacement for gnocchi and the telemetry tests should continue to work as they do right now. -- Matthias Runge <mrunge@matthias-runge.de>
On 11/20/23 18:50, Matthias Runge wrote:
On 16/11/2023 16:35, Takashi Kajinami wrote:
Hi,
As I mentioned in the heat patch, I have some concerns about this, and now I disagree with the exception request as a heat core.
1. The proposed change is a new feature, not a bug fix. Heat have asserted that we follow the stable branch policy.[1] IMO we should not accept its backport unless - TC updates the stable policy - Heat decides to abandon following the global stable policy and use own one
Thank you Takashi!
The argument here is, that it is a feature backport that would not change existing behavior, thus is safe for current users of Bobcat or Antelope.
My main point is not whether backporting this is "safe", but whether it is aligned with the global policies and whether there is any critical requirement which justifies an exception.
3. This is similar to 2, but heat would be broken in case a user attemptes upgrade from 20.1.0 (antelope with the feature backported) to 21.0.0 (bobcat without the feature), after the new resource is created. We need a proper block to prohibit this upgrade path.
To me, the scenario of a user using the latest antelope branch that would update to the earliest bobcat release seems artificial. Backporting patches or features requires to backport in this case from Caracal to Bobcat and then to Antelope.
For a user to land in a "broken" state or regression would then require: - use the latest Antelope version AND the new prometheus feature - upgrade to Bobcat in a version where the feature does not exist (i.e not including latest backports)
"Upgrade to an older version" might sound artificial, but is still a possible scenario. This has been working with the current global policy but is broken by approving an exception, and we should evaluate the risk of potential breaking expectations by some users.
For users of distributions, no regressions should be guaranteed by the distribution.
This may work for products, but we still have users using non-product distributions such as upstream code or middle-stream like RDO/UCA.
4. The feature was added quite recently, and has never been tested in upstream due to lack of jobs with prometheus. Even if we backport the feature, we should have a proper test coverage in CI or at least manual testing done, to avoid backportimg multiple bug fixes later.
We are working on jobs that involve prometheus, see https://review.opendev.org/c/openstack/ceilometer/+/898087 https://review.opendev.org/c/openstack/ceilometer/+/900509
and following (functional tests for observabilityclient) https://review.opendev.org/c/openstack/python-observabilityclient/+/899924
Ultimately, prometheus should be a drop-in replacement for gnocchi and the telemetry tests should continue to work as they do right now.
It's very nice to see some work going on about test coverage !
On Fri, 2023-11-17 at 00:35 +0900, Takashi Kajinami wrote:
Hi,
As I mentioned in the heat patch, I have some concerns about this, and now I disagree with the exception request as a heat core.
1. The proposed change is a new feature, not a bug fix. Heat have asserted that we follow the stable branch policy.[1] IMO we should not accept its backport unless - TC updates the stable policy - Heat decides to abandon following the global stable policy and use own one
I can't speak for TC, but from my view as a heat core, I'd prefer not changing our project maintenance policy after we've followed the global stable maintenance policy for long period. If we accept this exception then there can be others, and we may eventually accept all feature backports. This changes the expectation for stable branches and can confuse users.
[1] https://docs.openstack.org/project-team-guide/stable-branches.html
we are allowed to do feature backports in very limited cases under the stable proicy such execption included operator tooling require to mitigate operational issue like adding a cli option to <service>-manage command to adresss a probelm that woudl have previously required manual db sql commands to fix. the specific exmaple of that i was tinking of in nova were not actully backported but we have said that they would be approcate to backport if there was a need. we have done this in the past at least once in nova but its very very rare and generally only for non end user facing changes. this happens so rarely that i have been unable to find an example of it in nova at all. the closes i have found is we added the ablity to hostmount cpeh volume (which was a feature) as a workaround for a performance regression Ied3732042738a6194b635c55e0304d71a6fb66e3 in an external depndecy. we consider taht to be more of a bug rather a feature as it was a bug just not in nova. The bug was in a c libary shipped in the oeprating systems that nova was installed on. As such we added a tempory feature to nova to easy the live of opearotrs. minor feature like this are accpable as a bugfix and therefor can be backported if they meet the other critia noted in https://docs.openstack.org/project-team-guide/stable-branches.html#appropria.... i dont think https://review.opendev.org/c/openstack/heat/+/893673 falls in that catagory so with my stable maintainer hat on i would also not backport this as i dont think its in line with the intent of the appropriate-fixes section.
2. Heat exposes the version a specific resource was added, deprecated or removed. However this interface does not explain a backported feature. The feature was merged after Bobcat release so the support information should look like version >= 21.1.0 or (version < 21.0.0 and version >= 20.1.0( but this does not fit the schema we currently use.
3. This is similar to 2, but heat would be broken in case a user attemptes upgrade from 20.1.0 (antelope with the feature backported) to 21.0.0 (bobcat without the feature), after the new resource is created. We need a proper block to prohibit this upgrade path.
4. The feature was added quite recently, and has never been tested in upstream due to lack of jobs with prometheus. Even if we backport the feature, we should have a proper test coverage in CI or at least manual testing done, to avoid backportimg multiple bug fixes later.
I'm open for feedback from the other cores.
Thank you, Takashi
On 11/14/23 00:58, Martin Magr wrote:
On Mon, Nov 13, 2023 at 4:50 PM Martin Magr <mmagr@redhat.com> wrote:
Adding more folks from requirements core to CC.
And also correct mailing list. I'm sorry for the noise.
On Mon, Nov 13, 2023 at 4:32 PM Martin Magr <mmagr@redhat.com> wrote:
Greetings,
we have introduced new feature to Aodh [1][2] (which introduced new dependency [3]) and Heat [4] recently. The changes to both projects are relatively small, it just added new respective resources and did not change existing logic. Also the new dependency is providing functionality to those new resources.
We are building our product on stable/antelope and so ideal state for us would be to have those changes included in this branch also on upstream side. I'm aware that Heat follows stable branch policy and it's written in requirements documentation, that new dependency cannot be added in stable ranches, but before we are going to start building workarounds on this I'd like to ask if there is any chance of getting exception (given the fact that changes are pretty safe) on backporting the Heat patch down to stable/antelope and(or at least) including the new dependency in stable/antelope.
i don't belive this is approate for an upstream backport this is something that could be considers in a downstream backport for our downstream product. with my downstream hat on i woudl generaly point out that any feature that require an api change (microversion) is not allowed to be backported downstream eitehr. so the https://review.opendev.org/c/openstack/aodh/+/890529 chagne the heat one depend on is questionable it also add a new depency https://review.opendev.org/c/openstack/aodh/+/890529/13/requirements.txt which is not somethign you are allowed to do in stable branch backports. if you want to prepose the patches to the upstream gerrit so that others can apply the patches to there own openstack distibution i think that is ok but i don't think its correct to ask for stabel branch exception in any of the projects. aodh, requirements, heat or aodhclient.
Thanks for answer in advance, Martin
[1] https://review.opendev.org/c/openstack/aodh/+/890529 [2] https://review.opendev.org/c/openstack/python-aodhclient/+/893319 [3] https://review.opendev.org/c/openstack/requirements/+/891291 [4] https://review.opendev.org/c/openstack/heat/+/893673
-- Martin Mágr Principal Software Engineer - Service Telemetry Framework Red Hat Czech
-- Martin
On 17/11/23 04:35, Takashi Kajinami wrote:
Hi,
As I mentioned in the heat patch, I have some concerns about this, and now I disagree with the exception request as a heat core.
1. The proposed change is a new feature, not a bug fix. Heat have asserted that we follow the stable branch policy.[1] IMO we should not accept its backport unless - TC updates the stable policy - Heat decides to abandon following the global stable policy and use own one
I can't speak for TC, but from my view as a heat core, I'd prefer not changing our project maintenance policy after we've followed the global stable maintenance policy for long period. If we accept this exception then there can be others, and we may eventually accept all feature backports. This changes the expectation for stable branches and can confuse users.
[1] https://docs.openstack.org/project-team-guide/stable-branches.html
2. Heat exposes the version a specific resource was added, deprecated or removed. However this interface does not explain a backported feature. The feature was merged after Bobcat release so the support information should look like version >= 21.1.0 or (version < 21.0.0 and version >= 20.1.0( but this does not fit the schema we currently use.
3. This is similar to 2, but heat would be broken in case a user attemptes upgrade from 20.1.0 (antelope with the feature backported) to 21.0.0 (bobcat without the feature), after the new resource is created. We need a proper block to prohibit this upgrade path.
4. The feature was added quite recently, and has never been tested in upstream due to lack of jobs with prometheus. Even if we backport the feature, we should have a proper test coverage in CI or at least manual testing done, to avoid backportimg multiple bug fixes later.
I'm open for feedback from the other cores.
FWIW I completely agree with Takashi here.
Thank you, Takashi
On 11/14/23 00:58, Martin Magr wrote:
On Mon, Nov 13, 2023 at 4:50 PM Martin Magr <mmagr@redhat.com> wrote:
Adding more folks from requirements core to CC.
And also correct mailing list. I'm sorry for the noise.
On Mon, Nov 13, 2023 at 4:32 PM Martin Magr <mmagr@redhat.com> wrote:
Greetings,
we have introduced new feature to Aodh [1][2] (which introduced new dependency [3]) and Heat [4] recently. The changes to both projects are relatively small, it just added new respective resources and did not change existing logic. Also the new dependency is providing functionality to those new resources.
We are building our product on stable/antelope and so ideal state for us would be to have those changes included in this branch also on upstream side. I'm aware that Heat follows stable branch policy and it's written in requirements documentation, that new dependency cannot be added in stable ranches, but before we are going to start building workarounds on this I'd like to ask if there is any chance of getting exception (given the fact that changes are pretty safe) on backporting the Heat patch down to stable/antelope and(or at least) including the new dependency in stable/antelope.
Thanks for answer in advance, Martin
[1] https://review.opendev.org/c/openstack/aodh/+/890529 [2] https://review.opendev.org/c/openstack/python-aodhclient/+/893319 [3] https://review.opendev.org/c/openstack/requirements/+/891291 [4] https://review.opendev.org/c/openstack/heat/+/893673
-- Martin Mágr Principal Software Engineer - Service Telemetry Framework Red Hat Czech
-- Martin
participants (5)
-
Martin Magr
-
Matthias Runge
-
smooney＠redhat.com
-
Takashi Kajinami
-
Zane Bitter