[oslo][nova][stable][requirements] Fixing a high CPU usage from oslo.service into stable/rocky branch
Hey all! Here is a thread to coordinate all the teams (oslo, nova, stable, requirements) working on the update of the oslo.service constraint in the Rocky requirements. # Summary Usage of threading event with eventlet caused inefficient code (causing many useless system calls and high CPU usage). This issue was already fixed on oslo.service master and we also want to fix it in stable/rocky. Our main issue is how to fix the high CPU usage on stable/rocky without break the nova CI. Indeed, we already have backported the eventlet related fix to oslo.service but this fix requires also a nova update to avoid nova CI errors due to threading removal on oslo.service that introduce the nova CI errors. A fix was proposed and merged on oslo.service master to introduce a new feature (fixture) that avoid the nova CI errors, but backporting the master fix to Rocky introduces a new feature into a stable branch so this is also an issue. So we need to discuss with all the teams to find a proper solution. # History A few weeks ago this issue was opened on oslo.service ( https://bugs.launchpad.net/oslo.service/+bug/1798774) and it was fixed by this submited patch on the master branch ( https://review.openstack.org/#/c/611807/ ). This change use the proper event primitive to fix the performance issue. A new version of oslo.service was released (1.32.1) Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master. Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release. When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version). Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error. # Issue Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements. In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers. The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue. Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes. # Questions and proposed solutions This thread try to summarize the current situation. We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix. 1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ? 2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...) ? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above. Personaly I think we need to find an another approach like the mocking remplacement (c.f 2). We need to decide which way we use and to discuss about other solutions. -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Herve Beraud <hberaud@redhat.com> writes:
Hey all!
Here is a thread to coordinate all the teams (oslo, nova, stable, requirements) working on the update of the oslo.service constraint in the Rocky requirements.
# Summary
Usage of threading event with eventlet caused inefficient code (causing many useless system calls and high CPU usage). This issue was already fixed on oslo.service master and we also want to fix it in stable/rocky.
Our main issue is how to fix the high CPU usage on stable/rocky without break the nova CI.
Indeed, we already have backported the eventlet related fix to oslo.service but this fix requires also a nova update to avoid nova CI errors due to threading removal on oslo.service that introduce the nova CI errors.
A fix was proposed and merged on oslo.service master to introduce a new feature (fixture) that avoid the nova CI errors, but backporting the master fix to Rocky introduces a new feature into a stable branch so this is also an issue.
So we need to discuss with all the teams to find a proper solution.
# History
A few weeks ago this issue was opened on oslo.service ( https://bugs.launchpad.net/oslo.service/+bug/1798774) and it was fixed by this submited patch on the master branch ( https://review.openstack.org/#/c/611807/ ).
This change use the proper event primitive to fix the performance issue.
A new version of oslo.service was released (1.32.1)
Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master.
Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release.
When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version).
Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error.
# Issue
Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements.
In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers.
The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue.
Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes.
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ?
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...) ? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Thank you for summarizing this issue, Hervé, and for working on the patches we need. I think I would be happy with either solution. Using clean backports seems less risky, and even though we are adding a new feature to oslo.service it's only a unit test fixture. On the other hand if we want to be very strict about not adding features in stable branches and we are OK with creating a change to nova's unit tests that is not backported from master, then that works for me, too. I have a slight preference for the first proposal, but not strong enough to vote fight for it if the majority decides to go with the second option. -- Doug
On Wed, 2018-11-21 at 09:57 -0500, Doug Hellmann wrote:
Herve Beraud <hberaud@redhat.com> writes:
Hey all!
Here is a thread to coordinate all the teams (oslo, nova, stable, requirements) working on the update of the oslo.service constraint in the Rocky requirements.
# Summary
Usage of threading event with eventlet caused inefficient code (causing many useless system calls and high CPU usage). This issue was already fixed on oslo.service master and we also want to fix it in stable/rocky.
Our main issue is how to fix the high CPU usage on stable/rocky without break the nova CI.
Indeed, we already have backported the eventlet related fix to oslo.service but this fix requires also a nova update to avoid nova CI errors due to threading removal on oslo.service that introduce the nova CI errors.
A fix was proposed and merged on oslo.service master to introduce a new feature (fixture) that avoid the nova CI errors, but backporting the master fix to Rocky introduces a new feature into a stable branch so this is also an issue.
So we need to discuss with all the teams to find a proper solution.
# History
A few weeks ago this issue was opened on oslo.service ( https://bugs.launchpad.net/oslo.service/+bug/1798774) and it was fixed by this submited patch on the master branch ( https://review.openstack.org/#/c/611807/ ).
This change use the proper event primitive to fix the performance issue.
A new version of oslo.service was released (1.32.1)
Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master.
Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release.
When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version).
Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error.
# Issue
Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements.
In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers.
The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue.
Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes.
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ?
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...) ? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Thank you for summarizing this issue, Hervé, and for working on the patches we need.
I think I would be happy with either solution. Using clean backports seems less risky, and even though we are adding a new feature to oslo.service it's only a unit test fixture. On the other hand if we want to be very strict about not adding features in stable branches and we are OK with creating a change to nova's unit tests that is not backported from master, then that works for me, too.
it should be noted this is not just a blocker for the nova ci if we dont fix the unit test it will break ditrobution that run the unitest as part of there packaging of nova downstream. i would prefer to backport the fixture personally and do a clean backport of the nova patches also rather then a stable only patch. while thecnically a feature i dont really consider a test fixture to be a feature in the normal sense and it is relitivly small and self contained.
I have a slight preference for the first proposal, but not strong enough to vote fight for it if the majority decides to go with the second option.
I agree this is not a feature in the normal sense. Fixture is isolated from the rest and doesn't affecting the oslo.service consumers. I have a slight preference for the second solution but I've no problems to use the first proposed solution (backport). Le mer. 21 nov. 2018 à 16:02, Sean Mooney <smooney@redhat.com> a écrit :
Herve Beraud <hberaud@redhat.com> writes:
Hey all!
Here is a thread to coordinate all the teams (oslo, nova, stable, requirements) working on the update of the oslo.service constraint in
On Wed, 2018-11-21 at 09:57 -0500, Doug Hellmann wrote: the
Rocky requirements.
# Summary
Usage of threading event with eventlet caused inefficient code (causing many useless system calls and high CPU usage). This issue was already fixed on oslo.service master and we also want to fix it in stable/rocky.
Our main issue is how to fix the high CPU usage on stable/rocky without break the nova CI.
Indeed, we already have backported the eventlet related fix to oslo.service but this fix requires also a nova update to avoid nova CI errors due to threading removal on oslo.service that introduce the nova CI errors.
A fix was proposed and merged on oslo.service master to introduce a new feature (fixture) that avoid the nova CI errors, but backporting the master fix to Rocky introduces a new feature into a stable branch so this is also an issue.
So we need to discuss with all the teams to find a proper solution.
# History
A few weeks ago this issue was opened on oslo.service ( https://bugs.launchpad.net/oslo.service/+bug/1798774) and it was fixed by this submited patch on the master branch ( https://review.openstack.org/#/c/611807/ ).
This change use the proper event primitive to fix the performance issue.
A new version of oslo.service was released (1.32.1)
Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master.
Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release.
When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version).
Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error.
# Issue
Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements.
In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers.
The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue.
Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes.
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ?
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example:
https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu... )
? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
Thank you for summarizing this issue, Hervé, and for working on the patches we need.
I think I would be happy with either solution. Using clean backports seems less risky, and even though we are adding a new feature to oslo.service it's only a unit test fixture. On the other hand if we want to be very strict about not adding features in stable branches and we are OK with creating a change to nova's unit tests that is not backported from master, then that works for me, too.
it should be noted this is not just a blocker for the nova ci if we dont fix the unit test it will break ditrobution that run the unitest as part of there packaging of nova downstream. i would prefer to backport the fixture personally and do a clean backport of the nova patches also rather then a stable only patch. while thecnically a feature i dont really consider a test fixture to be a feature in the normal sense and it is relitivly small and self contained.
I have a slight preference for the first proposal, but not strong enough to vote fight for it if the majority decides to go with the second option.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
I have no preference. For #1, the backports won't be perfectly clean, because the requirements files will have different versions of oslo.service in them. But that's no big deal. It's also more "work", but the patches are already proposed, so meh. For #2, the original objection [1] was that it's still mocking private things from a 3rd-party library. But that's not as big a deal for stable, which is less likely to yank the rug out. (You know, except for the backport that led to this issue...) -efried [1] https://review.openstack.org/#/c/615724/1/nova/tests/unit/compute/test_compu... On 11/21/18 09:28, Herve Beraud wrote:
I agree this is not a feature in the normal sense. Fixture is isolated from the rest and doesn't affecting the oslo.service consumers. I have a slight preference for the second solution but I've no problems to use the first proposed solution (backport).
Le mer. 21 nov. 2018 à 16:02, Sean Mooney <smooney@redhat.com <mailto:smooney@redhat.com>> a écrit :
On Wed, 2018-11-21 at 09:57 -0500, Doug Hellmann wrote: > Herve Beraud <hberaud@redhat.com <mailto:hberaud@redhat.com>> writes: > > > Hey all! > > > > Here is a thread to coordinate all the teams (oslo, nova, stable, > > requirements) working on the update of the oslo.service constraint in the > > Rocky requirements. > > > > # Summary > > > > Usage of threading event with eventlet caused inefficient code (causing > > many useless system calls and high CPU usage). > > This issue was already fixed on oslo.service master and we also want to fix > > it in stable/rocky. > > > > Our main issue is how to fix the high CPU usage on stable/rocky without > > break the nova CI. > > > > Indeed, we already have backported the eventlet related fix to oslo.service > > but this fix requires also a nova update to avoid nova CI errors due to > > threading removal on oslo.service that introduce the nova CI errors. > > > > A fix was proposed and merged on oslo.service master to introduce a new > > feature (fixture) that avoid the nova CI errors, but > > backporting the master fix to Rocky introduces a new feature into a stable > > branch so this is also an issue. > > > > So we need to discuss with all the teams to find a proper solution. > > > > # History > > > > A few weeks ago this issue was opened on oslo.service ( > > https://bugs.launchpad.net/oslo.service/+bug/1798774) and it was fixed by > > this submited patch on the master branch ( > > https://review.openstack.org/#/c/611807/ ). > > > > This change use the proper event primitive to fix the performance issue. > > > > A new version of oslo.service was released (1.32.1) > > > > Since these changes was introduced into oslo.service master, nova facing > > some issues into the master CI process, due to the threading changes, and > > they was fixed by these patches ( https://review.openstack.org/#/c/615724/, > > https://review.openstack.org/#/c/617989/ ) into master. > > > > Few weeks ago I have backport to oslo.service some changes ( > > https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to > > also fix the problem in the rocky release. > > > > When this backport was merged we have created a new release of oslo.service > > (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky > > version). > > > > Then the openstack proposal bot submit a patch to requirements on stable > > rocky to update the oslo.service version with the latest version (1.31.6) > > but if we'll use it we'll then break the CI > > https://review.openstack.org/#/c/618834/ so this patch is currently blocked > > to avoid nova CI error. > > > > # Issue > > > > Since the oslo.services threading changes were backported to rocky we risk > > to faces the same issues inside the nova rocky CI if we update the > > requirements. > > > > In parallel in oslo.service we have started to backport a new patch who > > introduces fixture ( https://review.openstack.org/#/c/617989/ ) from > > master to rocky, and also we start to backport on nova rocky branch ( > > https://review.openstack.org/619019, https://review.openstack.org/619022 ) > > patches who use oslo.service.fixture and who solve the nova CI issue. The > > patch on oslo.service exposes a public oslo_service.fixture.SleepFixture > > for this purpose. It can be maintained opaquely as internals change without > > affecting its consumers. > > > > The main problem is that the patch bring a new functionality to a stable > > branch (oslo.service rocky) but this patch help to fix the nova issue. > > > > Also openstack proposal bot submit a patch to requirements on stable rocky > > to update the oslo.service version with the latest version (1.31.6) but if > > we'll use it we'll then break the CI > > https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is > > incompatible with novas stable rocky unittest due to the threading changes. > > > > # Questions and proposed solutions > > > > This thread try to summarize the current situation. > > > > We need to find how to be able to proceed, so this thread aim to allow to > > discuss between team to find the best way to fix. > > > > 1. Do we need to continue to try to backport fixture on oslo.service to fix > > the CI problem (https://review.openstack.org/#/c/617989/) ? > > > > 2. Do we need to find an another approach like mocking > > oslo.service.loopingcall._Event.wait in nova instead of mocking > > oslo_service.loopingcall._ThreadingEvent.wait (example: > > https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...) > > ? > > This is only a fix on the nova side and it allows us to update oslo.service > > requirements and allows us to fix the high CPU usage issue. I've submit > > this patch (https://review.openstack.org/619246) who implement the > > description above. > > > > Personaly I think we need to find an another approach like the mocking > > remplacement (c.f 2). > > > > We need to decide which way we use and to discuss about other solutions. > > > > -- > > Hervé Beraud > > Senior Software Engineer > > Red Hat - Openstack Oslo > > irc: hberaud > > -----BEGIN PGP SIGNATURE----- > > > > wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ > > Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ > > RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP > > F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G > > 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g > > glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw > > m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ > > hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 > > qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y > > F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 > > B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O > > v6rDpkeNksZ9fFSyoY2o > > =ECSj > > -----END PGP SIGNATURE----- > > Thank you for summarizing this issue, Hervé, and for working on the > patches we need. > > I think I would be happy with either solution. Using clean backports > seems less risky, and even though we are adding a new feature to > oslo.service it's only a unit test fixture. On the other hand if we want > to be very strict about not adding features in stable branches and we > are OK with creating a change to nova's unit tests that is not > backported from master, then that works for me, too.
it should be noted this is not just a blocker for the nova ci if we dont fix the unit test it will break ditrobution that run the unitest as part of there packaging of nova downstream. i would prefer to backport the fixture personally and do a clean backport of the nova patches also rather then a stable only patch. while thecnically a feature i dont really consider a test fixture to be a feature in the normal sense and it is relitivly small and self contained. > > I have a slight preference for the first proposal, but not strong enough > to vote fight for it if the majority decides to go with the second > option. >
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
On 21/11/18 9:47 AM, Herve Beraud wrote:
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/)?
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...
3. Doesn't this get solved if we add a line like: _ThreadingEvent = _Event in oslo.service on stable/rocky? That seems harmless and the easiest way to maintain the same sort-of-public interface so nothing else ought to break either. And with no change in Nova people won't need to worry about needing to update oslo.service at the same time they update Nova to avoid breakage. Here's a patch: https://review.openstack.org/619342 cheers, Zane.
This is only a fix on the nova side and itallowsus to update oslo.service requirements and allowsus to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246)who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
On Wed, 2018-11-21 at 14:38 -0500, Zane Bitter wrote:
On 21/11/18 9:47 AM, Herve Beraud wrote:
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/)?
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...
3. Doesn't this get solved if we add a line like:
_ThreadingEvent = _Event
in oslo.service on stable/rocky? That seems harmless and the easiest way to maintain the same sort-of-public interface so nothing else ought to break either. And with no change in Nova people won't need to worry about needing to update oslo.service at the same time they update Nova to avoid breakage.
Here's a patch: https://review.openstack.org/619342 a stable only patch is not really any better in my view then 2 you are also chaning the loopingcall modules semantics as it is a different type even if you are allowing previously syntaxly vaild code to execute it does not maintain backwards compatiblity.
we are not using a staticaly compiled language so we dont need to consider abi breakage this would result in for c or c++ but it i would stonglely prefer 1 or 2.
cheers, Zane.
This is only a fix on the nova side and itallowsus to update oslo.service requirements and allowsus to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246)who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
On 21/11/18 2:50 PM, Sean Mooney wrote:
On Wed, 2018-11-21 at 14:38 -0500, Zane Bitter wrote:
On 21/11/18 9:47 AM, Herve Beraud wrote:
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/)?
This is the worst option, because you won't be able to use either an older nova with a newer oslo.service, nor an older oslo.service with a newer nova. In fact, if I'm interpreting Matt's comment on https://review.openstack.org/619246 correctly, then this may be a non-starter because increasing lower-constraints is not allowed on stable branches.
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...
This is marginally better, provided that it's done in a backwards-compatible way (i.e. that doesn't require bumping the lower-constraints like https://review.openstack.org/619246 and https://review.openstack.org/619022 do). Here's an example that should do the trick: https://review.openstack.org/619360 However, this does still mean that you can't use an older Nova with a newer oslo.service. Which is bound to cause trouble for somebody. It also relies on a different private interface, although we're less likely to need to change this one in stable/rocky.
3. Doesn't this get solved if we add a line like:
_ThreadingEvent = _Event
in oslo.service on stable/rocky? That seems harmless and the easiest way to maintain the same sort-of-public interface so nothing else ought to break either. And with no change in Nova people won't need to worry about needing to update oslo.service at the same time they update Nova to avoid breakage.
Here's a patch: https://review.openstack.org/619342 a stable only patch is not really any better in my view then 2
Surely being able to update oslo.service and nova independently is objectively better than having to upgrade them in lock-step. Would it make you feel better if this patch were also on master? Why?
you are also chaning the loopingcall modules semantics as it is a different type even if you are allowing previously syntaxly vaild code to execute it does not maintain backwards compatiblity.
This is a fair point; only the clear()/wait()/stop() methods are the same. is_running() changes to is_set() and done() changes to set(). So it is a bit hacky. But it still solves the instance of the problem we know about (and FWIW any instance of the problem that *could*, in principle, be solved by the fixture).
we are not using a staticaly compiled language so we dont need to consider abi breakage this would result in for c or c++ but it i would stonglely prefer 1 or 2.
cheers, Zane.
This is only a fix on the nova side and itallowsus to update oslo.service requirements and allowsus to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246)who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
On 21/11/18 2:50 PM, Sean Mooney wrote:
On Wed, 2018-11-21 at 14:38 -0500, Zane Bitter wrote:
On 21/11/18 9:47 AM, Herve Beraud wrote:
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/)?
This is the worst option, because you won't be able to use either an older nova with a newer oslo.service, nor an older oslo.service with a newer nova.
In fact, if I'm interpreting Matt's comment on https://review.openstack.org/619246 correctly, then this may be a non-starter because increasing lower-constraints is not allowed on stable branches. you also cannot use a newer versions then are in upper-constaints
On Wed, 2018-11-21 at 17:03 -0500, Zane Bitter wrote: this thread arose from the fact that performacne feature/bug fix form 1.32.x is being backported to 1.31.6. upper-constainst set a limit on oslo.service of 1.31.x so you are not expect to be able to use newer relases of oslo.service with the rocky release of nova.
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...
This is marginally better, provided that it's done in a backwards-compatible way (i.e. that doesn't require bumping the lower-constraints like https://review.openstack.org/619246 and https://review.openstack.org/619022 do). Here's an example that should do the trick:
https://review.openstack.org/619360
However, this does still mean that you can't use an older Nova with a newer oslo.service. Which is bound to cause trouble for somebody.
It also relies on a different private interface, although we're less likely to need to change this one in stable/rocky.
3. Doesn't this get solved if we add a line like:
_ThreadingEvent = _Event
in oslo.service on stable/rocky? That seems harmless and the easiest way to maintain the same sort-of-public interface so nothing else ought to break either. And with no change in Nova people won't need to worry about needing to update oslo.service at the same time they update Nova to avoid breakage.
Here's a patch: https://review.openstack.org/619342
a stable only patch is not really any better in my view then 2
Surely being able to update oslo.service and nova independently is objectively better than having to upgrade them in lock-step.
Would it make you feel better if this patch were also on master? Why?
you are also chaning the loopingcall modules semantics as it is a different type even if you are allowing previously syntaxly vaild code to execute it does not maintain backwards compatiblity.
This is a fair point; only the clear()/wait()/stop() methods are the same. is_running() changes to is_set() and done() changes to set(). So it is a bit hacky. But it still solves the instance of the problem we know about (and FWIW any instance of the problem that *could*, in principle, be solved by the fixture).
if we go with this appoch and are doing a stable only change then instead of just doing _ThreadingEvent =_Event could we either use the debtcoltor moduel to make it as deprecated or stub out the signigure of the old interface and have it log a warning if invoked nova is only using this in test but if we are consusred about people upgrading then when we remove this workaourd at some point e.g. we shoudl at least warn other that might have done the same and depended on the ThredingEvents.
we are not using a staticaly compiled language so we dont need to consider abi breakage this would result in for c or c++ but it i would stonglely prefer 1 or 2.
cheers, Zane.
This is only a fix on the nova side and itallowsus to update oslo.service requirements and allowsus to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246)who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
On 21/11/18 5:35 PM, Sean Mooney wrote:
On 21/11/18 2:50 PM, Sean Mooney wrote:
On Wed, 2018-11-21 at 14:38 -0500, Zane Bitter wrote:
On 21/11/18 9:47 AM, Herve Beraud wrote:
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/)?
This is the worst option, because you won't be able to use either an older nova with a newer oslo.service, nor an older oslo.service with a newer nova.
In fact, if I'm interpreting Matt's comment on https://review.openstack.org/619246 correctly, then this may be a non-starter because increasing lower-constraints is not allowed on stable branches. you also cannot use a newer versions then are in upper-constaints
On Wed, 2018-11-21 at 17:03 -0500, Zane Bitter wrote: this thread arose from the fact that performacne feature/bug fix form 1.32.x is being backported to 1.31.6. upper-constainst set a limit on oslo.service of 1.31.x so you are not expect to be able to use newer relases of oslo.service with the rocky release of nova.
Right, but the whole point of this discussion is that once we fix this we can bump the upper-constraints to 1.31.6 or 1.31.7. Whereas apparently we *cannot* bump the lower-constraints.
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...
This is marginally better, provided that it's done in a backwards-compatible way (i.e. that doesn't require bumping the lower-constraints like https://review.openstack.org/619246 and https://review.openstack.org/619022 do). Here's an example that should do the trick:
https://review.openstack.org/619360
However, this does still mean that you can't use an older Nova with a newer oslo.service. Which is bound to cause trouble for somebody.
It also relies on a different private interface, although we're less likely to need to change this one in stable/rocky.
3. Doesn't this get solved if we add a line like:
_ThreadingEvent = _Event
in oslo.service on stable/rocky? That seems harmless and the easiest way to maintain the same sort-of-public interface so nothing else ought to break either. And with no change in Nova people won't need to worry about needing to update oslo.service at the same time they update Nova to avoid breakage.
Here's a patch: https://review.openstack.org/619342
a stable only patch is not really any better in my view then 2
Surely being able to update oslo.service and nova independently is objectively better than having to upgrade them in lock-step.
Would it make you feel better if this patch were also on master? Why?
you are also chaning the loopingcall modules semantics as it is a different type even if you are allowing previously syntaxly vaild code to execute it does not maintain backwards compatiblity.
This is a fair point; only the clear()/wait()/stop() methods are the same. is_running() changes to is_set() and done() changes to set(). So it is a bit hacky. But it still solves the instance of the problem we know about (and FWIW any instance of the problem that *could*, in principle, be solved by the fixture). if we go with this appoch and are doing a stable only change then instead of just doing _ThreadingEvent =_Event could we either use the debtcoltor moduel to make it as deprecated
Sadly, this won't work because if they're not referring to the same object then stubbing out the method won't have the desired effect.
or stub out the signigure of the old interface and have it log a warning if invoked
Happy to do this.
nova is only using this in test but if we are consusred about people upgrading then when we remove this workaourd at some point e.g. we shoudl at least warn other that might have done the same and depended on the ThredingEvents.
Well, we already removed it because it's long gone on master. So I doubt that adding deprecation warnings in a stable branch will catch any interesting cases that haven't been found already. cheers, Zane.
HI folks, I admit my initial response to this was mor pragmatic 'take the bakport' but as I thought it through I saw more problems with that approach. On Wed, Nov 21, 2018 at 03:47:25PM +0100, Herve Beraud wrote:
Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master.
Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release.
Okay that was a mistake, backporting a patch from master to stable that is known to break consumers, I admit this isn't explicitly called out in the stable policy but it is kind of the spirit of the stable policy. The quickest fix would be to revert 614489, release 1.31.7 and blacklist 1.31.6. Yes this leaves the High CPU usage bug open on rocky. That isn't great but it also isn't terrible.
When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version).
Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error.
Huzzah for cross-project gateing!
# Issue
Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements.
In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers.
The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue.
Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes.
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ?
Doing this is a violation of the stable policy. I get that the new feature is just a testing only fixture but that doesn't really matter it's still a feature. To use it consumers would need to raise the value for oslo.service in lower-constraints.txt which is a policy violation. The is an additional complication that this backport adds fixtures to requirements.txt for oslo.service, at the very least this would mean we're into a minor semver bump (1.32.X, which is already taken). This also means the vendors need to ensure that there is a 'fixtures' package available. Now I expect that all packagers have such a thing but there is a small chance that it exists as a build only package and needs to be exposed/published. We're previously said to vendors we wouldn't do that on stable branches.
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...) ? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
I think the only way forward is the revert, release and block path. The existing open reviews just add more policy violations. Yours Tony.
Le jeu. 22 nov. 2018 à 05:55, Tony Breeds <tony@bakeyournoodle.com> a écrit :
HI folks, I admit my initial response to this was mor pragmatic 'take the bakport' but as I thought it through I saw more problems with that approach.
On Wed, Nov 21, 2018 at 03:47:25PM +0100, Herve Beraud wrote:
Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master.
Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release.
Okay that was a mistake, backporting a patch from master to stable that is known to break consumers, I admit this isn't explicitly called out in the stable policy but it is kind of the spirit of the stable policy.
The quickest fix would be to revert 614489, release 1.31.7 and blacklist 1.31.6.
Yes this leaves the High CPU usage bug open on rocky. That isn't great but it also isn't terrible.
When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version).
Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error.
Huzzah for cross-project gateing!
# Issue
Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements.
In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers.
The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue.
Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes.
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ?
Doing this is a violation of the stable policy. I get that the new feature is just a testing only fixture but that doesn't really matter it's still a feature. To use it consumers would need to raise the value for oslo.service in lower-constraints.txt which is a policy violation.
The is an additional complication that this backport adds fixtures to requirements.txt for oslo.service, at the very least this would mean we're into a minor semver bump (1.32.X, which is already taken). This also means the vendors need to ensure that there is a 'fixtures' package available. Now I expect that all packagers have such a thing but there is a small chance that it exists as a build only package and needs to be exposed/published. We're previously said to vendors we wouldn't do that on stable branches.
I agree this solution come with a minor bump and issues to keep the semver coherent.
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example:
https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu... )
? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
I think the only way forward is the revert, release and block path. The existing open reviews just add more policy violations.
Yours Tony.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
I'm waiting for a consensus between us to update my reviews if needed or abandon them in the case we choose an another approach. Le jeu. 22 nov. 2018 à 09:31, Herve Beraud <hberaud@redhat.com> a écrit :
Le jeu. 22 nov. 2018 à 05:55, Tony Breeds <tony@bakeyournoodle.com> a écrit :
HI folks, I admit my initial response to this was mor pragmatic 'take the bakport' but as I thought it through I saw more problems with that approach.
On Wed, Nov 21, 2018 at 03:47:25PM +0100, Herve Beraud wrote:
Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master.
Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release.
Okay that was a mistake, backporting a patch from master to stable that is known to break consumers, I admit this isn't explicitly called out in the stable policy but it is kind of the spirit of the stable policy.
The quickest fix would be to revert 614489, release 1.31.7 and blacklist 1.31.6.
Yes this leaves the High CPU usage bug open on rocky. That isn't great but it also isn't terrible.
When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version).
Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error.
Huzzah for cross-project gateing!
# Issue
Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements.
In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers.
The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue.
Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes.
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ?
Doing this is a violation of the stable policy. I get that the new feature is just a testing only fixture but that doesn't really matter it's still a feature. To use it consumers would need to raise the value for oslo.service in lower-constraints.txt which is a policy violation.
The is an additional complication that this backport adds fixtures to requirements.txt for oslo.service, at the very least this would mean we're into a minor semver bump (1.32.X, which is already taken). This also means the vendors need to ensure that there is a 'fixtures' package available. Now I expect that all packagers have such a thing but there is a small chance that it exists as a build only package and needs to be exposed/published. We're previously said to vendors we wouldn't do that on stable branches.
I agree this solution come with a minor bump and issues to keep the semver coherent.
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example:
https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu... )
? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
I think the only way forward is the revert, release and block path. The existing open reviews just add more policy violations.
Yours Tony.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
On 22-11-18 15:55:07, Tony Breeds wrote:
HI folks, I admit my initial response to this was mor pragmatic 'take the bakport' but as I thought it through I saw more problems with that approach.
On Wed, Nov 21, 2018 at 03:47:25PM +0100, Herve Beraud wrote:
Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master.
Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release.
Okay that was a mistake, backporting a patch from master to stable that is known to break consumers, I admit this isn't explicitly called out in the stable policy but it is kind of the spirit of the stable policy.
The quickest fix would be to revert 614489, release 1.31.7 and blacklist 1.31.6.
Yes this leaves the High CPU usage bug open on rocky. That isn't great but it also isn't terrible.
Agreed, I'm in favor of doing this and asking the oslo.service team to rework the original fix on stable/rocky into something that doesn't break Nova CI.
When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version).
Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error.
Huzzah for cross-project gateing!
# Issue
Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements.
In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers.
The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue.
Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes.
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ?
Doing this is a violation of the stable policy. I get that the new feature is just a testing only fixture but that doesn't really matter it's still a feature. To use it consumers would need to raise the value for oslo.service in lower-constraints.txt which is a policy violation.
I'm being slightly dense this morning, I can't seem to find a reference to this lower-constraints.txt policy anywhere in the stable docs: https://docs.openstack.org/project-team-guide/stable-branches.html
The is an additional complication that this backport adds fixtures to requirements.txt for oslo.service, at the very least this would mean we're into a minor semver bump (1.32.X, which is already taken). This also means the vendors need to ensure that there is a 'fixtures' package available. Now I expect that all packagers have such a thing but there is a small chance that it exists as a build only package and needs to be exposed/published. We're previously said to vendors we wouldn't do that on stable branches.
I wouldn't be concerned with the impact on packagers and more that we can't use 1.32.x anyway thus blocking this approach.
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...) ? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
I think the only way forward is the revert, release and block path. The existing open reviews just add more policy violations.
Again I agree. Cheers, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
On Thu, Nov 22, 2018 at 09:46:51AM +0000, Lee Yarwood wrote:
Agreed, I'm in favor of doing this and asking the oslo.service team to rework the original fix on stable/rocky into something that doesn't break Nova CI.
Depending on how that looks maybe? I'm really not a fan of doing development on stable branches :( But I guess if the impact really is limited to CI it's probably okay. <snip>
I'm being slightly dense this morning, I can't seem to find a reference to this lower-constraints.txt policy anywhere in the stable docs:
https://docs.openstack.org/project-team-guide/stable-branches.html
Nope your density is the same as always ;P It isn't covered in the stable docs (but should be) The closest we have written is: https://docs.openstack.org/project-team-guide/dependency-management.html#sta... and even then it's only a passing comment about raising effective minimums. Those docs are outdated WRT the new lower-constraints.txt and extended maintenance work. <snip> Yours Tony.
Just a tiny report to summarize the situation to offer a big picture and to analyze proposed solutions since peoples have submits new patches and some other people have shared their opinions. Each solution contains the impacted projects, the related gerrit reviews, pro/cons. ### 1 - Backport master changer and also backport new feature into a stable branch impacted projects: oslo.service, nova related reviews: - https://review.openstack.org/617989 - https://review.openstack.org/619022 - https://review.openstack.org/619019 pros: - follow the master changes - just use a backport cons: - backport a new feature into a stable branch and then this a violation of the stable policy - introduce a semver minor bump and a version 1.32 already exist - won't be able to use either an older nova with a newer oslo.service - increasing lower-constraints is not allowed on stable branches - vendors need to ensure that there is a 'fixture' package available ### 2 - Update the nova test to mock the new private interface impacted projects: nova reviews: - https://review.openstack.org/619246 pros: - straightforward cons: - still mocking a private inferface - stable only patches ### 3 - Maintain a private interface for ThreadingEvent only on stable/rocky impacted projects: oslo.service related reviews: - https://review.openstack.org/619342/ pros: - straightforward cons: - Changing the loopingcall module semantics as it is different type - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior ### 4 - Don't use private interface in oslo.service impacted projects: nova related reviews: - https://review.openstack.org/#/c/619360/ pros: - straightforward cons: - this could work but it is not the same sematics as before as the type has changed - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior ### 5 - Leave the CPU bug open on rocky impacted projects: oslo.service related reviews: - pros: - Nova project doesn't impacted cons: - reintroduce the CPU issue ### 6 - Revert CPU fix and totally rework it into someting that doesn't break the Nova CI impacted projects: oslo.service related reviews: - pros: - Nova project doesn't impacted cons: - potentially introduce more rewrites in the futures that depends on fix on oslo.service loopingcall master - stable only patches - increase potential backport difficulties on oslo.service upstream and downstream - increase work load on upstream and downstream Personally: - I prefer the #4 or the #2 and they seem to be smooth changes without earthquake or anything like this - the #6 seems to be the worst solution on my side - the #1 introduce semver issue and policy violations so I don't think we can continue with it Thoughts? I hope this summarize help you to have a better overview :) I hope I have not forgotten anything and if so I apologize in advance. Le mer. 21 nov. 2018 à 15:47, Herve Beraud <hberaud@redhat.com> a écrit :
Hey all!
Here is a thread to coordinate all the teams (oslo, nova, stable, requirements) working on the update of the oslo.service constraint in the Rocky requirements.
# Summary
Usage of threading event with eventlet caused inefficient code (causing many useless system calls and high CPU usage). This issue was already fixed on oslo.service master and we also want to fix it in stable/rocky.
Our main issue is how to fix the high CPU usage on stable/rocky without break the nova CI.
Indeed, we already have backported the eventlet related fix to oslo.service but this fix requires also a nova update to avoid nova CI errors due to threading removal on oslo.service that introduce the nova CI errors.
A fix was proposed and merged on oslo.service master to introduce a new feature (fixture) that avoid the nova CI errors, but backporting the master fix to Rocky introduces a new feature into a stable branch so this is also an issue.
So we need to discuss with all the teams to find a proper solution.
# History
A few weeks ago this issue was opened on oslo.service ( https://bugs.launchpad.net/oslo.service/+bug/1798774) and it was fixed by this submited patch on the master branch ( https://review.openstack.org/#/c/611807/ ).
This change use the proper event primitive to fix the performance issue.
A new version of oslo.service was released (1.32.1)
Since these changes was introduced into oslo.service master, nova facing some issues into the master CI process, due to the threading changes, and they was fixed by these patches ( https://review.openstack.org/#/c/615724/, https://review.openstack.org/#/c/617989/ ) into master.
Few weeks ago I have backport to oslo.service some changes ( https://review.openstack.org/#/c/614489/ ) from master to stable/rocky to also fix the problem in the rocky release.
When this backport was merged we have created a new release of oslo.service (1.31.6) ( https://review.openstack.org/#/c/616505/ ) (stable/rocky version).
Then the openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ so this patch is currently blocked to avoid nova CI error.
# Issue
Since the oslo.services threading changes were backported to rocky we risk to faces the same issues inside the nova rocky CI if we update the requirements.
In parallel in oslo.service we have started to backport a new patch who introduces fixture ( https://review.openstack.org/#/c/617989/ ) from master to rocky, and also we start to backport on nova rocky branch ( https://review.openstack.org/619019, https://review.openstack.org/619022 ) patches who use oslo.service.fixture and who solve the nova CI issue. The patch on oslo.service exposes a public oslo_service.fixture.SleepFixture for this purpose. It can be maintained opaquely as internals change without affecting its consumers.
The main problem is that the patch bring a new functionality to a stable branch (oslo.service rocky) but this patch help to fix the nova issue.
Also openstack proposal bot submit a patch to requirements on stable rocky to update the oslo.service version with the latest version (1.31.6) but if we'll use it we'll then break the CI https://review.openstack.org/#/c/618834/ since the oslo service 1.31.6 is incompatible with novas stable rocky unittest due to the threading changes.
# Questions and proposed solutions
This thread try to summarize the current situation.
We need to find how to be able to proceed, so this thread aim to allow to discuss between team to find the best way to fix.
1. Do we need to continue to try to backport fixture on oslo.service to fix the CI problem (https://review.openstack.org/#/c/617989/) ?
2. Do we need to find an another approach like mocking oslo.service.loopingcall._Event.wait in nova instead of mocking oslo_service.loopingcall._ThreadingEvent.wait (example: https://review.openstack.org/#/c/616697/2/nova/tests/unit/compute/test_compu...) ? This is only a fix on the nova side and it allows us to update oslo.service requirements and allows us to fix the high CPU usage issue. I've submit this patch (https://review.openstack.org/619246) who implement the description above.
Personaly I think we need to find an another approach like the mocking remplacement (c.f 2).
We need to decide which way we use and to discuss about other solutions.
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
On Thu, Nov 22, 2018 at 05:58:08PM +0100, Herve Beraud wrote:
Just a tiny report to summarize the situation to offer a big picture and to analyze proposed solutions since peoples have submits new patches and some other people have shared their opinions.
Each solution contains the impacted projects, the related gerrit reviews, pro/cons.
### 1 - Backport master changer and also backport new feature into a stable branch impacted projects: oslo.service, nova related reviews: - https://review.openstack.org/617989 - https://review.openstack.org/619022 - https://review.openstack.org/619019
pros: - follow the master changes - just use a backport cons: - backport a new feature into a stable branch and then this a violation of the stable policy - introduce a semver minor bump and a version 1.32 already exist - won't be able to use either an older nova with a newer oslo.service - increasing lower-constraints is not allowed on stable branches - vendors need to ensure that there is a 'fixture' package available
My position is that this is a non-starter.
### 2 - Update the nova test to mock the new private interface impacted projects: nova reviews: - https://review.openstack.org/619246
pros: - straightforward cons: - still mocking a private inferface - stable only patches
This one is also a stable policy issue in that it forces nova to require a newer version of oslo.service and IIUC will never land as this approach can never be compatible with both oslo.service < 1.31.6 and 1.31.6
### 3 - Maintain a private interface for ThreadingEvent only on stable/rocky impacted projects: oslo.service related reviews: - https://review.openstack.org/619342/
pros: - straightforward cons: - Changing the loopingcall module semantics as it is different type - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior
A variation of this (without adding the debtcollector requirement) might work as IIUC all versions of oslo.service < 1.32.0 (except 1.31.6) would work and doesn't introduce any new policy violations. Adding debcollector is great but introduces at least the semver issues from option #1
### 4 - Don't use private interface in oslo.service impacted projects: nova related reviews: - https://review.openstack.org/#/c/619360/
pros: - straightforward cons: - this could work but it is not the same sematics as before as the type has changed - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior
I think this is more or less the same as Option #3 in terms of impact. If that's right it could be acceptable.
### 5 - Leave the CPU bug open on rocky impacted projects: oslo.service related reviews: -
pros: - Nova project doesn't impacted cons: - reintroduce the CPU issue
I think if #3, or #4 works we can avoid the revert part of this but regardless we're going to need to block 1.31.6 so I've created: https://review.openstack.org/#/c/619655/ ; and abandoned https://review.openstack.org/#/c/618834/
### 6 - Revert CPU fix and totally rework it into someting that doesn't break the Nova CI impacted projects: oslo.service related reviews: -
pros: - Nova project doesn't impacted cons: - potentially introduce more rewrites in the futures that depends on fix on oslo.service loopingcall master - stable only patches - increase potential backport difficulties on oslo.service upstream and downstream - increase work load on upstream and downstream
Personally: - I prefer the #4 or the #2 and they seem to be smooth changes without earthquake or anything like this - the #6 seems to be the worst solution on my side - the #1 introduce semver issue and policy violations so I don't think we can continue with it
I'm not certain how this is really different to the options already presented but I agree anything substantial on stable branches is pretty much a non-starter.
Thoughts?
My preference is for a modified version of #3, unless I'm beaten to it I'll create a matrix that tests old and new olso.service against old and new nova to ensure that we're not missing something.
I hope this summarize help you to have a better overview :)
Thanks!
I hope I have not forgotten anything and if so I apologize in advance.
Really appreciate you keeping this on track and pulling the information together. Thanks! Tony.
On 22/11/18 6:43 PM, Tony Breeds wrote:
### 3 - Maintain a private interface for ThreadingEvent only on stable/rocky impacted projects: oslo.service related reviews: -https://review.openstack.org/619342/
pros: - straightforward cons: - Changing the loopingcall module semantics as it is different type - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior A variation of this (without adding the debtcollector requirement) might work as IIUC all versions of oslo.service < 1.32.0 (except 1.31.6) would work and doesn't introduce any new policy violations.
Adding debcollector is great but introduces at least the semver issues from option #1
Hey Tony, can you explain the debtcollector issue a bit more? Is it just that we generally promise to not adding anything to requirements.txt on stable branches? In this case, debtcollector, appears to already be a transitive dependency of something that oslo.service already requires (it appears in lower-constraints.txt, at least), so I was assuming that there wouldn't be any real-world consequences. Does that change things? TBH it would be trivial to do something equivalent without adding the new requirement, so I guess the easiest thing is if I just update the patch to do that.
### 4 - Don't use private interface in oslo.service impacted projects: nova related reviews: -https://review.openstack.org/#/c/619360/
pros: - straightforward cons: - this could work but it is not the same sematics as before as the type has changed - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior I think this is more or less the same as Option #3 in terms of impact. If that's right it could be acceptable.
It's slightly higher impact, because you wouldn't be able to upgrade oslo.service past 1.31.5 and still run the unit tests on old versions of Nova. Whereas with option #3 you could just upgrade oslo.service to 1.31.7, or just upgrade Nova, and either way everything would work. Not that we shouldn't do this *as well*, but IMHO we still need to do #3.
### 5 - Leave the CPU bug open on rocky impacted projects: oslo.service related reviews: -
pros: - Nova project doesn't impacted cons: - reintroduce the CPU issue
I think if #3, or #4 works we can avoid the revert part of this but regardless we're going to need to block 1.31.6 so I've created:
https://review.openstack.org/#/c/619655/ ; and abandonedhttps://review.openstack.org/#/c/618834/
I think with #4 we wouldn't need to block, but I agree that blocking 1.31.6 and merging some version of #3 is the better solution.
Personally: - I prefer the #4 or the #2 and they seem to be smooth changes without earthquake or anything like this - the #6 seems to be the worst solution on my side - the #1 introduce semver issue and policy violations so I don't think we can continue with it I'm not certain how this is really different to the options already presented but I agree anything substantial on stable branches is pretty much a non-starter.
Thoughts? My preference is for a modified version of #3, unless I'm beaten to it
+1 cheers, Zane.
On 11/26/18 9:47 AM, Zane Bitter wrote:
On 22/11/18 6:43 PM, Tony Breeds wrote:
### 3 - Maintain a private interface for ThreadingEvent only on stable/rocky impacted projects: oslo.service related reviews: -https://review.openstack.org/619342/
pros: - straightforward cons: - Changing the loopingcall module semantics as it is different type - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior A variation of this (without adding the debtcollector requirement) might work as IIUC all versions of oslo.service < 1.32.0 (except 1.31.6) would work and doesn't introduce any new policy violations.
Adding debcollector is great but introduces at least the semver issues from option #1
Hey Tony, can you explain the debtcollector issue a bit more? Is it just that we generally promise to not adding anything to requirements.txt on stable branches? In this case, debtcollector, appears to already be a transitive dependency of something that oslo.service already requires (it appears in lower-constraints.txt, at least), so I was assuming that there wouldn't be any real-world consequences. Does that change things?
Technically adding a new dep requires a minor version update per our release guidelines. I guess it gets a little fuzzy when you're adding a dep that's already part of the transitive set, but I'm guessing that's what Tony was referring to. That said, as I mentioned on the review I don't feel any particular need to debtcollector this. It's in a private module that nobody should ever have been using and we're only doing this because we prioritize fixing the bug over absolute API purity. :-) In any case, I'm +1 on doing this to avoid creating weird version dependencies on a stable branch.
TBH it would be trivial to do something equivalent without adding the new requirement, so I guess the easiest thing is if I just update the patch to do that.
### 4 - Don't use private interface in oslo.service impacted projects: nova related reviews: -https://review.openstack.org/#/c/619360/
pros: - straightforward cons: - this could work but it is not the same sematics as before as the type has changed - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior I think this is more or less the same as Option #3 in terms of impact. If that's right it could be acceptable.
It's slightly higher impact, because you wouldn't be able to upgrade oslo.service past 1.31.5 and still run the unit tests on old versions of Nova. Whereas with option #3 you could just upgrade oslo.service to 1.31.7, or just upgrade Nova, and either way everything would work.
Not that we shouldn't do this *as well*, but IMHO we still need to do #3.
I like doing this as well. If we need an argument as to why it's okay to do this stable-only patch, it would be that this is essentially a backport of the original Nova fix proposed before we decided to do the fixture. In retrospect maybe we should have gone ahead and merged that simple, backportable fix and done the fixture as a followup, but in spirit this has the same effect. -Ben
FYI Introducing oslo.service 1.31.7 with the latest merged changes. These refers to #3. https://review.openstack.org/#/c/620913/ Le mer. 28 nov. 2018 à 00:31, Ben Nemec <openstack@nemebean.com> a écrit :
On 11/26/18 9:47 AM, Zane Bitter wrote:
On 22/11/18 6:43 PM, Tony Breeds wrote:
### 3 - Maintain a private interface for ThreadingEvent only on stable/rocky impacted projects: oslo.service related reviews: -https://review.openstack.org/619342/
pros: - straightforward cons: - Changing the loopingcall module semantics as it is different type - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior A variation of this (without adding the debtcollector requirement) might work as IIUC all versions of oslo.service < 1.32.0 (except 1.31.6) would work and doesn't introduce any new policy violations.
Adding debcollector is great but introduces at least the semver issues from option #1
Hey Tony, can you explain the debtcollector issue a bit more? Is it just that we generally promise to not adding anything to requirements.txt on stable branches? In this case, debtcollector, appears to already be a transitive dependency of something that oslo.service already requires (it appears in lower-constraints.txt, at least), so I was assuming that there wouldn't be any real-world consequences. Does that change things?
Technically adding a new dep requires a minor version update per our release guidelines. I guess it gets a little fuzzy when you're adding a dep that's already part of the transitive set, but I'm guessing that's what Tony was referring to.
That said, as I mentioned on the review I don't feel any particular need to debtcollector this. It's in a private module that nobody should ever have been using and we're only doing this because we prioritize fixing the bug over absolute API purity. :-)
In any case, I'm +1 on doing this to avoid creating weird version dependencies on a stable branch.
TBH it would be trivial to do something equivalent without adding the new requirement, so I guess the easiest thing is if I just update the patch to do that.
### 4 - Don't use private interface in oslo.service impacted projects: nova related reviews: -https://review.openstack.org/#/c/619360/
pros: - straightforward cons: - this could work but it is not the same sematics as before as the type has changed - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior I think this is more or less the same as Option #3 in terms of impact. If that's right it could be acceptable.
It's slightly higher impact, because you wouldn't be able to upgrade oslo.service past 1.31.5 and still run the unit tests on old versions of Nova. Whereas with option #3 you could just upgrade oslo.service to 1.31.7, or just upgrade Nova, and either way everything would work.
Not that we shouldn't do this *as well*, but IMHO we still need to do #3.
I like doing this as well. If we need an argument as to why it's okay to do this stable-only patch, it would be that this is essentially a backport of the original Nova fix proposed before we decided to do the fixture. In retrospect maybe we should have gone ahead and merged that simple, backportable fix and done the fixture as a followup, but in spirit this has the same effect.
-Ben
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
For resume we decide to use: - #4 (https://review.openstack.org/#/c/619360/) - #3 (https://review.openstack.org/#/c/619342/) Also I've propose the oslo.service 1.31.7 since #3 have been merged ( https://review.openstack.org/#/c/619342/) : https://review.openstack.org/#/c/620913/ Le jeu. 29 nov. 2018 à 14:56, Herve Beraud <hberaud@redhat.com> a écrit :
FYI Introducing oslo.service 1.31.7 with the latest merged changes. These refers to #3. https://review.openstack.org/#/c/620913/
Le mer. 28 nov. 2018 à 00:31, Ben Nemec <openstack@nemebean.com> a écrit :
On 11/26/18 9:47 AM, Zane Bitter wrote:
On 22/11/18 6:43 PM, Tony Breeds wrote:
### 3 - Maintain a private interface for ThreadingEvent only on stable/rocky impacted projects: oslo.service related reviews: -https://review.openstack.org/619342/
pros: - straightforward cons: - Changing the loopingcall module semantics as it is different type - stable only patches - misunderstoud service customer between Threading, eventlet, etc.. and master behavior A variation of this (without adding the debtcollector requirement) might work as IIUC all versions of oslo.service < 1.32.0 (except 1.31.6) would work and doesn't introduce any new policy violations.
Adding debcollector is great but introduces at least the semver issues from option #1
Hey Tony, can you explain the debtcollector issue a bit more? Is it just that we generally promise to not adding anything to requirements.txt on stable branches? In this case, debtcollector, appears to already be a transitive dependency of something that oslo.service already requires (it appears in lower-constraints.txt, at least), so I was assuming that there wouldn't be any real-world consequences. Does that change things?
Technically adding a new dep requires a minor version update per our release guidelines. I guess it gets a little fuzzy when you're adding a dep that's already part of the transitive set, but I'm guessing that's what Tony was referring to.
That said, as I mentioned on the review I don't feel any particular need to debtcollector this. It's in a private module that nobody should ever have been using and we're only doing this because we prioritize fixing the bug over absolute API purity. :-)
In any case, I'm +1 on doing this to avoid creating weird version dependencies on a stable branch.
TBH it would be trivial to do something equivalent without adding the new requirement, so I guess the easiest thing is if I just update the patch to do that.
### 4 - Don't use private interface in oslo.service impacted projects: nova related reviews: -https://review.openstack.org/#/c/619360/
pros: - straightforward cons: - this could work but it is not the same sematics as before as the type has changed - stable only patches - misunderstoud service customer between Threading, eventlet, etc..
and
master behavior I think this is more or less the same as Option #3 in terms of impact. If that's right it could be acceptable.
It's slightly higher impact, because you wouldn't be able to upgrade oslo.service past 1.31.5 and still run the unit tests on old versions of Nova. Whereas with option #3 you could just upgrade oslo.service to 1.31.7, or just upgrade Nova, and either way everything would work.
Not that we shouldn't do this *as well*, but IMHO we still need to do #3.
I like doing this as well. If we need an argument as to why it's okay to do this stable-only patch, it would be that this is essentially a backport of the original Nova fix proposed before we decided to do the fixture. In retrospect maybe we should have gone ahead and merged that simple, backportable fix and done the fixture as a followup, but in spirit this has the same effect.
-Ben
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE-----
wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
participants (8)
-
Ben Nemec
-
Doug Hellmann
-
Eric Fried
-
Herve Beraud
-
Lee Yarwood
-
Sean Mooney
-
Tony Breeds
-
Zane Bitter