[all][oslo][ironic][tc] Eventlet version bumps (topic:bump-eventlet in gerrit)
Hi all,
https://review.opendev.org/q/topic:%22bump-eventlet%22 was pushed recently, to bump eventlet versions to a consistent lower bound across OpenStack projects. I saw some concerns about this across various OpenStack IRC channels, and wanted to start a larger conversation around the changes rather than having what would be my third separate small conversation about it :). I cannot speak to exactly why Herve pushed these changes, but I think they are a good idea and should be merged.
If you're unaware or haven't read up on the context; versions of eventlet before 0.34 were released with minimal/failing CI, and done so with known bugs in many cases. A group of developers has taken the last months to improve the state of eventlet considerably, enhancing and improving CI, discovering and fixing known issues and enabling support for Python 3.12. From my perspective; this is a significant enough improvement to warrant our community moving forward generally[0] with setting a new higher low bound for eventlet -- the versions that are maintained.
I do think that we can do a better job of advertising the value of this change in gerrit to other OpenStack contributors; simply stating it's for python 3.12 support downplays the significant improvements made across that project and the overall value of landing the version bump. I am unsure if we should advertise the improvements in a user/operator facing manner; I lean towards not doing so.
From a single project perspective (with my Ironic PTL hat on): I have no desire to troubleshoot bugs in Ironic, or related projects, that only occur in eventlet versions prior to 0.34. This leads me to believe landing those changes in Ironic are the good path forward, and I'll be advocating for that inside Ironic.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
0: I'll note this should not be read as a suggestion of a mandate; only that it's a good idea to bump this requirement in OpenStack projects and should be done unless cores for that project see a specific reason why not.
Despite my efforts to move some of the discussion to be asynchronous, the conversation continues to progress synchronously in IRC.
I would strongly suggest based on the outcomes of some of those conversations (logs at https://meetings.opendev.org/irclogs/%23openstack-tc/%23openstack-tc.2024-01... ) that projects hold off on merging these changes until a larger discussion happens.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
On Wed, Jan 17, 2024 at 9:39 AM Jay Faulkner jay@gr-oss.io wrote:
Hi all,
https://review.opendev.org/q/topic:%22bump-eventlet%22 was pushed recently, to bump eventlet versions to a consistent lower bound across OpenStack projects. I saw some concerns about this across various OpenStack IRC channels, and wanted to start a larger conversation around the changes rather than having what would be my third separate small conversation about it :). I cannot speak to exactly why Herve pushed these changes, but I think they are a good idea and should be merged.
If you're unaware or haven't read up on the context; versions of eventlet before 0.34 were released with minimal/failing CI, and done so with known bugs in many cases. A group of developers has taken the last months to improve the state of eventlet considerably, enhancing and improving CI, discovering and fixing known issues and enabling support for Python 3.12. From my perspective; this is a significant enough improvement to warrant our community moving forward generally[0] with setting a new higher low bound for eventlet -- the versions that are maintained.
I do think that we can do a better job of advertising the value of this change in gerrit to other OpenStack contributors; simply stating it's for python 3.12 support downplays the significant improvements made across that project and the overall value of landing the version bump. I am unsure if we should advertise the improvements in a user/operator facing manner; I lean towards not doing so.
From a single project perspective (with my Ironic PTL hat on): I have no desire to troubleshoot bugs in Ironic, or related projects, that only occur in eventlet versions prior to 0.34. This leads me to believe landing those changes in Ironic are the good path forward, and I'll be advocating for that inside Ironic.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
0: I'll note this should not be read as a suggestion of a mandate; only that it's a good idea to bump this requirement in OpenStack projects and should be done unless cores for that project see a specific reason why not.
Hey all,
Concerning my upper-constraints bump of eventlet [1] it was a simple bump like all those that we do regularly bump all kinds of libs to their latest pypi versions. Eventlet got new pypi versions, I simply reflected them on Openstack, as usual, as made with our automated updates for all our requirements.
Concerning that list of patches to update min version in requirements.txt [2], even without these patches, the UC upgrade [1] would have allowed Openstack deliverables to pull the latest version.
By doing that series of requirements files updates, my main goal was that I wanted to avoid tons of broken install related to pip resolver issues between all our deliverables by uniformizing all our requirements files ASAP. This list of patches contains libraries and services of all kinds. Many services use libraries from their team scope and from outside their team scope (oslo, etc...). Keeping them in different states could lead to so many potential conflicts to resolve. I was thinking uniformity would simplify our life.
To illustrate my previous goal, here is a similar scenario, with a pip resolver issue, between os-brick, nova, and cinder: - https://opendev.org/openstack/nova/commit/e5b264bf7b28d3987b4ea30c61da4262ae... - https://opendev.org/openstack/cinder/commit/a74513db9a5d8c0fcc92614bd95262ec... - https://opendev.org/openstack/os-brick/commit/9a59547a
The previous example is between 3 deliverables. I think that, without uniformizing our requirements file, we currently own at least 76² opportunities to face the same issue.
We should also notice that similar patches were made in the past, As an example, swift made similar things in the past for more or less the same reasons [3]. Or again with Ironic [4], oslo.service [5], neutron [6].
A side goal was also to limit the requirements file to the recent eventlet versions who are actively maintained and where Python 3.12 is supported, to prepare for 2024.2/Dalmatian ASAP. Using it as soon as possible in real situations, would have allowed us to observe possible problems early. The UC upgrade allowed us to catch a bug in eventlet within swift.
To close the requirements file update topic, the situation with bugs would have been the same without upgrading all these requirements files and simply by upgrading UC, but we would have left room to future pip resolver issues. Now that eventlet has been downgraded, that's not an issue anymore.
I should have communicated more broadly before submitting this series of patches. I apologize for the noise generated by my previous series of patches.
To finish, we currently prepare a new version of eventlet with a patch that will fix the swift bug https://github.com/eventlet/eventlet/issues/897
Thanks for your attention
[1] https://review.opendev.org/c/openstack/requirements/+/904147 [2] https://review.opendev.org/q/topic:%22bump-eventlet%22 [3] https://opendev.org/openstack/swift/commit/0ebfeddf65adb473fda2adfd1705534b6... [4] https://opendev.org/openstack/ironic/commit/69bdbfb6e97e53998d5a30c60f621131... [5] https://opendev.org/openstack/oslo.service/commit/7ec187317167c514f92162be3e... [6] https://opendev.org/openstack/neutron/commit/e41d82e37444418f2a3ef770c9def01...
Le mer. 17 janv. 2024 à 19:58, Jay Faulkner jay@gr-oss.io a écrit :
Despite my efforts to move some of the discussion to be asynchronous, the conversation continues to progress synchronously in IRC.
I would strongly suggest based on the outcomes of some of those conversations (logs at https://meetings.opendev.org/irclogs/%23openstack-tc/%23openstack-tc.2024-01... ) that projects hold off on merging these changes until a larger discussion happens.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
On Wed, Jan 17, 2024 at 9:39 AM Jay Faulkner jay@gr-oss.io wrote:
Hi all,
https://review.opendev.org/q/topic:%22bump-eventlet%22 was pushed recently, to bump eventlet versions to a consistent lower bound across OpenStack projects. I saw some concerns about this across various OpenStack IRC channels, and wanted to start a larger conversation around the changes rather than having what would be my third separate small conversation about it :). I cannot speak to exactly why Herve pushed these changes, but I think they are a good idea and should be merged.
If you're unaware or haven't read up on the context; versions of eventlet before 0.34 were released with minimal/failing CI, and done so with known bugs in many cases. A group of developers has taken the last months to improve the state of eventlet considerably, enhancing and improving CI, discovering and fixing known issues and enabling support for Python 3.12. From my perspective; this is a significant enough improvement to warrant our community moving forward generally[0] with setting a new higher low bound for eventlet -- the versions that are maintained.
I do think that we can do a better job of advertising the value of this change in gerrit to other OpenStack contributors; simply stating it's for python 3.12 support downplays the significant improvements made across that project and the overall value of landing the version bump. I am unsure if we should advertise the improvements in a user/operator facing manner; I lean towards not doing so.
From a single project perspective (with my Ironic PTL hat on): I have no desire to troubleshoot bugs in Ironic, or related projects, that only occur in eventlet versions prior to 0.34. This leads me to believe landing those changes in Ironic are the good path forward, and I'll be advocating for that inside Ironic.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
0: I'll note this should not be read as a suggestion of a mandate; only that it's a good idea to bump this requirement in OpenStack projects and should be done unless cores for that project see a specific reason why not.
Hi all,
FYI, last week I released a version of eventlet (0.35.0) [1][2][3]. For now we did not notice new issues related to this new version. This morning I proposed a new eventlet bump against our general upper-constraints [4].
This new version is an important one. It introduces the capabilities to run eventlet and asyncio in the same process. It also solves previously discussed issues.
Do not hesitate to continue the discussion directly on the requirements patch [4].
Thanks for your attention.
[1] https://github.com/eventlet/eventlet/issues/897 [2] https://github.com/eventlet/eventlet/releases/tag/v0.35.0 [3] https://pypi.org/project/eventlet/0.35.0/ [4] https://review.opendev.org/c/openstack/requirements/+/907048
Le jeu. 18 janv. 2024 à 15:00, Herve Beraud hberaud@redhat.com a écrit :
Hey all,
Concerning my upper-constraints bump of eventlet [1] it was a simple bump like all those that we do regularly bump all kinds of libs to their latest pypi versions. Eventlet got new pypi versions, I simply reflected them on Openstack, as usual, as made with our automated updates for all our requirements.
Concerning that list of patches to update min version in requirements.txt [2], even without these patches, the UC upgrade [1] would have allowed Openstack deliverables to pull the latest version.
By doing that series of requirements files updates, my main goal was that I wanted to avoid tons of broken install related to pip resolver issues between all our deliverables by uniformizing all our requirements files ASAP. This list of patches contains libraries and services of all kinds. Many services use libraries from their team scope and from outside their team scope (oslo, etc...). Keeping them in different states could lead to so many potential conflicts to resolve. I was thinking uniformity would simplify our life.
To illustrate my previous goal, here is a similar scenario, with a pip resolver issue, between os-brick, nova, and cinder:
https://opendev.org/openstack/nova/commit/e5b264bf7b28d3987b4ea30c61da4262ae...
https://opendev.org/openstack/cinder/commit/a74513db9a5d8c0fcc92614bd95262ec...
The previous example is between 3 deliverables. I think that, without uniformizing our requirements file, we currently own at least 76² opportunities to face the same issue.
We should also notice that similar patches were made in the past, As an example, swift made similar things in the past for more or less the same reasons [3]. Or again with Ironic [4], oslo.service [5], neutron [6].
A side goal was also to limit the requirements file to the recent eventlet versions who are actively maintained and where Python 3.12 is supported, to prepare for 2024.2/Dalmatian ASAP. Using it as soon as possible in real situations, would have allowed us to observe possible problems early. The UC upgrade allowed us to catch a bug in eventlet within swift.
To close the requirements file update topic, the situation with bugs would have been the same without upgrading all these requirements files and simply by upgrading UC, but we would have left room to future pip resolver issues. Now that eventlet has been downgraded, that's not an issue anymore.
I should have communicated more broadly before submitting this series of patches. I apologize for the noise generated by my previous series of patches.
To finish, we currently prepare a new version of eventlet with a patch that will fix the swift bug https://github.com/eventlet/eventlet/issues/897
Thanks for your attention
[1] https://review.opendev.org/c/openstack/requirements/+/904147 [2] https://review.opendev.org/q/topic:%22bump-eventlet%22 [3] https://opendev.org/openstack/swift/commit/0ebfeddf65adb473fda2adfd1705534b6... [4] https://opendev.org/openstack/ironic/commit/69bdbfb6e97e53998d5a30c60f621131... [5] https://opendev.org/openstack/oslo.service/commit/7ec187317167c514f92162be3e... [6] https://opendev.org/openstack/neutron/commit/e41d82e37444418f2a3ef770c9def01...
Le mer. 17 janv. 2024 à 19:58, Jay Faulkner jay@gr-oss.io a écrit :
Despite my efforts to move some of the discussion to be asynchronous, the conversation continues to progress synchronously in IRC.
I would strongly suggest based on the outcomes of some of those conversations (logs at https://meetings.opendev.org/irclogs/%23openstack-tc/%23openstack-tc.2024-01... ) that projects hold off on merging these changes until a larger discussion happens.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
On Wed, Jan 17, 2024 at 9:39 AM Jay Faulkner jay@gr-oss.io wrote:
Hi all,
https://review.opendev.org/q/topic:%22bump-eventlet%22 was pushed recently, to bump eventlet versions to a consistent lower bound across OpenStack projects. I saw some concerns about this across various OpenStack IRC channels, and wanted to start a larger conversation around the changes rather than having what would be my third separate small conversation about it :). I cannot speak to exactly why Herve pushed these changes, but I think they are a good idea and should be merged.
If you're unaware or haven't read up on the context; versions of eventlet before 0.34 were released with minimal/failing CI, and done so with known bugs in many cases. A group of developers has taken the last months to improve the state of eventlet considerably, enhancing and improving CI, discovering and fixing known issues and enabling support for Python 3.12. From my perspective; this is a significant enough improvement to warrant our community moving forward generally[0] with setting a new higher low bound for eventlet -- the versions that are maintained.
I do think that we can do a better job of advertising the value of this change in gerrit to other OpenStack contributors; simply stating it's for python 3.12 support downplays the significant improvements made across that project and the overall value of landing the version bump. I am unsure if we should advertise the improvements in a user/operator facing manner; I lean towards not doing so.
From a single project perspective (with my Ironic PTL hat on): I have no desire to troubleshoot bugs in Ironic, or related projects, that only occur in eventlet versions prior to 0.34. This leads me to believe landing those changes in Ironic are the good path forward, and I'll be advocating for that inside Ironic.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
0: I'll note this should not be read as a suggestion of a mandate; only that it's a good idea to bump this requirement in OpenStack projects and should be done unless cores for that project see a specific reason why not.
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
The 0.34.x release caused issues with documentation builds, which are not tested in requirements. I'm unsure at this point in the cycle if it's wise to bump this constraint (even though it's obviously a massive improvement) -- but after testing, I no longer have specific objections for projects I work on closely.
It appears https://github.com/eventlet/eventlet/pull/866/files fixed the documentation builds; 0.35.0 gets a clean pass on ironic-inspector docs when run locally.
Thanks, Jay
On Mon, Jan 29, 2024 at 8:22 AM Herve Beraud hberaud@redhat.com wrote:
Hi all,
FYI, last week I released a version of eventlet (0.35.0) [1][2][3]. For now we did not notice new issues related to this new version. This morning I proposed a new eventlet bump against our general upper-constraints [4].
This new version is an important one. It introduces the capabilities to run eventlet and asyncio in the same process. It also solves previously discussed issues.
Do not hesitate to continue the discussion directly on the requirements patch [4].
Thanks for your attention.
[1] https://github.com/eventlet/eventlet/issues/897 [2] https://github.com/eventlet/eventlet/releases/tag/v0.35.0 [3] https://pypi.org/project/eventlet/0.35.0/ [4] https://review.opendev.org/c/openstack/requirements/+/907048
Le jeu. 18 janv. 2024 à 15:00, Herve Beraud hberaud@redhat.com a écrit :
Hey all,
Concerning my upper-constraints bump of eventlet [1] it was a simple bump like all those that we do regularly bump all kinds of libs to their latest pypi versions. Eventlet got new pypi versions, I simply reflected them on Openstack, as usual, as made with our automated updates for all our requirements.
Concerning that list of patches to update min version in requirements.txt [2], even without these patches, the UC upgrade [1] would have allowed Openstack deliverables to pull the latest version.
By doing that series of requirements files updates, my main goal was that I wanted to avoid tons of broken install related to pip resolver issues between all our deliverables by uniformizing all our requirements files ASAP. This list of patches contains libraries and services of all kinds. Many services use libraries from their team scope and from outside their team scope (oslo, etc...). Keeping them in different states could lead to so many potential conflicts to resolve. I was thinking uniformity would simplify our life.
To illustrate my previous goal, here is a similar scenario, with a pip resolver issue, between os-brick, nova, and cinder:
https://opendev.org/openstack/nova/commit/e5b264bf7b28d3987b4ea30c61da4262ae...
https://opendev.org/openstack/cinder/commit/a74513db9a5d8c0fcc92614bd95262ec...
The previous example is between 3 deliverables. I think that, without uniformizing our requirements file, we currently own at least 76² opportunities to face the same issue.
We should also notice that similar patches were made in the past, As an example, swift made similar things in the past for more or less the same reasons [3]. Or again with Ironic [4], oslo.service [5], neutron [6].
A side goal was also to limit the requirements file to the recent eventlet versions who are actively maintained and where Python 3.12 is supported, to prepare for 2024.2/Dalmatian ASAP. Using it as soon as possible in real situations, would have allowed us to observe possible problems early. The UC upgrade allowed us to catch a bug in eventlet within swift.
To close the requirements file update topic, the situation with bugs would have been the same without upgrading all these requirements files and simply by upgrading UC, but we would have left room to future pip resolver issues. Now that eventlet has been downgraded, that's not an issue anymore.
I should have communicated more broadly before submitting this series of patches. I apologize for the noise generated by my previous series of patches.
To finish, we currently prepare a new version of eventlet with a patch that will fix the swift bug https://github.com/eventlet/eventlet/issues/897
Thanks for your attention
[1] https://review.opendev.org/c/openstack/requirements/+/904147 [2] https://review.opendev.org/q/topic:%22bump-eventlet%22 [3] https://opendev.org/openstack/swift/commit/0ebfeddf65adb473fda2adfd1705534b6... [4] https://opendev.org/openstack/ironic/commit/69bdbfb6e97e53998d5a30c60f621131... [5] https://opendev.org/openstack/oslo.service/commit/7ec187317167c514f92162be3e... [6] https://opendev.org/openstack/neutron/commit/e41d82e37444418f2a3ef770c9def01...
Le mer. 17 janv. 2024 à 19:58, Jay Faulkner jay@gr-oss.io a écrit :
Despite my efforts to move some of the discussion to be asynchronous, the conversation continues to progress synchronously in IRC.
I would strongly suggest based on the outcomes of some of those conversations (logs at https://meetings.opendev.org/irclogs/%23openstack-tc/%23openstack-tc.2024-01... ) that projects hold off on merging these changes until a larger discussion happens.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
On Wed, Jan 17, 2024 at 9:39 AM Jay Faulkner jay@gr-oss.io wrote:
Hi all,
https://review.opendev.org/q/topic:%22bump-eventlet%22 was pushed recently, to bump eventlet versions to a consistent lower bound across OpenStack projects. I saw some concerns about this across various OpenStack IRC channels, and wanted to start a larger conversation around the changes rather than having what would be my third separate small conversation about it :). I cannot speak to exactly why Herve pushed these changes, but I think they are a good idea and should be merged.
If you're unaware or haven't read up on the context; versions of eventlet before 0.34 were released with minimal/failing CI, and done so with known bugs in many cases. A group of developers has taken the last months to improve the state of eventlet considerably, enhancing and improving CI, discovering and fixing known issues and enabling support for Python 3.12. From my perspective; this is a significant enough improvement to warrant our community moving forward generally[0] with setting a new higher low bound for eventlet -- the versions that are maintained.
I do think that we can do a better job of advertising the value of this change in gerrit to other OpenStack contributors; simply stating it's for python 3.12 support downplays the significant improvements made across that project and the overall value of landing the version bump. I am unsure if we should advertise the improvements in a user/operator facing manner; I lean towards not doing so.
From a single project perspective (with my Ironic PTL hat on): I have no desire to troubleshoot bugs in Ironic, or related projects, that only occur in eventlet versions prior to 0.34. This leads me to believe landing those changes in Ironic are the good path forward, and I'll be advocating for that inside Ironic.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
0: I'll note this should not be read as a suggestion of a mandate; only that it's a good idea to bump this requirement in OpenStack projects and should be done unless cores for that project see a specific reason why not.
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
The bump for 0.35.0 was landed in the requirements project overnight, and it broke more docs jobs. The Manila team reported a bug in their jobs: https://zuul.opendev.org/t/openstack/build/63914f9dc9e24e1180eea22bfd92e13c
I looked into the issue and pushed https://github.com/eventlet/eventlet/pull/907 to hopefully resolve this entire class of issue. My hope is that fix can be landed and released to prevent us needing to rollback again.
Thanks, Jay Faulkner OpenStack TC Chair
On Mon, Jan 29, 2024 at 9:05 AM Jay Faulkner jay@gr-oss.io wrote:
The 0.34.x release caused issues with documentation builds, which are not tested in requirements. I'm unsure at this point in the cycle if it's wise to bump this constraint (even though it's obviously a massive improvement) -- but after testing, I no longer have specific objections for projects I work on closely.
It appears https://github.com/eventlet/eventlet/pull/866/files fixed the documentation builds; 0.35.0 gets a clean pass on ironic-inspector docs when run locally.
Thanks, Jay
On Mon, Jan 29, 2024 at 8:22 AM Herve Beraud hberaud@redhat.com wrote:
Hi all,
FYI, last week I released a version of eventlet (0.35.0) [1][2][3]. For now we did not notice new issues related to this new version. This morning I proposed a new eventlet bump against our general upper-constraints [4].
This new version is an important one. It introduces the capabilities to run eventlet and asyncio in the same process. It also solves previously discussed issues.
Do not hesitate to continue the discussion directly on the requirements patch [4].
Thanks for your attention.
[1] https://github.com/eventlet/eventlet/issues/897 [2] https://github.com/eventlet/eventlet/releases/tag/v0.35.0 [3] https://pypi.org/project/eventlet/0.35.0/ [4] https://review.opendev.org/c/openstack/requirements/+/907048
Le jeu. 18 janv. 2024 à 15:00, Herve Beraud hberaud@redhat.com a écrit :
Hey all,
Concerning my upper-constraints bump of eventlet [1] it was a simple bump like all those that we do regularly bump all kinds of libs to their latest pypi versions. Eventlet got new pypi versions, I simply reflected them on Openstack, as usual, as made with our automated updates for all our requirements.
Concerning that list of patches to update min version in requirements.txt [2], even without these patches, the UC upgrade [1] would have allowed Openstack deliverables to pull the latest version.
By doing that series of requirements files updates, my main goal was that I wanted to avoid tons of broken install related to pip resolver issues between all our deliverables by uniformizing all our requirements files ASAP. This list of patches contains libraries and services of all kinds. Many services use libraries from their team scope and from outside their team scope (oslo, etc...). Keeping them in different states could lead to so many potential conflicts to resolve. I was thinking uniformity would simplify our life.
To illustrate my previous goal, here is a similar scenario, with a pip resolver issue, between os-brick, nova, and cinder:
https://opendev.org/openstack/nova/commit/e5b264bf7b28d3987b4ea30c61da4262ae...
https://opendev.org/openstack/cinder/commit/a74513db9a5d8c0fcc92614bd95262ec...
The previous example is between 3 deliverables. I think that, without uniformizing our requirements file, we currently own at least 76² opportunities to face the same issue.
We should also notice that similar patches were made in the past, As an example, swift made similar things in the past for more or less the same reasons [3]. Or again with Ironic [4], oslo.service [5], neutron [6].
A side goal was also to limit the requirements file to the recent eventlet versions who are actively maintained and where Python 3.12 is supported, to prepare for 2024.2/Dalmatian ASAP. Using it as soon as possible in real situations, would have allowed us to observe possible problems early. The UC upgrade allowed us to catch a bug in eventlet within swift.
To close the requirements file update topic, the situation with bugs would have been the same without upgrading all these requirements files and simply by upgrading UC, but we would have left room to future pip resolver issues. Now that eventlet has been downgraded, that's not an issue anymore.
I should have communicated more broadly before submitting this series of patches. I apologize for the noise generated by my previous series of patches.
To finish, we currently prepare a new version of eventlet with a patch that will fix the swift bug https://github.com/eventlet/eventlet/issues/897
Thanks for your attention
[1] https://review.opendev.org/c/openstack/requirements/+/904147 [2] https://review.opendev.org/q/topic:%22bump-eventlet%22 [3] https://opendev.org/openstack/swift/commit/0ebfeddf65adb473fda2adfd1705534b6... [4] https://opendev.org/openstack/ironic/commit/69bdbfb6e97e53998d5a30c60f621131... [5] https://opendev.org/openstack/oslo.service/commit/7ec187317167c514f92162be3e... [6] https://opendev.org/openstack/neutron/commit/e41d82e37444418f2a3ef770c9def01...
Le mer. 17 janv. 2024 à 19:58, Jay Faulkner jay@gr-oss.io a écrit :
Despite my efforts to move some of the discussion to be asynchronous, the conversation continues to progress synchronously in IRC.
I would strongly suggest based on the outcomes of some of those conversations (logs at https://meetings.opendev.org/irclogs/%23openstack-tc/%23openstack-tc.2024-01... ) that projects hold off on merging these changes until a larger discussion happens.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
On Wed, Jan 17, 2024 at 9:39 AM Jay Faulkner jay@gr-oss.io wrote:
Hi all,
https://review.opendev.org/q/topic:%22bump-eventlet%22 was pushed recently, to bump eventlet versions to a consistent lower bound across OpenStack projects. I saw some concerns about this across various OpenStack IRC channels, and wanted to start a larger conversation around the changes rather than having what would be my third separate small conversation about it :). I cannot speak to exactly why Herve pushed these changes, but I think they are a good idea and should be merged.
If you're unaware or haven't read up on the context; versions of eventlet before 0.34 were released with minimal/failing CI, and done so with known bugs in many cases. A group of developers has taken the last months to improve the state of eventlet considerably, enhancing and improving CI, discovering and fixing known issues and enabling support for Python 3.12. From my perspective; this is a significant enough improvement to warrant our community moving forward generally[0] with setting a new higher low bound for eventlet -- the versions that are maintained.
I do think that we can do a better job of advertising the value of this change in gerrit to other OpenStack contributors; simply stating it's for python 3.12 support downplays the significant improvements made across that project and the overall value of landing the version bump. I am unsure if we should advertise the improvements in a user/operator facing manner; I lean towards not doing so.
From a single project perspective (with my Ironic PTL hat on): I have no desire to troubleshoot bugs in Ironic, or related projects, that only occur in eventlet versions prior to 0.34. This leads me to believe landing those changes in Ironic are the good path forward, and I'll be advocating for that inside Ironic.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
0: I'll note this should not be read as a suggestion of a mandate; only that it's a good idea to bump this requirement in OpenStack projects and should be done unless cores for that project see a specific reason why not.
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
Hey all,
Jay's fix is now released with eventlet in version 0.35.1 https://pypi.org/project/eventlet/0.35.1/ Requirements are also up-to-date with this new version and the previous recent versions are excluded from our runtimes https://review.opendev.org/c/openstack/requirements/+/907243
Thanks Jay
Le mar. 30 janv. 2024 à 17:45, Jay Faulkner jay@gr-oss.io a écrit :
The bump for 0.35.0 was landed in the requirements project overnight, and it broke more docs jobs. The Manila team reported a bug in their jobs: https://zuul.opendev.org/t/openstack/build/63914f9dc9e24e1180eea22bfd92e13c
I looked into the issue and pushed https://github.com/eventlet/eventlet/pull/907 to hopefully resolve this entire class of issue. My hope is that fix can be landed and released to prevent us needing to rollback again.
Thanks, Jay Faulkner OpenStack TC Chair
On Mon, Jan 29, 2024 at 9:05 AM Jay Faulkner jay@gr-oss.io wrote:
The 0.34.x release caused issues with documentation builds, which are not tested in requirements. I'm unsure at this point in the cycle if it's wise to bump this constraint (even though it's obviously a massive improvement) -- but after testing, I no longer have specific objections for projects I work on closely.
It appears https://github.com/eventlet/eventlet/pull/866/files fixed the documentation builds; 0.35.0 gets a clean pass on ironic-inspector docs when run locally.
Thanks, Jay
On Mon, Jan 29, 2024 at 8:22 AM Herve Beraud hberaud@redhat.com wrote:
Hi all,
FYI, last week I released a version of eventlet (0.35.0) [1][2][3]. For now we did not notice new issues related to this new version. This morning I proposed a new eventlet bump against our general upper-constraints [4].
This new version is an important one. It introduces the capabilities to run eventlet and asyncio in the same process. It also solves previously discussed issues.
Do not hesitate to continue the discussion directly on the requirements patch [4].
Thanks for your attention.
[1] https://github.com/eventlet/eventlet/issues/897 [2] https://github.com/eventlet/eventlet/releases/tag/v0.35.0 [3] https://pypi.org/project/eventlet/0.35.0/ [4] https://review.opendev.org/c/openstack/requirements/+/907048
Le jeu. 18 janv. 2024 à 15:00, Herve Beraud hberaud@redhat.com a écrit :
Hey all,
Concerning my upper-constraints bump of eventlet [1] it was a simple bump like all those that we do regularly bump all kinds of libs to their latest pypi versions. Eventlet got new pypi versions, I simply reflected them on Openstack, as usual, as made with our automated updates for all our requirements.
Concerning that list of patches to update min version in requirements.txt [2], even without these patches, the UC upgrade [1] would have allowed Openstack deliverables to pull the latest version.
By doing that series of requirements files updates, my main goal was that I wanted to avoid tons of broken install related to pip resolver issues between all our deliverables by uniformizing all our requirements files ASAP. This list of patches contains libraries and services of all kinds. Many services use libraries from their team scope and from outside their team scope (oslo, etc...). Keeping them in different states could lead to so many potential conflicts to resolve. I was thinking uniformity would simplify our life.
To illustrate my previous goal, here is a similar scenario, with a pip resolver issue, between os-brick, nova, and cinder:
https://opendev.org/openstack/nova/commit/e5b264bf7b28d3987b4ea30c61da4262ae...
https://opendev.org/openstack/cinder/commit/a74513db9a5d8c0fcc92614bd95262ec...
The previous example is between 3 deliverables. I think that, without uniformizing our requirements file, we currently own at least 76² opportunities to face the same issue.
We should also notice that similar patches were made in the past, As an example, swift made similar things in the past for more or less the same reasons [3]. Or again with Ironic [4], oslo.service [5], neutron [6].
A side goal was also to limit the requirements file to the recent eventlet versions who are actively maintained and where Python 3.12 is supported, to prepare for 2024.2/Dalmatian ASAP. Using it as soon as possible in real situations, would have allowed us to observe possible problems early. The UC upgrade allowed us to catch a bug in eventlet within swift.
To close the requirements file update topic, the situation with bugs would have been the same without upgrading all these requirements files and simply by upgrading UC, but we would have left room to future pip resolver issues. Now that eventlet has been downgraded, that's not an issue anymore.
I should have communicated more broadly before submitting this series of patches. I apologize for the noise generated by my previous series of patches.
To finish, we currently prepare a new version of eventlet with a patch that will fix the swift bug https://github.com/eventlet/eventlet/issues/897
Thanks for your attention
[1] https://review.opendev.org/c/openstack/requirements/+/904147 [2] https://review.opendev.org/q/topic:%22bump-eventlet%22 [3] https://opendev.org/openstack/swift/commit/0ebfeddf65adb473fda2adfd1705534b6... [4] https://opendev.org/openstack/ironic/commit/69bdbfb6e97e53998d5a30c60f621131... [5] https://opendev.org/openstack/oslo.service/commit/7ec187317167c514f92162be3e... [6] https://opendev.org/openstack/neutron/commit/e41d82e37444418f2a3ef770c9def01...
Le mer. 17 janv. 2024 à 19:58, Jay Faulkner jay@gr-oss.io a écrit :
Despite my efforts to move some of the discussion to be asynchronous, the conversation continues to progress synchronously in IRC.
I would strongly suggest based on the outcomes of some of those conversations (logs at https://meetings.opendev.org/irclogs/%23openstack-tc/%23openstack-tc.2024-01... ) that projects hold off on merging these changes until a larger discussion happens.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
On Wed, Jan 17, 2024 at 9:39 AM Jay Faulkner jay@gr-oss.io wrote:
Hi all,
https://review.opendev.org/q/topic:%22bump-eventlet%22 was pushed recently, to bump eventlet versions to a consistent lower bound across OpenStack projects. I saw some concerns about this across various OpenStack IRC channels, and wanted to start a larger conversation around the changes rather than having what would be my third separate small conversation about it :). I cannot speak to exactly why Herve pushed these changes, but I think they are a good idea and should be merged.
If you're unaware or haven't read up on the context; versions of eventlet before 0.34 were released with minimal/failing CI, and done so with known bugs in many cases. A group of developers has taken the last months to improve the state of eventlet considerably, enhancing and improving CI, discovering and fixing known issues and enabling support for Python 3.12. From my perspective; this is a significant enough improvement to warrant our community moving forward generally[0] with setting a new higher low bound for eventlet -- the versions that are maintained.
I do think that we can do a better job of advertising the value of this change in gerrit to other OpenStack contributors; simply stating it's for python 3.12 support downplays the significant improvements made across that project and the overall value of landing the version bump. I am unsure if we should advertise the improvements in a user/operator facing manner; I lean towards not doing so.
From a single project perspective (with my Ironic PTL hat on): I have no desire to troubleshoot bugs in Ironic, or related projects, that only occur in eventlet versions prior to 0.34. This leads me to believe landing those changes in Ironic are the good path forward, and I'll be advocating for that inside Ironic.
Thanks, Jay Faulkner OpenStack TC Chair OpenStack Ironic PTL
0: I'll note this should not be read as a suggestion of a mandate; only that it's a good idea to bump this requirement in OpenStack projects and should be done unless cores for that project see a specific reason why not.
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
participants (2)
-
Herve Beraud
-
Jay Faulkner