[devstack][nova][cinder][ironic][glance][swift][neutron][all] Deprecating/removing non-uWSGI deployment mechanisms?
o/ tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is. I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack: * Eventlet server * Apache + uWSGI * Apache + mod_wsgi In addition, there are a number of overlapping config options that could be set to modify which of these was in use: * ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc. Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by pbr and DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi. My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would therefore like to propose we do the following: * Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services that are not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that they test with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE options like Glance has. This will affect Ironic and maybe more. I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think: https://review.opendev.org/c/openstack/devstack/+/932193/1 Cheers, Stephen [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/
Hello, This is an interesting topic and IMO has been in a weird state for years and years and I can only deduct that this has been because of, yet again, the broad spectrum of ways that OpenStack can be deployed. Every guide, every deployment tool, every project has their own ways; supported way, and recommend way, of deploying and multiple projects has either bugs with one but no the other or the other way around making deployment even harder. Specially on this topic though, this drops testing of one of those ways, is the plan/should we interpret this as a guideline on what developers need to test their features? Should we take this as advice on how to deploy the services and start dropping any usage of mod_wsgi in deployment tools? If no, should we expect mod_wsgi to break when eventlet is phased out because it will not be tested? I think this topic really need to be a specification (and perhaps a community goal) around what deliverables should support. If we want to take measures, perhaps it's time to consolidate and minimize the headache for deployers and just drop everything except for one way, the right way (that we decide and then stick with). I don't want this to sound like throwing shade at this effort, the opposite I want it to move along, but in a clear way, and just perhaps we should just drop everything and use oslo.service and go back to how it was before mod_wsgi was recommended now that we're getting rid of eventlet and just deploy more services instead of using something like mod_wsgi to handle N copies of services. /Tobias On Wed, Oct 16, 2024 at 11:12:03AM UTC, Stephen Finucane wrote:
o/
tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is.
I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack:
* Eventlet server * Apache + uWSGI * Apache + mod_wsgi
In addition, there are a number of overlapping config options that could be set to modify which of these was in use:
* ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc.
Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by pbr and DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi.
My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would therefore like to propose we do the following:
* Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services that are not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that they test with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE options like Glance has. This will affect Ironic and maybe more.
I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think:
https://review.opendev.org/c/openstack/devstack/+/932193/1
Cheers,
Stephen
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/
On Wed, 2024-10-16 at 14:16 +0200, Tobias Urdin wrote:
Hello,
This is an interesting topic and IMO has been in a weird state for years and years and I can only deduct that this has been because of, yet again, the broad spectrum of ways that OpenStack can be deployed.
Every guide, every deployment tool, every project has their own ways; supported way, and recommend way, of deploying and multiple projects has either bugs with one but no the other or the other way around making deployment even harder.
Specially on this topic though, this drops testing of one of those ways, is the plan/should we interpret this as a guideline on what developers need to test their features? Should we take this as advice on how to deploy the services and start dropping any usage of mod_wsgi in deployment tools? If no, should we expect mod_wsgi to break when eventlet is phased out because it will not be tested?
I think this topic really need to be a specification (and perhaps a community goal) around what deliverables should support.
If we want to take measures, perhaps it's time to consolidate and minimize the headache for deployers and just drop everything except for one way, the right way (that we decide and then stick with).
I don't want this to sound like throwing shade at this effort, the opposite I want it to move along, but in a clear way, and just perhaps we should just drop everything and use oslo.service and go back to how it was before mod_wsgi was recommended now that we're getting rid of eventlet and just deploy more services instead of using something like mod_wsgi to handle N copies of services.
In theory, so long as services support WSGI then it shouldn't matter what we WSGI HTTP server we use: they all speak WSGI. Sure, the configuration files are going to look different and the performance and feature sets will vary, but IMO that's a deployment problem/decision rather than something the services (or our CIs) need to worry about. I certainly don't feel *I* am in a position to state that e.g. gunicorn should be preferred to a long-standing alternative like uWSGI or mod_wsgi, an ASGI-first implementation like Daphne or uvicorn, or a newcomer like nginx unit, and this is a position apparently also shared by both the Django [1] or Flask [2] communities (to pick two well-known examples). Maybe deployers could share feedback on their chosen solution and put guidance in the deployment docs but again, that's out of scope for a DevStack change. In addition, I should highlight that we are not currently testing against anything but uWSGI and, in rare cases, eventlet server. A grok for the various config options I mentioned reveals nothing of interest [3][4] (for example, the only override of 'WSGI_MODE' I can find is in monasca-api which, based on recent attempts to push stuff to monasca projects, is either dead or on life support). Nothing is changing here and for the most part we're just removing dead code and pushing a few long-running efforts forwards towards their conclusion. This is also true for the removal of support for deploying with eventlet server: as I noted, neither Cinder nor Nova actually test these paths any more, and Neutron is well on the way to doing the same. The only services that appear to be long- term interested in testing with eventlet server are not those interested in using eventlet for their API service but rather those that need to support a standalone mode, with multiple servers provided by a single binary. Again, as I noted below, this list includes Glance, Ironic, and possibly Swift (???). There's definitely a discussion to be had around what the replacement for eventlet server is for these projects, but that's being tackled elsewhere [5] and is not something I want to wade into here. I hope this clarifies my suggestions here, Stephen [1] https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/ [2] https://flask.palletsprojects.com/en/2.3.x/deploying/ [3] https://codesearch.opendev.org/?q=ENABLE_HTTPD_MOD_WSGI_SERVICES [4] https://codesearch.opendev.org/?q=WSGI_MODE%3D [5] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
/Tobias
On Wed, Oct 16, 2024 at 11:12:03AM UTC, Stephen Finucane wrote:
o/
tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is.
I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack:
* Eventlet server * Apache + uWSGI * Apache + mod_wsgi
In addition, there are a number of overlapping config options that could be set to modify which of these was in use:
* ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc.
Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by pbr and DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi.
My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would therefore like to propose we do the following:
* Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services that are not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that they test with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE options like Glance has. This will affect Ironic and maybe more.
I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think:
https://review.opendev.org/c/openstack/devstack/+/932193/1
Cheers,
Stephen
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/
Hey folks, FYI, we think that his topic overlaps a bit with some aspects of the Eventlet Removal Initiative. Indeed, removing Eventlet means removing our usage of the WSGI features offered by Eventlet. https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... I invite you to read this parallel discussion and I wanted to share with you the HTTP SGI working group => https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group IMO we could capitalize from each other by having cross discussion. Choosing the next SGI candidate will dictate how we deal with HTTP for the coming decade, so I think we have to put our focus on it ASAP. We created comparisons between the alternatives at our disposal, here is an example with the HTTP/SGI comparison https://wiki.openstack.org/wiki/SGI-servers-comparison Please find all the useful details in the official wiki: https://wiki.openstack.org/wiki/Eventlet-removal We strongly encourage a collaborative decision, to avoid having multiple ways to manage things. This specialized working group would be responsible for producing a unified approach that will simplify maintenance and enhance user experience. Thanks for your time and consideration. Le jeu. 17 oct. 2024 à 13:06, Stephen Finucane <stephenfin@redhat.com> a écrit :
On Wed, 2024-10-16 at 14:16 +0200, Tobias Urdin wrote:
Hello,
This is an interesting topic and IMO has been in a weird state for years and years and I can only deduct that this has been because of, yet again, the broad spectrum of ways that OpenStack can be deployed.
Every guide, every deployment tool, every project has their own ways; supported way, and recommend way, of deploying and multiple projects has either bugs with one but no the other or the other way around making deployment even harder.
Specially on this topic though, this drops testing of one of those ways, is the plan/should we interpret this as a guideline on what developers need to test their features? Should we take this as advice on how to deploy the services and start dropping any usage of mod_wsgi in deployment tools? If no, should we expect mod_wsgi to break when eventlet is phased out because it will not be tested?
I think this topic really need to be a specification (and perhaps a community goal) around what deliverables should support.
If we want to take measures, perhaps it's time to consolidate and minimize the headache for deployers and just drop everything except for one way, the right way (that we decide and then stick with).
I don't want this to sound like throwing shade at this effort, the opposite I want it to move along, but in a clear way, and just perhaps we should just drop everything and use oslo.service and go back to how it was before mod_wsgi was recommended now that we're getting rid of eventlet and just deploy more services instead of using something like mod_wsgi to handle N copies of services.
In theory, so long as services support WSGI then it shouldn't matter what we WSGI HTTP server we use: they all speak WSGI. Sure, the configuration files are going to look different and the performance and feature sets will vary, but IMO that's a deployment problem/decision rather than something the services (or our CIs) need to worry about. I certainly don't feel *I* am in a position to state that e.g. gunicorn should be preferred to a long-standing alternative like uWSGI or mod_wsgi, an ASGI-first implementation like Daphne or uvicorn, or a newcomer like nginx unit, and this is a position apparently also shared by both the Django [1] or Flask [2] communities (to pick two well-known examples). Maybe deployers could share feedback on their chosen solution and put guidance in the deployment docs but again, that's out of scope for a DevStack change.
In addition, I should highlight that we are not currently testing against anything but uWSGI and, in rare cases, eventlet server. A grok for the various config options I mentioned reveals nothing of interest [3][4] (for example, the only override of 'WSGI_MODE' I can find is in monasca-api which, based on recent attempts to push stuff to monasca projects, is either dead or on life support). Nothing is changing here and for the most part we're just removing dead code and pushing a few long-running efforts forwards towards their conclusion. This is also true for the removal of support for deploying with eventlet server: as I noted, neither Cinder nor Nova actually test these paths any more, and Neutron is well on the way to doing the same. The only services that appear to be long- term interested in testing with eventlet server are not those interested in using eventlet for their API service but rather those that need to support a standalone mode, with multiple servers provided by a single binary. Again, as I noted below, this list includes Glance, Ironic, and possibly Swift (???). There's definitely a discussion to be had around what the replacement for eventlet server is for these projects, but that's being tackled elsewhere [5] and is not something I want to wade into here.
I hope this clarifies my suggestions here, Stephen
[1] https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/ [2] https://flask.palletsprojects.com/en/2.3.x/deploying/ [3] https://codesearch.opendev.org/?q=ENABLE_HTTPD_MOD_WSGI_SERVICES [4] https://codesearch.opendev.org/?q=WSGI_MODE%3D [5] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
/Tobias
On Wed, Oct 16, 2024 at 11:12:03AM UTC, Stephen Finucane wrote:
o/
tl;dr: I'd like to remove the ability to deploy services under
DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is.
I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack:
* Eventlet server * Apache + uWSGI * Apache + mod_wsgi
In addition, there are a number of overlapping config options that could be set to modify which of these was in use:
* ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc.
Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by
DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi.
My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would
to propose we do the following:
* Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services
not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that
with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE
mod_wsgi from pbr and therefore like that are they test options like
Glance has. This will affect Ironic and maybe more.
I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think:
https://review.opendev.org/c/openstack/devstack/+/932193/1
Cheers,
Stephen
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
On Mon, 2024-10-28 at 17:58 +0100, Herve Beraud wrote:
Hey folks,
FYI, we think that his topic overlaps a bit with some aspects of the Eventlet Removal Initiative. Indeed, removing Eventlet means removing our usage of the WSGI features offered by Eventlet.
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
I invite you to read this parallel discussion and I wanted to share with you the HTTP SGI working group => https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group
IMO we could capitalize from each other by having cross discussion. Choosing the next SGI candidate will dictate how we deal with HTTP for the coming decade, so I think we have to put our focus on it ASAP. We created comparisons between the alternatives at our disposal, here is an example with the HTTP/SGI comparison https://wiki.openstack.org/wiki/SGI-servers-comparison
I took a read through these. Just to be clear, are we suggesting projects change their web frameworks or that DevStack/DevStack plugins change the WSGI server used? If so, why? I understand that some projects might want to explore ASGI, which would necessitate something other than our current Apache+mod_wsgi+uWSGI combo for the server and whatever web framework they're using for the server, but assuming they are going to continue using WSGI then any change of server or framework would seem to be be secondary (if not entirely unrelated) to the eventlet removal goal. While e.g. nova might have a few reasons to explore long- term alternatives to Paste+WebOb, eventlet removal doesn't feel like one of them. Stephen
Please find all the useful details in the official wiki: https://wiki.openstack.org/wiki/Eventlet-removal We strongly encourage a collaborative decision, to avoid having multiple ways to manage things. This specialized working group would be responsible for producing a unified approach that will simplify maintenance and enhance user experience.
Thanks for your time and consideration.
Le jeu. 17 oct. 2024 à 13:06, Stephen Finucane <stephenfin@redhat.com> a écrit :
Hello,
This is an interesting topic and IMO has been in a weird state for years and years and I can only deduct that this has been because of, yet again, the broad spectrum of ways that OpenStack can be deployed.
Every guide, every deployment tool, every project has their own ways; supported way, and recommend way, of deploying and multiple projects has either bugs with one but no the other or the other way around making deployment even harder.
Specially on this topic though, this drops testing of one of those ways, is the plan/should we interpret this as a guideline on what developers need to test their features? Should we take this as advice on how to deploy the services and start dropping any usage of mod_wsgi in deployment tools? If no, should we expect mod_wsgi to break when eventlet is phased out because it will not be tested?
I think this topic really need to be a specification (and perhaps a community goal) around what deliverables should support.
If we want to take measures, perhaps it's time to consolidate and minimize
On Wed, 2024-10-16 at 14:16 +0200, Tobias Urdin wrote: the headache for deployers and just drop everything except for one way, the right way (that we decide and then stick with).
I don't want this to sound like throwing shade at this effort, the opposite I want it to move along, but in a clear way, and just perhaps we should just drop everything and use oslo.service and go back to how it was before mod_wsgi was recommended now that we're getting rid of eventlet and just deploy more services instead of using something like mod_wsgi to handle N copies of services.
In theory, so long as services support WSGI then it shouldn't matter what we WSGI HTTP server we use: they all speak WSGI. Sure, the configuration files are going to look different and the performance and feature sets will vary, but IMO that's a deployment problem/decision rather than something the services (or our CIs) need to worry about. I certainly don't feel *I* am in a position to state that e.g. gunicorn should be preferred to a long-standing alternative like uWSGI or mod_wsgi, an ASGI-first implementation like Daphne or uvicorn, or a newcomer like nginx unit, and this is a position apparently also shared by both the Django [1] or Flask [2] communities (to pick two well-known examples). Maybe deployers could share feedback on their chosen solution and put guidance in the deployment docs but again, that's out of scope for a DevStack change.
In addition, I should highlight that we are not currently testing against anything but uWSGI and, in rare cases, eventlet server. A grok for the various config options I mentioned reveals nothing of interest [3][4] (for example, the only override of 'WSGI_MODE' I can find is in monasca-api which, based on recent attempts to push stuff to monasca projects, is either dead or on life support). Nothing is changing here and for the most part we're just removing dead code and pushing a few long-running efforts forwards towards their conclusion. This is also true for the removal of support for deploying with eventlet server: as I noted, neither Cinder nor Nova actually test these paths any more, and Neutron is well on the way to doing the same. The only services that appear to be long- term interested in testing with eventlet server are not those interested in using eventlet for their API service but rather those that need to support a standalone mode, with multiple servers provided by a single binary. Again, as I noted below, this list includes Glance, Ironic, and possibly Swift (???). There's definitely a discussion to be had around what the replacement for eventlet server is for these projects, but that's being tackled elsewhere [5] and is not something I want to wade into here.
I hope this clarifies my suggestions here, Stephen
[1] https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/ [2] https://flask.palletsprojects.com/en/2.3.x/deploying/ [3] https://codesearch.opendev.org/?q=ENABLE_HTTPD_MOD_WSGI_SERVICES [4] https://codesearch.opendev.org/?q=WSGI_MODE%3D [5] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
/Tobias
On Wed, Oct 16, 2024 at 11:12:03AM UTC, Stephen Finucane wrote:
o/
tl;dr: I'd like to remove the ability to deploy services under mod_wsgi
DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is.
I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack:
* Eventlet server * Apache + uWSGI * Apache + mod_wsgi
In addition, there are a number of overlapping config options that could be set to modify which of these was in use:
* ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc.
Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by
DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi.
My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would
to propose we do the following:
* Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services
not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that
with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE
from pbr and therefore like that are they test options like
Glance has. This will affect Ironic and maybe more.
I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think:
https://review.opendev.org/c/openstack/devstack/+/932193/1
Cheers,
Stephen
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
Le mer. 30 oct. 2024 à 13:05, Stephen Finucane <stephenfin@redhat.com> a écrit :
On Mon, 2024-10-28 at 17:58 +0100, Herve Beraud wrote:
Hey folks,
FYI, we think that his topic overlaps a bit with some aspects of the Eventlet Removal Initiative. Indeed, removing Eventlet means removing our usage of the WSGI features offered by Eventlet.
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
I invite you to read this parallel discussion and I wanted to share with you the HTTP SGI working group => https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group
IMO we could capitalize from each other by having cross discussion. Choosing the next SGI candidate will dictate how we deal with HTTP for the coming decade, so I think we have to put our focus on it ASAP. We created comparisons between the alternatives at our disposal, here is an example with the HTTP/SGI comparison https://wiki.openstack.org/wiki/SGI-servers-comparison
I took a read through these. Just to be clear, are we suggesting projects change their web frameworks or that DevStack/DevStack plugins change the WSGI server used? If so, why?
As Eventlet is going to be removed, its WSGI server too, I think we will mostly try to find a standardized way to replace it. On our side I think the discussion is mostly about projects relying on Eventlet needing to change their WSGI server and so it potentially opens the doors of a wider discussion, to avoid finishing with 10 different ways to serve HTTP. Maybe uWSGI and WSGI is a credible short term solution, but if components express needs to explore something like ASGI or other alternatives to uWSGI, then I think we should have some consultations, because in some way these 2 topics (DevStack and Eventet removal) overlap. The question is not about requesting to change the DevStack WSGI server, but if we decide to follow some uniformity, then the WSGI aspect of DevStack would also have to join the discussion about standardization. Some ASGI stacks can handle sync and async at the same time, so, if people push this path and if the discussion is going this way, maybe it would be worth the effort to transform DevStack too. On the other hand, if DevStack encourages uWSGI, probably uWSGI would be the default option for escaping from Eventlet, but locking us in a limited support of async. As uWSGI is in maintenance mode, we should not expect new updates from uWSGI to introduce/increase async support.
I understand that some projects might want to explore ASGI, which would necessitate something other than our current Apache+mod_wsgi+uWSGI combo for the server and whatever web framework they're using for the server, but assuming they are going to continue using WSGI then any change of server or framework would seem to be be secondary (if not entirely unrelated) to the eventlet removal goal. While e.g. nova might have a few reasons to explore long-term alternatives to Paste+WebOb, eventlet removal doesn't feel like one of them.
Stephen
Please find all the useful details in the official wiki: https://wiki.openstack.org/wiki/Eventlet-removal We strongly encourage a collaborative decision, to avoid having multiple ways to manage things. This specialized working group would be responsible for producing a unified approach that will simplify maintenance and enhance user experience.
Thanks for your time and consideration.
Le jeu. 17 oct. 2024 à 13:06, Stephen Finucane <stephenfin@redhat.com> a écrit :
On Wed, 2024-10-16 at 14:16 +0200, Tobias Urdin wrote:
Hello,
This is an interesting topic and IMO has been in a weird state for years and years and I can only deduct that this has been because of, yet again, the broad spectrum of ways that OpenStack can be deployed.
Every guide, every deployment tool, every project has their own ways; supported way, and recommend way, of deploying and multiple projects has either bugs with one but no the other or the other way around making deployment even harder.
Specially on this topic though, this drops testing of one of those ways, is the plan/should we interpret this as a guideline on what developers need to test their features? Should we take this as advice on how to deploy the services and start dropping any usage of mod_wsgi in deployment tools? If no, should we expect mod_wsgi to break when eventlet is phased out because it will not be tested?
I think this topic really need to be a specification (and perhaps a community goal) around what deliverables should support.
If we want to take measures, perhaps it's time to consolidate and minimize the headache for deployers and just drop everything except for one way, the right way (that we decide and then stick with).
I think our main question is the same as the one expressed by Tobias in this thread ^. Consolidate. But consolidate without ignoring potential coming async needs.
I don't want this to sound like throwing shade at this effort, the opposite I want it to move along, but in a clear way, and just perhaps we should just drop everything and use oslo.service and go back to how it was before mod_wsgi was recommended now that we're getting rid of eventlet and just deploy more services instead of using something like mod_wsgi to handle N copies of services.
In theory, so long as services support WSGI then it shouldn't matter what we WSGI HTTP server we use: they all speak WSGI. Sure, the configuration files are going to look different and the performance and feature sets will vary, but IMO that's a deployment problem/decision rather than something the services (or our CIs) need to worry about.
And as you said, having plurality in our WSGI servers would lead us to deployment diversity. If we put the ASGI aspect on top of that, we will obtain another layer of complexity. I think the current DevStack discussion is in some sort the other face of the eventlet removal piece Let me know if I answered your question.
I certainly don't feel *I* am in a position to state that e.g. gunicorn should be preferred to a long-standing alternative like uWSGI or mod_wsgi, an ASGI-first implementation like Daphne or uvicorn, or a newcomer like nginx unit, and this is a position apparently also shared by both the Django [1] or Flask [2] communities (to pick two well-known examples). Maybe deployers could share feedback on their chosen solution and put guidance in the deployment docs but again, that's out of scope for a DevStack change.
In addition, I should highlight that we are not currently testing against anything but uWSGI and, in rare cases, eventlet server. A grok for the various config options I mentioned reveals nothing of interest [3][4] (for example, the only override of 'WSGI_MODE' I can find is in monasca-api which, based on recent attempts to push stuff to monasca projects, is either dead or on life support). Nothing is changing here and for the most part we're just removing dead code and pushing a few long-running efforts forwards towards their conclusion. This is also true for the removal of support for deploying with eventlet server: as I noted, neither Cinder nor Nova actually test these paths any more, and Neutron is well on the way to doing the same. The only services that appear to be long- term interested in testing with eventlet server are not those interested in using eventlet for their API service but rather those that need to support a standalone mode, with multiple servers provided by a single binary. Again, as I noted below, this list includes Glance, Ironic, and possibly Swift (???). There's definitely a discussion to be had around what the replacement for eventlet server is for these projects, but that's being tackled elsewhere [5] and is not something I want to wade into here.
I hope this clarifies my suggestions here, Stephen
[1] https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/ [2] https://flask.palletsprojects.com/en/2.3.x/deploying/ [3] https://codesearch.opendev.org/?q=ENABLE_HTTPD_MOD_WSGI_SERVICES [4] https://codesearch.opendev.org/?q=WSGI_MODE%3D [5] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
/Tobias
On Wed, Oct 16, 2024 at 11:12:03AM UTC, Stephen Finucane wrote:
o/
tl;dr: I'd like to remove the ability to deploy services under
DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is.
I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack:
* Eventlet server * Apache + uWSGI * Apache + mod_wsgi
In addition, there are a number of overlapping config options that could be set to modify which of these was in use:
* ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc.
Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by
DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi.
My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would
to propose we do the following:
* Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services
not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that
with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE
mod_wsgi from pbr and therefore like that are they test options like
Glance has. This will affect Ironic and maybe more.
I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think:
https://review.opendev.org/c/openstack/devstack/+/932193/1
Cheers,
Stephen
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
This is indeed interesting times, thanks for driving the discussion on Eventlet and everything that follows down that rabbit hole, Herve! I’ve always wondered why there is no oslo.api part that consolidated the usage of Flask, Pecan etc the same way that oslo.service were meant to consolidate the service layer (but being pluggable with a “backend”). Sorry for the noice thinking out loud, there probably is (by now) ancient mailing list thread, blueprint or wiki article about that somewhere. /Tobias
On 30 Oct 2024, at 15:55, Herve Beraud <hberaud@redhat.com> wrote:
Le mer. 30 oct. 2024 à 13:05, Stephen Finucane <stephenfin@redhat.com <mailto:stephenfin@redhat.com>> a écrit :
On Mon, 2024-10-28 at 17:58 +0100, Herve Beraud wrote:
Hey folks,
FYI, we think that his topic overlaps a bit with some aspects of the Eventlet Removal Initiative. Indeed, removing Eventlet means removing our usage of the WSGI features offered by Eventlet.
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
I invite you to read this parallel discussion and I wanted to share with you the HTTP SGI working group => https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group
IMO we could capitalize from each other by having cross discussion. Choosing the next SGI candidate will dictate how we deal with HTTP for the coming decade, so I think we have to put our focus on it ASAP. We created comparisons between the alternatives at our disposal, here is an example with the HTTP/SGI comparison https://wiki.openstack.org/wiki/SGI-servers-comparison
I took a read through these. Just to be clear, are we suggesting projects change their web frameworks or that DevStack/DevStack plugins change the WSGI server used? If so, why?
As Eventlet is going to be removed, its WSGI server too, I think we will mostly try to find a standardized way to replace it.
On our side I think the discussion is mostly about projects relying on Eventlet needing to change their WSGI server and so it potentially opens the doors of a wider discussion, to avoid finishing with 10 different ways to serve HTTP. Maybe uWSGI and WSGI is a credible short term solution, but if components express needs to explore something like ASGI or other alternatives to uWSGI, then I think we should have some consultations, because in some way these 2 topics (DevStack and Eventet removal) overlap.
The question is not about requesting to change the DevStack WSGI server, but if we decide to follow some uniformity, then the WSGI aspect of DevStack would also have to join the discussion about standardization.
Some ASGI stacks can handle sync and async at the same time, so, if people push this path and if the discussion is going this way, maybe it would be worth the effort to transform DevStack too.
On the other hand, if DevStack encourages uWSGI, probably uWSGI would be the default option for escaping from Eventlet, but locking us in a limited support of async. As uWSGI is in maintenance mode, we should not expect new updates from uWSGI to introduce/increase async support.
I understand that some projects might want to explore ASGI, which would necessitate something other than our current Apache+mod_wsgi+uWSGI combo for the server and whatever web framework they're using for the server, but assuming they are going to continue using WSGI then any change of server or framework would seem to be be secondary (if not entirely unrelated) to the eventlet removal goal. While e.g. nova might have a few reasons to explore long-term alternatives to Paste+WebOb, eventlet removal doesn't feel like one of them.
Stephen
Please find all the useful details in the official wiki: https://wiki.openstack.org/wiki/Eventlet-removal We strongly encourage a collaborative decision, to avoid having multiple ways to manage things. This specialized working group would be responsible for producing a unified approach that will simplify maintenance and enhance user experience.
Thanks for your time and consideration.
Le jeu. 17 oct. 2024 à 13:06, Stephen Finucane <stephenfin@redhat.com <mailto:stephenfin@redhat.com>> a écrit :
On Wed, 2024-10-16 at 14:16 +0200, Tobias Urdin wrote:
Hello,
This is an interesting topic and IMO has been in a weird state for years and years and I can only deduct that this has been because of, yet again, the broad spectrum of ways that OpenStack can be deployed.
Every guide, every deployment tool, every project has their own ways; supported way, and recommend way, of deploying and multiple projects has either bugs with one but no the other or the other way around making deployment even harder.
Specially on this topic though, this drops testing of one of those ways, is the plan/should we interpret this as a guideline on what developers need to test their features? Should we take this as advice on how to deploy the services and start dropping any usage of mod_wsgi in deployment tools? If no, should we expect mod_wsgi to break when eventlet is phased out because it will not be tested?
I think this topic really need to be a specification (and perhaps a community goal) around what deliverables should support.
If we want to take measures, perhaps it's time to consolidate and minimize the headache for deployers and just drop everything except for one way, the right way (that we decide and then stick with).
I think our main question is the same as the one expressed by Tobias in this thread ^. Consolidate. But consolidate without ignoring potential coming async needs.
I don't want this to sound like throwing shade at this effort, the opposite I want it to move along, but in a clear way, and just perhaps we should just drop everything and use oslo.service and go back to how it was before mod_wsgi was recommended now that we're getting rid of eventlet and just deploy more services instead of using something like mod_wsgi to handle N copies of services.
In theory, so long as services support WSGI then it shouldn't matter what we WSGI HTTP server we use: they all speak WSGI. Sure, the configuration files are going to look different and the performance and feature sets will vary, but IMO that's a deployment problem/decision rather than something the services (or our CIs) need to worry about.
And as you said, having plurality in our WSGI servers would lead us to deployment diversity. If we put the ASGI aspect on top of that, we will obtain another layer of complexity. I think the current DevStack discussion is in some sort the other face of the eventlet removal piece
Let me know if I answered your question.
I certainly don't feel *I* am in a position to state that e.g. gunicorn should be preferred to a long-standing alternative like uWSGI or mod_wsgi, an ASGI-first implementation like Daphne or uvicorn, or a newcomer like nginx unit, and this is a position apparently also shared by both the Django [1] or Flask [2] communities (to pick two well-known examples). Maybe deployers could share feedback on their chosen solution and put guidance in the deployment docs but again, that's out of scope for a DevStack change.
In addition, I should highlight that we are not currently testing against anything but uWSGI and, in rare cases, eventlet server. A grok for the various config options I mentioned reveals nothing of interest [3][4] (for example, the only override of 'WSGI_MODE' I can find is in monasca-api which, based on recent attempts to push stuff to monasca projects, is either dead or on life support). Nothing is changing here and for the most part we're just removing dead code and pushing a few long-running efforts forwards towards their conclusion. This is also true for the removal of support for deploying with eventlet server: as I noted, neither Cinder nor Nova actually test these paths any more, and Neutron is well on the way to doing the same. The only services that appear to be long- term interested in testing with eventlet server are not those interested in using eventlet for their API service but rather those that need to support a standalone mode, with multiple servers provided by a single binary. Again, as I noted below, this list includes Glance, Ironic, and possibly Swift (???). There's definitely a discussion to be had around what the replacement for eventlet server is for these projects, but that's being tackled elsewhere [5] and is not something I want to wade into here.
I hope this clarifies my suggestions here, Stephen
[1] https://docs.djangoproject.com/en/5.1/howto/deployment/wsgi/ [2] https://flask.palletsprojects.com/en/2.3.x/deploying/ [3] https://codesearch.opendev.org/?q=ENABLE_HTTPD_MOD_WSGI_SERVICES [4] https://codesearch.opendev.org/?q=WSGI_MODE%3D [5] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack....
/Tobias
On Wed, Oct 16, 2024 at 11:12:03AM UTC, Stephen Finucane wrote:
o/
tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is.
I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack:
* Eventlet server * Apache + uWSGI * Apache + mod_wsgi
In addition, there are a number of overlapping config options that could be set to modify which of these was in use:
* ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc.
Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by pbr and DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi.
My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would therefore like to propose we do the following:
* Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services that are not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that they test with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE options like Glance has. This will affect Ironic and maybe more.
I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think:
https://review.opendev.org/c/openstack/devstack/+/932193/1
Cheers,
Stephen
[1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/
On 10/16/24 12:12, Stephen Finucane wrote:
My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...).
After *many* people told me that Glance would fully work over uwsgi, we tried in our new Caracal deployment, and it *FAILED* to process server backups over Swift. So by all means, let's fix this *first*. On 10/16/24 12:12, Stephen Finucane wrote:
The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI.
Same: despite what I've been told, we tried, and it *FAILS* miserably. The discussion between the proxies and the stores doesn't work, as it's not really an HTTP protocol. We tried hard, and then reverted. None of them work, not even the proxy. I once wrote a patch to have the uwsgi binaries shipped by swift, but gave-up considering the above. I would *strongly* support Swift to move to uwsgi, because it increases performance a lot: it's at least twice as fast, as much as we could see. This would reduce the amount of swift proxy one would need to deploy. In our case (moderately large clusters, totaling 100+ PB, with very busy proxies) this would reduce the amount of proxy servers we would need to deploy, saving a lot on the total cost of ownership of the cluster. Cheers, Thomas Goirand (zigo)
After *many* people told me that Glance would fully work over uwsgi, we tried in our new Caracal deployment, and it *FAILED* to process server backups over Swift.
So by all means, let's fix this *first*.
This is news to me, got a bug link? Remember, glance runs under uwsgi in many of the jobs upstream... --Dan
I would agree that this is likely a configuration issue. There are a number of settings required for long/large transfers. You might want to check your config against the one created by devstack: https://opendev.org/openstack/devstack/src/branch/master/lib/apache#L317 Take note of the comment about requiring chunked encoding and not using mod_proxy_uwsgi. Michael On Thu, Oct 31, 2024 at 6:54 AM Dan Smith <dms@danplanet.com> wrote:
After *many* people told me that Glance would fully work over uwsgi, we tried in our new Caracal deployment, and it *FAILED* to process server backups over Swift.
So by all means, let's fix this *first*.
This is news to me, got a bug link?
Remember, glance runs under uwsgi in many of the jobs upstream...
--Dan
On 10/31/24 20:49, Michael Johnson wrote:
I would agree that this is likely a configuration issue. There are a number of settings required for long/large transfers.
You might want to check your config against the one created by devstack: https://opendev.org/openstack/devstack/src/branch/master/lib/apache#L317
The only thing which I didn't have in my config was http-keepalive=true, the rest of the options, I had them already. The other difference I have with the devstack setup, is that I do not use Apache mod_proxy, and a socket to uwsgi, but uwsgi directly. Could that be the cause of failure during backups? If so, could we imagine that the devstack setup stops using mod_proxy, which really, is just very annoying?
Take note of the comment about requiring chunked encoding and not using mod_proxy_uwsgi.
We're not using mod_proxy_uwsgi, as per above, but just haproxy for the API, talking directly to uwsgi (with TLS re-encryption with a cluster local PKI). Cheers, Thomas Goirand (zigo)
participants (6)
-
Dan Smith
-
Herve Beraud
-
Michael Johnson
-
Stephen Finucane
-
Thomas Goirand
-
Tobias Urdin