[all][oslo][neutron][glance][ironic] Deprecating eventlet based WSGI server
Hello,
As you know there has been a long discussion about eventlet usage in OpenStack, and after seeing multiple problems caused by the library and no long term plan in the project, and we are moving toward doping the dependency on eventlet.
https://review.opendev.org/c/openstack/governance/+/916546
One of the core implementation which heavily depends on eventlet is WSGI server. Multiple projects have been providing the -api console scripts (eg nova-api) based on the oslo.service's WSGI server implementation which depends on eventlet.
On the other hand, nowadays we have multiple methods to launch api services, such as httpd+mod_wsgi, uwsgi and gunicorn. And a few projects in OpenStack don't provide the -api console script (eg. aodh, barbican and placement) but suggest using these external frameworks to run their api WSGI application. I know that multiple deployment tools (for example TripleO in the past) and some distros (Debian) already started using the external framework for consistency, better concurrency and so on.
Given the situation, I wonder if we can consider deprecating the eventlet WSGI server in oslo.service in 2024.2/2025.1 and eventually remove all -api script in 2025.2. This was discussed during the previous nova PTG sessions and I think this makes complete sense as a global direction given the variety of alternative approaches currently available.
I'm tagging neutron(*1) and glance(*2) because IIRC these had(or even have) some problems with running API in wsgi mode, and also ironic because I've seen some works coming from it, iiuc. Please let me know if you have any concern applying this approach to these (or the other) projects.
Thank you, Takashi Kajinami
(*1) For neutron, there are a few issues being fixed now https://bugs.launchpad.net/neutron/+bug/2062009 but ml2-ovn may need additional fixes. https://bugs.launchpad.net/neutron/+bug/1912359
(*2) As far as I know the issue with glance was already fixed some time ago[1]. https://bugs.launchpad.net/glance/+bug/1888713
On 4/29/24 07:20, Takashi Kajinami wrote:
Given the situation, I wonder if we can consider deprecating the eventlet WSGI server in oslo.service in 2024.2/2025.1 and eventually remove all -api script in 2025.2. This was discussed during the previous nova PTG sessions and I think this makes complete sense as a global direction given the variety of alternative approaches currently available.
I would agree only if: - glance is fixed upstream, and tested using uwsgi (or something else) - swift starts supporting uwsgi as well. We tried in production to switch to uwsgi, and it was a nightmare.
For swift, this would be an awesome move, as switching to uwsgi doubles the performances of all http servers like the container server or the object server. For large clusters, that's a thing... Though same thing: this must be implemented in Swift's CI.
I'm tagging neutron(*1) and glance(*2) because IIRC these had(or even have) some problems with running API in wsgi mode
As much as I know, we never had any issue with Neutron. Though I heard there still is, if using OVN.
For both, IMO we need our CI to run uwsgi (or something else) *before* removing eventlet support in oslo.service.
For neutron, there are a few issues being fixed now https://bugs.launchpad.net/neutron/+bug/2062009 but ml2-ovn may need additional fixes. https://bugs.launchpad.net/neutron/+bug/1912359
We must have these fixed first, no?
(*2) As far as I know the issue with glance was already fixed some time ago[1]. https://bugs.launchpad.net/glance/+bug/1888713
I can't remember the details, but last time I tried, this patch wasn't enough. Can someone else confirm?!?
Cheers,
Thomas Goirand (zigo)
On 4/29/24 19:23, Thomas Goirand wrote:
On 4/29/24 07:20, Takashi Kajinami wrote:
Given the situation, I wonder if we can consider deprecating the eventlet WSGI server in oslo.service in 2024.2/2025.1 and eventually remove all -api script in 2025.2. This was discussed during the previous nova PTG sessions and I think this makes complete sense as a global direction given the variety of alternative approaches currently available.
I would agree only if:
- glance is fixed upstream, and tested using uwsgi (or something else)
AFAIK glance is now deployed with uwsgi by default in functional jobs and if I read the job definition/result correctly the empest-integrated-storage-import job covers image import which was a blocker in the past.
- swift starts supporting uwsgi as well. We tried in production to switch to uwsgi, and it was a nightmare.
Swift is currently out of my scope and I don't think it should/can be a blocker because swift is not using the oslo.service's implementation but its own one.
For swift, this would be an awesome move, as switching to uwsgi doubles the performances of all http servers like the container server or the object server. For large clusters, that's a thing... Though same thing: this must be implemented in Swift's CI.
I'm tagging neutron(*1) and glance(*2) because IIRC these had(or even have) some problems with running API in wsgi mode
As much as I know, we never had any issue with Neutron. Though I heard there still is, if using OVN.
For both, IMO we need our CI to run uwsgi (or something else) *before* removing eventlet support in oslo.service.
I agree.
My ask is not removing the eventlet WSGI server immediately, but invest in the direction so that we can actually deprecate and remove it in 2025.1/2 .
For neutron, there are a few issues being fixed now https://bugs.launchpad.net/neutron/+bug/2062009 but ml2-ovn may need additional fixes. https://bugs.launchpad.net/neutron/+bug/1912359
We must have these fixed first, no?
Yes.
(*2) As far as I know the issue with glance was already fixed some time ago[1]. https://bugs.launchpad.net/glance/+bug/1888713
I can't remember the details, but last time I tried, this patch wasn't enough. Can someone else confirm?!?
Cheers,
Thomas Goirand (zigo)
On 4/29/24 15:02, Takashi Kajinami wrote:
AFAIK glance is now deployed with uwsgi by default in functional jobs and if I read the job definition/result correctly the empest-integrated-storage-import job covers image import which was a blocker in the past.
Thanks for confirming, then I should probably fix back the Debian package of Glance to use uwsgi again.
- swift starts supporting uwsgi as well. We tried in production to
switch to uwsgi, and it was a nightmare.
Swift is currently out of my scope and I don't think it should/can be a blocker because swift is not using the oslo.service's implementation but its own one.
Yes, sure, but it'd be a *very* good candidate for getting rid of the Eventlet web server, as that would speed-up x2...
I'd love it if the Swift team could work on this. This would also make it so that we'd have a nice TLS endpoint. I'd love it too if we could have some kind of auth between Swift services, so that we'd have a better solution than just firewalling inside the cluster. (sorry if this is a bit off-topic for the current thread.)
Cheers,
Thomas Goirand (zigo)
On Mon, 2024-04-29 at 14:20 +0900, Takashi Kajinami wrote:
Hello,
As you know there has been a long discussion about eventlet usage in OpenStack, and after seeing multiple problems caused by the library and no long term plan in the project, and we are moving toward doping the dependency on eventlet.
https://review.opendev.org/c/openstack/governance/+/916546
One of the core implementation which heavily depends on eventlet is WSGI server. Multiple projects have been providing the -api console scripts (eg nova-api) based on the oslo.service's WSGI server implementation which depends on eventlet.
On the other hand, nowadays we have multiple methods to launch api services, such as httpd+mod_wsgi, uwsgi and gunicorn. And a few projects in OpenStack don't provide the -api console script (eg. aodh, barbican and placement) but suggest using these external frameworks to run their api WSGI application. I know that multiple deployment tools (for example TripleO in the past) and some distros (Debian) already started using the external framework for consistency, better concurrency and so on.
Given the situation, I wonder if we can consider deprecating the eventlet WSGI server in oslo.service in 2024.2/2025.1 and eventually remove all -api script in 2025.2. This was discussed during the previous nova PTG sessions and I think this makes complete sense as a global direction given the variety of alternative approaches currently available.
I'm tagging neutron(*1) and glance(*2) because IIRC these had(or even have) some problems with running API in wsgi mode, and also ironic because I've seen some works coming from it, iiuc. Please let me know if you have any concern applying this approach to these (or the other) projects.
Good idea. We already have a goal to migrate from WSGI scripts to Python module paths, so I wonder if we want to tack this onto that effort?
https://review.opendev.org/c/openstack/governance/+/902807
Cheers, Stephen
Thank you, Takashi Kajinami
(*1) For neutron, there are a few issues being fixed now https://bugs.launchpad.net/neutron/+bug/2062009 but ml2-ovn may need additional fixes. https://bugs.launchpad.net/neutron/+bug/1912359
(*2) As far as I know the issue with glance was already fixed some time ago[1]. https://bugs.launchpad.net/glance/+bug/1888713
On 4/29/24 20:00, Stephen Finucane wrote:
On Mon, 2024-04-29 at 14:20 +0900, Takashi Kajinami wrote:
Hello,
As you know there has been a long discussion about eventlet usage in OpenStack, and after seeing multiple problems caused by the library and no long term plan in the project, and we are moving toward doping the dependency on eventlet.
https://review.opendev.org/c/openstack/governance/+/916546
One of the core implementation which heavily depends on eventlet is WSGI server. Multiple projects have been providing the -api console scripts (eg nova-api) based on the oslo.service's WSGI server implementation which depends on eventlet.
On the other hand, nowadays we have multiple methods to launch api services, such as httpd+mod_wsgi, uwsgi and gunicorn. And a few projects in OpenStack don't provide the -api console script (eg. aodh, barbican and placement) but suggest using these external frameworks to run their api WSGI application. I know that multiple deployment tools (for example TripleO in the past) and some distros (Debian) already started using the external framework for consistency, better concurrency and so on.
Given the situation, I wonder if we can consider deprecating the eventlet WSGI server in oslo.service in 2024.2/2025.1 and eventually remove all -api script in 2025.2. This was discussed during the previous nova PTG sessions and I think this makes complete sense as a global direction given the variety of alternative approaches currently available.
I'm tagging neutron(*1) and glance(*2) because IIRC these had(or even have) some problems with running API in wsgi mode, and also ironic because I've seen some works coming from it, iiuc. Please let me know if you have any concern applying this approach to these (or the other) projects.
Good idea. We already have a goal to migrate from WSGI scripts to Python module paths, so I wonder if we want to tack this onto that effort?
That can be an independent topic, but as it touches the same area I agree that we may merge the two topics to simplify the changes we make (instead of making two commits for each topic separately, for example).
Cheers, Stephen
Thank you, Takashi Kajinami
(*1) For neutron, there are a few issues being fixed now https://bugs.launchpad.net/neutron/+bug/2062009 but ml2-ovn may need additional fixes. https://bugs.launchpad.net/neutron/+bug/1912359
(*2) As far as I know the issue with glance was already fixed some time ago[1]. https://bugs.launchpad.net/glance/+bug/1888713
Hey,
On 29.04.24 7:20 AM, Takashi Kajinami wrote:
One of the core implementation which heavily depends on eventlet is WSGI server. Multiple projects have been providing the -api console scripts (eg nova-api) based on the oslo.service's WSGI server implementation which depends on eventlet.
On the other hand, nowadays we have multiple methods to launch api services, such as httpd+mod_wsgi, uwsgi and gunicorn. And a few projects in OpenStack don't provide the -api console script (eg. aodh, barbican and placement) but suggest using these external frameworks to run their api WSGI application. I know that multiple deployment tools (for example TripleO in the past) and some distros (Debian) already started using the external framework for consistency, better concurrency and so on.
Yes please!
Apart from having the ability to use one "style" of configuration, having all components / projects uni-formally use in e.g. Gunicorn allows for much better consistency in all of them operational aspects: (Request) logging, extracting metrics regarding e.g. number of workers, their status and utilization, memory usage, latency of requests, .... Not to mention more deep-dive debugging of requests and tracing.
Regards
Christian
Hi,
To be clear, it's not *just* WSGI scripts-vs-modules, it means rewriting our WSGI service code on top of something else. We can deprecate ironic-api, but we also have the all-in-one ironic executable that is critical for our standalone case and must be migrated.
I'm -1 to this proposal until we find a better migration path than just deprecate-and-pray.
Dmitry
On 4/29/24 07:20, Takashi Kajinami wrote:
Hello,
As you know there has been a long discussion about eventlet usage in OpenStack, and after seeing multiple problems caused by the library and no long term plan in the project, and we are moving toward doping the dependency on eventlet.
https://review.opendev.org/c/openstack/governance/+/916546
One of the core implementation which heavily depends on eventlet is WSGI server. Multiple projects have been providing the -api console scripts (eg nova-api) based on the oslo.service's WSGI server implementation which depends on eventlet.
On the other hand, nowadays we have multiple methods to launch api services, such as httpd+mod_wsgi, uwsgi and gunicorn. And a few projects in OpenStack don't provide the -api console script (eg. aodh, barbican and placement) but suggest using these external frameworks to run their api WSGI application. I know that multiple deployment tools (for example TripleO in the past) and some distros (Debian) already started using the external framework for consistency, better concurrency and so on.
Given the situation, I wonder if we can consider deprecating the eventlet WSGI server in oslo.service in 2024.2/2025.1 and eventually remove all -api script in 2025.2. This was discussed during the previous nova PTG sessions and I think this makes complete sense as a global direction given the variety of alternative approaches currently available.
I'm tagging neutron(*1) and glance(*2) because IIRC these had(or even have) some problems with running API in wsgi mode, and also ironic because I've seen some works coming from it, iiuc. Please let me know if you have any concern applying this approach to these (or the other) projects.
Thank you, Takashi Kajinami
(*1) For neutron, there are a few issues being fixed now https://bugs.launchpad.net/neutron/+bug/2062009 but ml2-ovn may need additional fixes. https://bugs.launchpad.net/neutron/+bug/1912359
(*2) As far as I know the issue with glance was already fixed some time ago[1]. https://bugs.launchpad.net/glance/+bug/1888713
-- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit
On 4/29/24 21:56, Dmitry Tantsur wrote:
Hi,
To be clear, it's not *just* WSGI scripts-vs-modules, it means rewriting our WSGI service code on top of something else. We can deprecate ironic-api, but we also have the all-in-one ironic executable that is critical for our standalone case and must be migrated.
I'm -1 to this proposal until we find a better migration path than just deprecate-and-pray.
My call here is not for deprecate-and-pray but to understand impact of this change and check if we can agree that this is the direction we want to/can invest in the upcoming 2-3 cycles.
We might eventually need to keep the built-in WSGI service but I'd like to understand which / how many projects have it for what reason, before we invest too much on replacing all existing eventlet-based api services.
Dmitry
On 4/29/24 07:20, Takashi Kajinami wrote:
Hello,
As you know there has been a long discussion about eventlet usage in OpenStack, and after seeing multiple problems caused by the library and no long term plan in the project, and we are moving toward doping the dependency on eventlet.
https://review.opendev.org/c/openstack/governance/+/916546
One of the core implementation which heavily depends on eventlet is WSGI server. Multiple projects have been providing the -api console scripts (eg nova-api) based on the oslo.service's WSGI server implementation which depends on eventlet.
On the other hand, nowadays we have multiple methods to launch api services, such as httpd+mod_wsgi, uwsgi and gunicorn. And a few projects in OpenStack don't provide the -api console script (eg. aodh, barbican and placement) but suggest using these external frameworks to run their api WSGI application. I know that multiple deployment tools (for example TripleO in the past) and some distros (Debian) already started using the external framework for consistency, better concurrency and so on.
Given the situation, I wonder if we can consider deprecating the eventlet WSGI server in oslo.service in 2024.2/2025.1 and eventually remove all -api script in 2025.2. This was discussed during the previous nova PTG sessions and I think this makes complete sense as a global direction given the variety of alternative approaches currently available.
I'm tagging neutron(*1) and glance(*2) because IIRC these had(or even have) some problems with running API in wsgi mode, and also ironic because I've seen some works coming from it, iiuc. Please let me know if you have any concern applying this approach to these (or the other) projects.
Thank you, Takashi Kajinami
(*1) For neutron, there are a few issues being fixed now https://bugs.launchpad.net/neutron/+bug/2062009 but ml2-ovn may need additional fixes. https://bugs.launchpad.net/neutron/+bug/1912359
(*2) As far as I know the issue with glance was already fixed some time ago[1]. https://bugs.launchpad.net/glance/+bug/1888713
-- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit
Thanks for raising the issue.
(*1) For neutron, there are a few issues being fixed now https://bugs.launchpad.net/neutron/+bug/2062009
We merged this last week.
but ml2-ovn may need additional fixes. https://bugs.launchpad.net/neutron/+bug/1912359
This will need further work, it's on our list for this cycle now.
Thanks,
-Brian
On 4/29/24 1:20 AM, Takashi Kajinami wrote:
Hello,
As you know there has been a long discussion about eventlet usage in OpenStack, and after seeing multiple problems caused by the library and no long term plan in the project, and we are moving toward doping the dependency on eventlet.
https://review.opendev.org/c/openstack/governance/+/916546
One of the core implementation which heavily depends on eventlet is WSGI server. Multiple projects have been providing the -api console scripts (eg nova-api) based on the oslo.service's WSGI server implementation which depends on eventlet.
On the other hand, nowadays we have multiple methods to launch api services, such as httpd+mod_wsgi, uwsgi and gunicorn. And a few projects in OpenStack don't provide the -api console script (eg. aodh, barbican and placement) but suggest using these external frameworks to run their api WSGI application. I know that multiple deployment tools (for example TripleO in the past) and some distros (Debian) already started using the external framework for consistency, better concurrency and so on.
Given the situation, I wonder if we can consider deprecating the eventlet WSGI server in oslo.service in 2024.2/2025.1 and eventually remove all -api script in 2025.2. This was discussed during the previous nova PTG sessions and I think this makes complete sense as a global direction given the variety of alternative approaches currently available.
I'm tagging neutron(*1) and glance(*2) because IIRC these had(or even have) some problems with running API in wsgi mode, and also ironic because I've seen some works coming from it, iiuc. Please let me know if you have any concern applying this approach to these (or the other) projects.
Thank you, Takashi Kajinami
(*1) For neutron, there are a few issues being fixed now https://bugs.launchpad.net/neutron/+bug/2062009 but ml2-ovn may need additional fixes. https://bugs.launchpad.net/neutron/+bug/1912359
(*2) As far as I know the issue with glance was already fixed some time ago[1]. https://bugs.launchpad.net/glance/+bug/1888713
participants (7)
-
Brian Haley
-
Christian Rohmann
-
Dmitry Tantsur
-
Stephen Finucane
-
Takashi Kajinami
-
Thomas Goirand
-
Thomas Goirand