This is indeed interesting times, thanks for driving the discussion on Eventlet and everything
that follows down that rabbit hole, Herve!

I’ve always wondered why there is no oslo.api part that consolidated the usage of Flask, Pecan etc
the same way that oslo.service were meant to consolidate the service layer (but being pluggable with a “backend”).

Sorry for the noice thinking out loud, there probably is (by now) ancient mailing list thread, blueprint or wiki article about that somewhere.

/Tobias

On 30 Oct 2024, at 15:55, Herve Beraud <hberaud@redhat.com> wrote:



Le mer. 30 oct. 2024 à 13:05, Stephen Finucane <stephenfin@redhat.com> a écrit :
On Mon, 2024-10-28 at 17:58 +0100, Herve Beraud wrote:
Hey folks,

FYI, we think that his topic overlaps a bit with some aspects of the Eventlet Removal Initiative. Indeed, removing Eventlet means removing our usage of the WSGI features offered by Eventlet.


I invite you to read this parallel discussion and I wanted to share with you the HTTP SGI working group =>  https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group

IMO we could capitalize from each other by having cross discussion.
Choosing the next SGI candidate will dictate how we deal with HTTP for the coming decade, so I think we have to put our focus on it ASAP.
We created comparisons between the alternatives at our disposal, here is an example with the HTTP/SGI comparison https://wiki.openstack.org/wiki/SGI-servers-comparison

I took a read through these. Just to be clear, are we suggesting projects change their web frameworks or that DevStack/DevStack plugins change the WSGI server used? If so, why? 

As Eventlet is going to be removed, its WSGI server too, I think we will mostly try to find a standardized way to replace it. 

On our side I think the discussion is mostly about projects relying on Eventlet needing to change their WSGI server and so it potentially opens the doors of a wider discussion, to avoid finishing with 10 different ways to serve HTTP. Maybe uWSGI and WSGI is a credible short term solution, but if components express needs to explore something like ASGI or other alternatives to uWSGI, then I think we should have some consultations, because in some way these 2 topics (DevStack and Eventet removal) overlap.

The question is not about requesting to change the DevStack WSGI server, but if we decide to follow some uniformity, then the WSGI aspect of DevStack would also have to join the discussion about standardization.

Some ASGI stacks can handle sync and async at the same time, so, if people push this path and if the discussion is going this way, maybe it would be worth the effort to transform DevStack too.

On the other hand, if DevStack encourages uWSGI, probably uWSGI would be the default option for escaping from Eventlet, but locking us in a limited support of async. As uWSGI is in maintenance mode, we should not expect new updates from uWSGI to introduce/increase async support.
 
I understand that some projects might want to explore ASGI, which would necessitate something other than our current Apache+mod_wsgi+uWSGI combo for the server and whatever web framework they're using for the server, but assuming they are going to continue using WSGI then any change of server or framework would seem to be be secondary (if not entirely unrelated) to the eventlet removal goal. While e.g. nova might have a few reasons to explore long-term alternatives to Paste+WebOb, eventlet removal doesn't feel like one of them.

Stephen

Please find all the useful details in the official wiki: https://wiki.openstack.org/wiki/Eventlet-removal
We strongly encourage a collaborative decision, to avoid having multiple ways to manage things.
This specialized working group would be responsible for producing a unified approach that will simplify maintenance and enhance user experience.

Thanks for your time and consideration.


Le jeu. 17 oct. 2024 à 13:06, Stephen Finucane <stephenfin@redhat.com> a écrit :
On Wed, 2024-10-16 at 14:16 +0200, Tobias Urdin wrote:
> Hello,
> 
> This is an interesting topic and IMO has been in a weird state for years
> and years and I can only deduct that this has been because of, yet
> again, the broad spectrum of ways that OpenStack can be deployed.
> 
> Every guide, every deployment tool, every project has their own ways;
> supported way, and recommend way, of deploying and multiple projects has
> either bugs with one but no the other or the other way around making
> deployment even harder.
> 
> Specially on this topic though, this drops testing of one of those ways,
> is the plan/should we interpret this as a guideline on what developers
> need to test their features? Should we take this as advice on how to deploy
> the services and start dropping any usage of mod_wsgi in deployment
> tools? If no, should we expect mod_wsgi to break when eventlet is phased
> out because it will not be tested?
> 
> I think this topic really need to be a specification (and perhaps a community
> goal) around what deliverables should support.
> 
> If we want to take measures, perhaps it's time to consolidate and minimize the headache for deployers and just drop everything except for one way, the right way (that we decide and then stick with).

I think our main question is the same as the one expressed by Tobias in this thread ^. Consolidate. But consolidate without ignoring potential coming async needs. 

> 
> I don't want this to sound like throwing shade at this effort, the
> opposite I want it to move along, but in a clear way, and just perhaps
> we should just drop everything and use oslo.service and go back to how
> it was before mod_wsgi was recommended now that we're getting rid of
> eventlet and just deploy more services instead of using something like
> mod_wsgi to handle N copies of services.

In theory, so long as services support WSGI then it shouldn't matter what we
WSGI HTTP server we use: they all speak WSGI. Sure, the configuration files are
going to look different and the performance and feature sets will vary, but IMO
that's a deployment problem/decision rather than something the services (or our
CIs) need to worry about. 

And as you said, having plurality in our WSGI servers would lead us to deployment diversity.
If we put the ASGI aspect on top of that, we will obtain another layer of complexity.
I think the current DevStack discussion is in some sort the other face of the eventlet removal piece

Let me know if I answered your question.
 
I certainly don't feel *I* am in a position to state
that e.g. gunicorn should be preferred to a long-standing alternative like uWSGI
or mod_wsgi, an ASGI-first implementation like Daphne or uvicorn, or a newcomer
like nginx unit, and this is a position apparently also shared by both the
Django [1] or Flask [2] communities (to pick two well-known examples). Maybe
deployers could share feedback on their chosen solution and put guidance in the
deployment docs but again, that's out of scope for a DevStack change.

In addition, I should highlight that we are not currently testing against
anything but uWSGI and, in rare cases, eventlet server. A grok for the various
config options I mentioned reveals nothing of interest [3][4] (for example, the
only override of 'WSGI_MODE' I can find is in monasca-api which, based on recent
attempts to push stuff to monasca projects, is either dead or on life support).
Nothing is changing here and for the most part we're just removing dead code and
pushing a few long-running efforts forwards towards their conclusion. This is
also true for the removal of support for deploying with eventlet server: as I
noted, neither Cinder nor Nova actually test these paths any more, and Neutron
is well on the way to doing the same. The only services that appear to be long-
term interested in testing with eventlet server are not those interested in
using eventlet for their API service but rather those that need to support a
standalone mode, with multiple servers provided by a single binary. Again, as I
noted below, this list includes Glance, Ironic, and possibly Swift (???).
There's definitely a discussion to be had around what the replacement for
eventlet server is for these projects, but that's being tackled elsewhere [5]
and is not something I want to wade into here.

I hope this clarifies my suggestions here,
Stephen

> 
> /Tobias
> 
> On Wed, Oct 16, 2024 at 11:12:03AM UTC, Stephen Finucane wrote:
> > o/
> > 
> > tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from
> > DevStack. I'd also like to remove the ability to deploy services under eventlet,
> > where this feature is not currently being tested, and deprecate it where it is.
> > 
> > I took a foray through DevStack last week and realised that, depending on the
> > service in question, there are currently 3 possible ways an API can be deployed
> > by DevStack:
> > 
> > * Eventlet server
> > * Apache + uWSGI
> > * Apache + mod_wsgi
> > 
> > In addition, there are a number of overlapping config options that could be set
> > to modify which of these was in use:
> > 
> > * ENABLE_HTTPD_MOD_WSGI_SERVICES
> > * WSGI_MODE
> > * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc.
> > 
> > Potential confusion aside, there are two issues I see with this. Firstly, and
> > most obviously, we are working on eventlet removal, which will necessarily
> > involve eventlet server removal [1]. In addition, there is a goal [2][3] to
> > remove use of '.wsgi' scripts in favour of Python module paths. Quite a few
> > services have migrated to this already, and those that haven't are going to need
> > to do so by time pip 25.0 comes out (since that removes the deprecated legacy
> > editable install flow [4] meaning these files will not be generated by pbr and
> > DevStack deployments will start failing...you have been warned). However,
> > mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in
> > DevStack still before last week, so to complete the wsgi script -> module
> > migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're
> > going to have to either (a) package a .wsgi script in DevStack or (b) remove the
> > ability to deploy with mod_wsgi.
> > 
> > My immediate thought on this was to simply remove all non-uWSGI deployment paths
> > immediately and deprecate the config options for removal to force DevStack
> > plugins to adapt. However, some services still want to test eventlet server for
> > their API while they stabilise support for standard WSGI-based deployments
> > (neutron), while others explicitly rely on evenlet to generate a single
> > standalone binary (ironic, glance, ...). That means we can't just go remove
> > everything now and must instead take a phased approach. I would therefore like
> > to propose we do the following:
> > 
> >  * Remove the ability to deploy services under mod_wsgi from DevStack and
> >    deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can
> >    continue to use mod_wsgi but DevStack won't.
> >  * Entirely remove the ability to deploy under eventlet from services that are
> >    not currently testing this. This includes Nova and Cinder.
> >  * Invert the default for services that are testing eventlet, so that they test
> >    with uWSGI by default now and jobs will need to opt-in to eventlet rather
> >    than opt-in to uWSGI. I think this just affects Neutron.
> >  * Optionally, and likely only one Neutron is no longer testing eventlet (next
> >    cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI
> >    config options entirely and replace it with {SERVICE}_STANDALONE options like
> >    Glance has. This will affect Ironic and maybe more.
> > 
> > I have proposed a series that will do much of this. The WIP patches indicate the
> > things that I am less clear about, such as Swift's readiness for deploying under
> > uWSGI. I would be grateful if folks from the individual service teams could take
> > a look and let me know what they think:
> > 
> > https://review.opendev.org/c/openstack/devstack/+/932193/1
> > 
> > Cheers,
> > 
> > Stephen
> > 
> > [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/PY6NUNYFU63XOIZHDDBONJDHRXJMZF5J/
> > [2] https://review.opendev.org/c/openstack/governance/+/902807
> > [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/5RQGZZMCKKHAVH4OLSJCLBQNIHYDPGZD/
> > [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/
> > 
> 



-- 
Hervé Beraud
Senior Software Engineer at Red Hat
irc: hberaud



-- 
Hervé Beraud
Senior Software Engineer at Red Hat
irc: hberaud