Open Stack

The latest patch is here, and finally passes CI:
https://review.openstack.org/#/c/626952/ . Note that it's been through
a bunch of iterations as I've played whack-a-bug, but I finally feel
pretty good about it.

A point of clarification: I believe this bug *is* a regression from
Rocky. I wasn't previously clear on that myself, but looking at the
history I have convinced myself, at least, that this is the case. In
Rocky, wsgi apps were not monkey patched, and they did not require it.
https://review.openstack.org/#/c/592698/ (Batch results per cell when
doing cross-cell listing) landed on 25th August 2018, which regressed
wsgi apps by introducing an eventlet dependency in nova-api without
adding monkey patching to wsgi apps.
https://review.openstack.org/#/c/592285/ (Make monkey patch work in
uWSGI mode) added monkey patching to wsgi apps, but added it late in
the import order such that it still fails for some library version
combinations. My patch moves the monkey patching of wsgi apps much
earlier in the import order, which finally fixes the original
regression introduced by cross-cell listing.

The whole eventlet monkey patching situation is still a hack, and this
change is really just rearranging the sticky tape. However, a fix is
necessary Stein which was not required for Rocky. For Train/U it would
be great to start ripping it out.

Matt

On Mon, 18 Mar 2019 at 12:04, Matthew Booth <mbooth at redhat.com> wrote:
>
> On Fri, 15 Mar 2019 at 19:00, Chris Dent <cdent+os at anticdent.org> wrote:
> >
> > On Fri, 15 Mar 2019, Ben Nemec wrote:
> >
> > > According to [0], the monkey-patching wasn't done in nova/__init__.py because
> > > it was supposed to be able to run without eventlet under WSGI. If that's not
> > > the case, then it's probably fine to do it there.
> > >
> > > 0:
> > > http://specs.openstack.org/openstack/openstack-specs/specs/eventlet-best-practices.html
> > >
> > > "Monkey patching should also be done in a way that allows services to run
> > > without it, such as when an API service runs under Apache. This is the reason
> > > for Nova not simply monkey patching in nova/__init__.py."
> >
> > Yeah, I think several people have run through the eventlet
> > initialization at various times, including me, with various results,
> > none of them quite good enough.
> >
> > The mod_wsgi/uwsgi side of things strived to be eventlet free as it
> > makes for weirdness, and at some point I did some work to be sure it
> > never showed up in placement [1] while it was still in nova. A side
> > effect of that work was that it also didn't need to show up in the
> > nova-api unless you used the cmd line script version of it. At the
> > same time I also explored not importing the world, and was able to
> > get some improvements (mostly by moving things out of __init__.py in
> > packages that had deeply nested members) but not as much as would
> > have liked.
> >
> > However, we later (as mentioned elsewhere in the thread) made
> > getting cell mappings parallelize, bring back the need for eventlet,
> > it seems.
> >
> > >> TL;DR I want to move monkey patching to nova/__init__.py because, for
> > >> now at least, we always require it anyway. When we eventually remove
> > >> the requirement for wsgi services to monkey patch we can move monkey
> > >> patching back to nova/cmd/__init__.py.
> > >
> > > This sounds fine to me, but it's been a loooong time since I looked at this
> > > and I'm not familiar with some of the more recent changes you talked about,
> > > so take my opinion with a grain of salt.
> >
> > This does seem like the expedient solution (assuming it works), but
> > if we continue to use eventlet anywhere in the api service it will
> > continue to introduce difficulties and inflexibility with deployment
> > choices as it will interact poorly with at least some of the
> > web-servers that could be chosen to run the nova API.
>
> Thanks for the feedback, folks.
>
> I appreciate the desire to drop eventlet entirely (elsewhere in this
> thread) but as has also been noted that's not a short term option.
> Probably feasible for U, though, and lets keep it in mind. On that
> note, my preference for multi-threaded in Nova has always been to use
> Python's own apis except when that's not possible. Eventlet monkey
> patches these to do its own thing when it's in use, and when not using
> eventlet this obviously works fine.
>
> I think I'm hearing that moving monkey patching to the top level is
> the right solution for now. I'm going to go ahead and knock that up.
>
> Matt
> --
> Matthew Booth
> Red Hat OpenStack Engineer, Compute DFG
>
> Phone: +442070094448 (UK)



-- 
Matthew Booth
Red Hat OpenStack Engineer, Compute DFG

Phone: +442070094448 (UK)