Open Stack

On Wed, 20 Mar 2019 07:39:20 +0000, Sean Mooney <smooney at redhat.com> wrote:
> On Wed, 2019-03-20 at 02:22 +0000, Monty Taylor wrote:
>>> we could proably add the concurrent.futures executor interface to privsep if we wanted its only 3 function after all
>>> https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor
>>> but since there is a python 2.7 backport we may be better served just using it.
>>
>> It's also worth pointing out the futurist library, which is part of oslo:
>>
>> https://opendev.org/openstack/futurist
>>
>> It's similar to (and uses) concurrent.futures - except it also has a
>> bunch of different backend impls - so you can have tasks run in threads,
>> or it can use eventlet, or processes - or work synchronously.
> 
> thanks i was looking for this.
> i was aware that oslo.messaging to name buy one lib supported
> a configurable executor.
> 
> i had expected to find those executos in oslo.concurancy but
> looking a futurist it makes sense.
> 
> i see the executors defiend in https://opendev.org/openstack/futurist/src/branch/master/futurist/_futures.py
> do indeed derive form the concurent future executor abstract base class.
> 
> the map function is in the executor base class is internally in terms of the submit function so the futuist
> executors are compatiable with the standard lib varients from python3.
> 
> i wonder if it would make sense to crete an optional dependency on privsep from futurist and add a privsep executor?
> 
> pivoting back to melenie's original question regarding what alternative we have for nova we could change
> our explict use of eventlets to the GreenThreadPoolExecutor
> https://opendev.org/openstack/futurist/src/branch/master/futurist/_futures.py#L322
> this would still require and use eventlets under the hood so the perfromace would be the same as our current explict
> use of the eventlet event loop but would allow us to change the eventloop via config the same way oslo.messaging
> allows changing for the eventlet executor to the threading executor.
> 
> if we removed all explcit use of eventlet this way we could then start looking at our implit use of eventlets which
> is largely for db acess, rest api calls to other services and some virt dirver specific calls.
> 
> anyway i had assumed that revisiting dropping eventlets form services would be a U conversation at the earliest
> but if we swap to using the futurist executors for the multi cell scater gather list operation we could stop
> monkeypatching under wsgi again and instead use the ThreadPoolExecutor or ProcessPoolExecutor in that case.
> when running the api without wsgi we can revert to the GreenThreadPoolExecutor.

Thanks for all this info. I've taken a hack at replacing direct eventlet 
use with futurist.GreenThreadPoolExecutor in scatter_gather_cells as a 
first step:

https://review.openstack.org/650172

-melanie

>>
>>> privsep is effectivly just a ThreadPoolExecutor that layers on the privlage escalation/descalation logic
>>> via either fork and drop privalages or sudo + rootwrap/privsep helper to gain privileges
> 
>