Open Stack

On Fri, 2019-03-15 at 17:06 -0500, Ben Nemec wrote:
> 
> On 3/15/19 4:41 PM, Monty Taylor wrote:
> > 
> > FWIW, we use explicit threads in Zuul and Nodepool and have been very 
> > happy with them- since they're explicit and don't require weird 
> > monkeypatching.
> > 
> > In sdk, there are a few things that need to be done in the background 
> > (when uploading a very large swift object and we're going to split it in 
> > to chunks and upload concurrently) In that case we used 
> > concurrent.futures.ThreadPoolExecutor which creates a pool of threads 
> > and allows you to submit jobs into it ... very simiar to the 
> > using-pool-of-workers example above.
> 
> This is what we did in the oslo.privsep daemon too when we added 
> parallel execution.
yes so on that point i wanted to talk to the oslo team about possibly
breaking out the core of privsep into oslo concurancy.

for a lot of the concurancy we need in nova we could create a privsep deamon context with
no permisions and simply dispatch the execution of api queries to it but that feels a bit
strainge. i was wondering if there was merrit in either packaging up the core of the deamon without
the actuall privlage seperation logic or provideing a very thin shim over the concurrent futures.
though by the looks of it there is a python2.7 backport already https://pypi.org/project/futures/
i had discounted using concurrent futrues with the ThreadPoolExecutor as i had tought it was python 3 only
but yes it provides a nice api and workflow form this type of concurrent or parallel dispatch.

its a topic i would like to discuss with people at the ptg but i would personally be interested in seeing if
we can 1 remove where resoanable our explict use of eventlets, and second explore transitioning to a more explitct
concurancy model if other were open to it either by  delegating execution to an un privalaged privesep deamon or
something like concurent futures with a thread pool or asyncio event loop using 
https://docs.python.org/3/library/asyncio-future.html. my distaste for eventlets has mellowed over the last year
or two as debugers have actully began to understand gevent and greenlets but if i had an architetural wand to change
one thin in the U cycle it would be breaking our eventlet depency and moving to a functionality form the standard libary
instead when we are python 3 only.
>