On Fri, 2019-03-15 at 17:06 -0500, Ben Nemec wrote:
On 3/15/19 4:41 PM, Monty Taylor wrote:
FWIW, we use explicit threads in Zuul and Nodepool and have been very happy with them- since they're explicit and don't require weird monkeypatching.
In sdk, there are a few things that need to be done in the background (when uploading a very large swift object and we're going to split it in to chunks and upload concurrently) In that case we used concurrent.futures.ThreadPoolExecutor which creates a pool of threads and allows you to submit jobs into it ... very simiar to the using-pool-of-workers example above.
This is what we did in the oslo.privsep daemon too when we added parallel execution.
yes so on that point i wanted to talk to the oslo team about possibly breaking out the core of privsep into oslo concurancy. for a lot of the concurancy we need in nova we could create a privsep deamon context with no permisions and simply dispatch the execution of api queries to it but that feels a bit strainge. i was wondering if there was merrit in either packaging up the core of the deamon without the actuall privlage seperation logic or provideing a very thin shim over the concurrent futures. though by the looks of it there is a python2.7 backport already https://pypi.org/project/futures/ i had discounted using concurrent futrues with the ThreadPoolExecutor as i had tought it was python 3 only but yes it provides a nice api and workflow form this type of concurrent or parallel dispatch. its a topic i would like to discuss with people at the ptg but i would personally be interested in seeing if we can 1 remove where resoanable our explict use of eventlets, and second explore transitioning to a more explitct concurancy model if other were open to it either by delegating execution to an un privalaged privesep deamon or something like concurent futures with a thread pool or asyncio event loop using https://docs.python.org/3/library/asyncio-future.html. my distaste for eventlets has mellowed over the last year or two as debugers have actully began to understand gevent and greenlets but if i had an architetural wand to change one thin in the U cycle it would be breaking our eventlet depency and moving to a functionality form the standard libary instead when we are python 3 only.