[nova][dev] Fixing eventlet monkey patching in Nova

Monty Taylor mordred at inaugust.com
Wed Mar 20 02:22:37 UTC 2019



On 3/19/19 11:39 PM, Sean Mooney wrote:
> On Wed, 2019-03-20 at 08:53 +1100, Michael Still wrote:
>> On Tue, Mar 19, 2019 at 6:57 PM Sean Mooney <smooney at redhat.com> wrote:
>>> On Fri, 2019-03-15 at 17:06 -0500, Ben Nemec wrote:
>>>> On 3/15/19 4:41 PM, Monty Taylor wrote:
>>
>> [snip]
>>   
>>>>> In sdk, there are a few things that need to be done in the background
>>>>> (when uploading a very large swift object and we're going to split it in
>>>>> to chunks and upload concurrently) In that case we used
>>>>> concurrent.futures.ThreadPoolExecutor which creates a pool of threads
>>>>> and allows you to submit jobs into it ... very simiar to the
>>>>> using-pool-of-workers example above.
>>>>
>>>> This is what we did in the oslo.privsep daemon too when we added
>>>> parallel execution.
>>
>>   
>>> yes so on that point i wanted to talk to the oslo team about possibly
>>> breaking out the core of privsep into oslo concurancy.
>>>
>>> for a lot of the concurancy we need in nova we could create a privsep deamon context with
>>> no permisions and simply dispatch the execution of api queries to it but that feels a bit
>>> strainge. i was wondering if there was merrit in either packaging up the core of the deamon without
>>> the actuall privlage seperation logic or provideing a very thin shim over the concurrent futures.
>>> though by the looks of it there is a python2.7 backport already https://pypi.org/project/futures/
>>> i had discounted using concurrent futrues with the ThreadPoolExecutor as i had tought it was python 3 only
>>> but yes it provides a nice api and workflow form this type of concurrent or parallel dispatch.
>>
>> Is there a lot of gain to splitting the concurrency out of privsep? The lack of escalated permissions for a privsep'd
>> function is a very small (one line IIRC) change.
>>
>> I haven't looked at the API for parallel privsep to be honest. How does it handle returning results to callers?
> privsep api is entrily blocking form a callers perspective.
> internally privsep has a small asycronos event loop that does paralel dispatch to a
> threadpoolexecutor. so while the privsep deamon itelf will execute request in parallel
> the caller of privsep rely on the eventlet context switch that happens when the priv sep decorator
> writes to the unix socket which causes the callers green tread to yeild until the result callback is
> executed by the privsep deamon which wites the result back to the unix socket in either a reply or error message.
> 
> while privsep has many of the feature you would want in a parallel sidecar that you could dispatch asyc task to
> it does not have the api you would want and looking at the implementation a little more closly it proably is
> not ideal for performace sensivive code. the way we decorate module level function to annotate them as privsep
> entrypoints means we cannot have class/instanstance methods that can be executed via privsep and while we do have
> the concept of channels and futures we could do better by creating a dedicated reusable event loop that can be seeded
> with an executor at creation time. the python comunity has already done this infact in the python 3 standard lib.
> 
> we could proably add the concurrent.futures executor interface to privsep if we wanted its only 3 function after all
> https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor
> but since there is a python 2.7 backport we may be better served just using it.

It's also worth pointing out the futurist library, which is part of oslo:

https://opendev.org/openstack/futurist

It's similar to (and uses) concurrent.futures - except it also has a 
bunch of different backend impls - so you can have tasks run in threads, 
or it can use eventlet, or processes - or work synchronously.

> privsep is effectivly just a ThreadPoolExecutor that layers on the privlage escalation/descalation logic
> via either fork and drop privalages or sudo + rootwrap/privsep helper to gain privileges.
> 
> 
> 
>>   
>>> its a topic i would like to discuss with people at the ptg but i would personally be interested in seeing if
>>> we can 1 remove where resoanable our explict use of eventlets, and second explore transitioning to a more explitct
>>> concurancy model if other were open to it either by  delegating execution to an un privalaged privesep deamon or
>>> something like concurent futures with a thread pool or asyncio event loop using
>>> https://docs.python.org/3/library/asyncio-future.html. my distaste for eventlets has mellowed over the last year
>>> or two as debugers have actully began to understand gevent and greenlets but if i had an architetural wand to change
>>> one thin in the U cycle it would be breaking our eventlet depency and moving to a functionality form the standard
>>> libary
>>> instead when we are python 3 only.
>>
>> One thing I wanted recently was a way to start a thread which ran as a "sidecar" to nova-compute but didn't share
>> anything with nova-compute apart from wanting to always be running when nova-compute was. Think the metadata server or
>> a prometheus metrics exporter. I nearly wrote an implementation with privsep, but didn't get around to it. I feel like
>> an unprivileged privsep parallel executor which never returns might be very close to what I wanted.

For this case, I don't think concurrent.futures or privsep pr futurist 
is what you want. You just actually want to start a thread directly.

http://codesearch.openstack.org/?q=threading.Thread&i=nope&files=&repos=zuul,nodepool

has a bunch of examples that may be worth looking at. We've just been 
using the direct threading.Thread interface and have been very pleased 
with it at very high scale.

https://opendev.org/openstack-infra/nodepool/src/branch/master/nodepool/launcher.py#L1105-L1117

Is where the main nodepool launcher thread starts the cleanup, deleter 
and stats worker threads.

>> I wont be at the PTG, but would like to be kept informed of where you get to on this.

I'm sad that you won't be at the PTG. Your face will be missed.



More information about the openstack-discuss mailing list