Open Stack

On Wed, 2019-03-20 at 08:53 +1100, Michael Still wrote:
> On Tue, Mar 19, 2019 at 6:57 PM Sean Mooney <smooney at redhat.com> wrote:
> > On Fri, 2019-03-15 at 17:06 -0500, Ben Nemec wrote:
> > > On 3/15/19 4:41 PM, Monty Taylor wrote:
> 
> [snip]
>  
> > > > In sdk, there are a few things that need to be done in the background 
> > > > (when uploading a very large swift object and we're going to split it in 
> > > > to chunks and upload concurrently) In that case we used 
> > > > concurrent.futures.ThreadPoolExecutor which creates a pool of threads 
> > > > and allows you to submit jobs into it ... very simiar to the 
> > > > using-pool-of-workers example above.
> > > 
> > > This is what we did in the oslo.privsep daemon too when we added 
> > > parallel execution.
> 
>  
> > yes so on that point i wanted to talk to the oslo team about possibly
> > breaking out the core of privsep into oslo concurancy.
> > 
> > for a lot of the concurancy we need in nova we could create a privsep deamon context with
> > no permisions and simply dispatch the execution of api queries to it but that feels a bit
> > strainge. i was wondering if there was merrit in either packaging up the core of the deamon without
> > the actuall privlage seperation logic or provideing a very thin shim over the concurrent futures.
> > though by the looks of it there is a python2.7 backport already https://pypi.org/project/futures/
> > i had discounted using concurrent futrues with the ThreadPoolExecutor as i had tought it was python 3 only
> > but yes it provides a nice api and workflow form this type of concurrent or parallel dispatch.
> 
> Is there a lot of gain to splitting the concurrency out of privsep? The lack of escalated permissions for a privsep'd
> function is a very small (one line IIRC) change.
> 
> I haven't looked at the API for parallel privsep to be honest. How does it handle returning results to callers?
privsep api is entrily blocking form a callers perspective.
internally privsep has a small asycronos event loop that does paralel dispatch to a 
threadpoolexecutor. so while the privsep deamon itelf will execute request in parallel
the caller of privsep rely on the eventlet context switch that happens when the priv sep decorator 
writes to the unix socket which causes the callers green tread to yeild until the result callback is
executed by the privsep deamon which wites the result back to the unix socket in either a reply or error message.

while privsep has many of the feature you would want in a parallel sidecar that you could dispatch asyc task to
it does not have the api you would want and looking at the implementation a little more closly it proably is
not ideal for performace sensivive code. the way we decorate module level function to annotate them as privsep
entrypoints means we cannot have class/instanstance methods that can be executed via privsep and while we do have
the concept of channels and futures we could do better by creating a dedicated reusable event loop that can be seeded
with an executor at creation time. the python comunity has already done this infact in the python 3 standard lib.

we could proably add the concurrent.futures executor interface to privsep if we wanted its only 3 function after all
https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor
but since there is a python 2.7 backport we may be better served just using it.

privsep is effectivly just a ThreadPoolExecutor that layers on the privlage escalation/descalation logic
via either fork and drop privalages or sudo + rootwrap/privsep helper to gain privileges.



>  
> > its a topic i would like to discuss with people at the ptg but i would personally be interested in seeing if
> > we can 1 remove where resoanable our explict use of eventlets, and second explore transitioning to a more explitct
> > concurancy model if other were open to it either by  delegating execution to an un privalaged privesep deamon or
> > something like concurent futures with a thread pool or asyncio event loop using 
> > https://docs.python.org/3/library/asyncio-future.html. my distaste for eventlets has mellowed over the last year
> > or two as debugers have actully began to understand gevent and greenlets but if i had an architetural wand to change
> > one thin in the U cycle it would be breaking our eventlet depency and moving to a functionality form the standard
> > libary
> > instead when we are python 3 only.
> 
> One thing I wanted recently was a way to start a thread which ran as a "sidecar" to nova-compute but didn't share
> anything with nova-compute apart from wanting to always be running when nova-compute was. Think the metadata server or
> a prometheus metrics exporter. I nearly wrote an implementation with privsep, but didn't get around to it. I feel like
> an unprivileged privsep parallel executor which never returns might be very close to what I wanted.
> 
> I wont be at the PTG, but would like to be kept informed of where you get to on this.
> 
> Michael