Open Stack

Josh,
Please find my reply inline.

Regards,
SURO
irc//freenode: suro-patz

On 12/16/15 6:37 PM, Joshua Harlow wrote:
>
>
> SURO wrote:
>> Hi all,
>> Please review and provide feedback on the following design proposal for
>> implementing the blueprint[1] on async-container-operations -
>>
>> 1. Magnum-conductor would have a pool of threads for executing the
>> container operations, viz. executor_threadpool. The size of the
>> executor_threadpool will be configurable. [Phase0]
>> 2. Every time, Magnum-conductor(Mcon) receives a
>> container-operation-request from Magnum-API(Mapi), it will do the
>> initial validation, housekeeping and then pick a thread from the
>> executor_threadpool to execute the rest of the operations. Thus Mcon
>> will return from the RPC request context much faster without blocking
>> the Mapi. If the executor_threadpool is empty, Mcon will execute in a
>> manner it does today, i.e. synchronously - this will be the
>> rate-limiting mechanism - thus relaying the feedback of exhaustion.
>> [Phase0]
>> How often we are hitting this scenario, may be indicative to the
>> operator to create more workers for Mcon.
>> 3. Blocking class of operations - There will be a class of operations,
>> which can not be made async, as they are supposed to return
>> result/content inline, e.g. 'container-logs'. [Phase0]
>> 4. Out-of-order considerations for NonBlocking class of operations -
>> there is a possible race around condition for create followed by
>> start/delete of a container, as things would happen in parallel. To
>> solve this, we will maintain a map of a container and executing thread,
>> for current execution. If we find a request for an operation for a
>> container-in-execution, we will block till the thread completes the
>> execution. [Phase0]
>> This mechanism can be further refined to achieve more asynchronous
>> behavior. [Phase2]
>> The approach above puts a prerequisite that operations for a given
>> container on a given Bay would go to the same Magnum-conductor instance.
>> [Phase0]
>> 5. The hand-off between Mcon and a thread from executor_threadpool can
>> be reflected through new states on the 'container' object. These states
>> can be helpful to recover/audit, in case of Mcon restart. [Phase1]
>>
>> Other considerations -
>> 1. Using eventlet.greenthread instead of real threads => This approach
>> would require further refactoring the execution code and embed yield
>> logic, otherwise a single greenthread would block others to progress.
>> Given, we will extend the mechanism for multiple COEs, and to keep the
>> approach straight forward to begin with, we will use 'threading.Thread'
>> instead of 'eventlet.greenthread'.
>>
>
> Also unsure about the above, not quite sure I connect how greenthread 
> usage requires more yield logic (I'm assuming you mean the yield 
> statement here)? Btw if magnum is running with all things monkey 
> patched (which it seems like 
> https://github.com/openstack/magnum/blob/master/magnum/common/rpc_service.py#L33 
> does) then magnum usage of 'threading.Thread' is a 
> 'eventlet.greenthread' underneath the covers, just fyi.
>
SURO> Let's consider this -
function A () {
         block B; // validation
         block C; // Blocking op
}
Now, if we make C a greenthread, as it is, would it not block the entire 
thread that runs through all the greenthreads? I assumed, it would and 
that's why we have to incorporate finer grain yield into C to leverage 
greenthread. If the answer is no, then we can use greenthread.
I will validate which version of threading.Thread was getting used.

In that case, keeping the code for thread.Threading is portable, as it 
would work as desired, even if we remove monkey_patching, right?
>>
>> Refs:-
>> [1] -
>> https://blueprints.launchpad.net/magnum/+spec/async-container-operations
>>
>
> __________________________________________________________________________ 
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev