<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hongbin,<br>
Very useful pointers! Thanks for bringing up the relevant contexts!<br>
<br>
The proposal to block here for consecutive operations on same
container, is the approach to start with. We can have a wait queue
implementation following - that way the approach will be amortized
over time. If you feel strongly, I am okay implementing the wait
queue on the first go itself. <br>
[ I felt step-by-step approach carries in sizable code, easier to
review ] <br>
<br>
By the way, I think the scope of bay lock and scope of
per-bay-per-container operation is different too, in terms of
blocking.<br>
<br>
I have a confusion about non-blocking bay-operations for horizontal
scale [1] -<br>
"
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<span class="author-a-qh5m1z77zz68zdz69zoz72zcbdz70z8">Heat will be
having concurrency support, so we can rely on heat for the
concurrency issue for now and drop the baylock implementation."<br>
- if user issues two consecutive updates on a Bay, and if the
updates go through different magnum-conductors, <br>
</span>they can land up at Heat in different order, resulting in
different state of the bay. How Heat-concurrency will prevent that I
am not very clear. [ Take an example of 'magnum bay-update k8sbay
replace node_count=100' followed by 'magnum bay-update k8sbay
replace node_count=10']<br>
<br>
<br>
[1] -
<a class="moz-txt-link-freetext" href="https://etherpad.openstack.org/p/liberty-work-magnum-horizontal-scale">https://etherpad.openstack.org/p/liberty-work-magnum-horizontal-scale</a>
(Line 33)<br>
<pre class="moz-signature" cols="72">Regards,
SURO
irc//freenode: suro-patz
</pre>
<div class="moz-cite-prefix">On 12/17/15 8:10 AM, Hongbin Lu wrote:<br>
</div>
<blockquote
cite="mid:0957CD8F4B55C0418161614FEC580D6BD64B44@SZXEMI503-MBS.china.huawei.com"
type="cite">
<pre wrap="">Suro,
FYI. In before, we tried a distributed lock implementation for bay operations (here are the patches [1,2,3,4,5]). However, after several discussions online and offline, we decided to drop the blocking implementation for bay operations, in favor of non-blocking implementation (which is not implemented yet). You can find more discussion in here [6,7].
For the async container operations, I would suggest to consider a non-blocking approach first. If it is impossible and we need a blocking implementation, suggest to use the bay operations patches below as a reference.
[1] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/171921/">https://review.openstack.org/#/c/171921/</a>
[2] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/172603/">https://review.openstack.org/#/c/172603/</a>
[3] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/172772/">https://review.openstack.org/#/c/172772/</a>
[4] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/172773/">https://review.openstack.org/#/c/172773/</a>
[5] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/172774/">https://review.openstack.org/#/c/172774/</a>
[6] <a class="moz-txt-link-freetext" href="https://blueprints.launchpad.net/magnum/+spec/horizontal-scale">https://blueprints.launchpad.net/magnum/+spec/horizontal-scale</a>
[7] <a class="moz-txt-link-freetext" href="https://etherpad.openstack.org/p/liberty-work-magnum-horizontal-scale">https://etherpad.openstack.org/p/liberty-work-magnum-horizontal-scale</a>
Best regards,
Hongbin
-----Original Message-----
From: Adrian Otto [<a class="moz-txt-link-freetext" href="mailto:adrian.otto@rackspace.com">mailto:adrian.otto@rackspace.com</a>]
Sent: December-16-15 10:20 PM
To: OpenStack Development Mailing List (not for usage questions)
Cc: <a class="moz-txt-link-abbreviated" href="mailto:suro@yahoo-inc.com">suro@yahoo-inc.com</a>
Subject: Re: [openstack-dev] [magnum] Magnum conductor async container operations
</pre>
<blockquote type="cite">
<pre wrap="">On Dec 16, 2015, at 6:24 PM, Joshua Harlow <a class="moz-txt-link-rfc2396E" href="mailto:harlowja@fastmail.com"><harlowja@fastmail.com></a> wrote:
SURO wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi all,
Please review and provide feedback on the following design proposal
for implementing the blueprint[1] on async-container-operations -
1. Magnum-conductor would have a pool of threads for executing the
container operations, viz. executor_threadpool. The size of the
executor_threadpool will be configurable. [Phase0] 2. Every time,
Magnum-conductor(Mcon) receives a container-operation-request from
Magnum-API(Mapi), it will do the initial validation, housekeeping and
then pick a thread from the executor_threadpool to execute the rest
of the operations. Thus Mcon will return from the RPC request context
much faster without blocking the Mapi. If the executor_threadpool is
empty, Mcon will execute in a manner it does today, i.e.
synchronously - this will be the rate-limiting mechanism - thus
relaying the feedback of exhaustion.
[Phase0]
How often we are hitting this scenario, may be indicative to the
operator to create more workers for Mcon.
3. Blocking class of operations - There will be a class of
operations, which can not be made async, as they are supposed to
return result/content inline, e.g. 'container-logs'. [Phase0] 4.
Out-of-order considerations for NonBlocking class of operations -
there is a possible race around condition for create followed by
start/delete of a container, as things would happen in parallel. To
solve this, we will maintain a map of a container and executing
thread, for current execution. If we find a request for an operation
for a container-in-execution, we will block till the thread completes
the execution. [Phase0]
</pre>
</blockquote>
<pre wrap="">
Does whatever do these operations (mcon?) run in more than one process?
</pre>
</blockquote>
<pre wrap="">
Yes, there may be multiple copies of magnum-conductor running on separate hosts.
</pre>
<blockquote type="cite">
<pre wrap="">Can it be requested to create in one process then delete in another?
If so is that map some distributed/cross-machine/cross-process map
that will be inspected to see what else is manipulating a given
container (so that the thread can block until that is not the case...
basically the map is acting like a operation-lock?)
</pre>
</blockquote>
<pre wrap="">
That’s how I interpreted it as well. This is a race prevention technique so that we don’t attempt to act on a resource until it is ready. Another way to deal with this is check the state of the resource, and return a “not ready” error if it’s not ready yet. If this happens in a part of the system that is unattended by a user, we can re-queue the call to retry after a minimum delay so that it proceeds only when the ready state is reached in the resource, or terminated after a maximum number of attempts, or if the resource enters an error state. This would allow other work to proceed while the retry waits in the queue.
</pre>
<blockquote type="cite">
<pre wrap="">If it's just local in one process, then I have a library for u that
can solve the problem of correctly ordering parallel operations ;)
</pre>
</blockquote>
<pre wrap="">
What we are aiming for is a bit more distributed.
Adrian
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">This mechanism can be further refined to achieve more asynchronous
behavior. [Phase2] The approach above puts a prerequisite that
operations for a given container on a given Bay would go to the same
Magnum-conductor instance.
[Phase0]
5. The hand-off between Mcon and a thread from executor_threadpool
can be reflected through new states on the 'container' object. These
states can be helpful to recover/audit, in case of Mcon restart.
[Phase1]
Other considerations -
1. Using eventlet.greenthread instead of real threads => This
approach would require further refactoring the execution code and
embed yield logic, otherwise a single greenthread would block others to progress.
Given, we will extend the mechanism for multiple COEs, and to keep
the approach straight forward to begin with, we will use 'threading.Thread'
instead of 'eventlet.greenthread'.
Refs:-
[1] -
<a class="moz-txt-link-freetext" href="https://blueprints.launchpad.net/magnum/+spec/async-container-operati">https://blueprints.launchpad.net/magnum/+spec/async-container-operati</a>
ons
</pre>
</blockquote>
<pre wrap="">
______________________________________________________________________
____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<pre wrap="">
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>