<div dir="ltr">Hi, Kai,<div>It is needed to sync stack in periodic tasks even if the bay status is UPDATE_COMPLETE in my solution.</div><div><br></div><div>Thanks</div><div><br></div><div>Regards,</div><div>Wanghua</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 13, 2015 at 3:42 PM, Kai Qiang Wu <span dir="ltr"><<a href="mailto:wkqwu@cn.ibm.com" target="_blank">wkqwu@cn.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
<p><font size="2" face="sans-serif">hi Hua,</font><br>
<br>
<font size="2" face="sans-serif">My comments in blue below. please check.</font><span class=""><br>
<br>
<font size="2" face="sans-serif">Thanks</font><br>
<br>
<br>
<font size="2" face="sans-serif">Best Wishes,</font><br>
<font size="2" face="sans-serif">--------------------------------------------------------------------------------</font><br>
<font size="2" face="sans-serif">Kai Qiang Wu (吴开强 Kennan）<br>
IBM China System and Technology Lab, Beijing<br>
<br>
E-mail: <a href="mailto:wkqwu@cn.ibm.com" target="_blank">wkqwu@cn.ibm.com</a><br>
Tel: 86-10-82451647<br>
Address: Building 28(Ring Building), ZhongGuanCun Software Park, <br>
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193</font><br>
<font size="2" face="sans-serif">--------------------------------------------------------------------------------</font><br>
<font size="2" face="sans-serif">Follow your heart. You are miracle! </font><br>
<br>
</span><img width="16" height="16" src="cid:1__=C7BBF433DFB983AD8f9e8a93df938@cn.ibm.com" border="0" alt="Inactive hide details for 王华 ---08/13/2015 03:32:53 PM---Hi Kai Qiang Wu, I have some comments in line."><font size="2" color="#424282" face="sans-serif">王华 ---08/13/2015 03:32:53 PM---Hi Kai Qiang Wu, I have some comments in line.</font><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">From: </font><font size="1" face="sans-serif">王华 <<a href="mailto:wanghua.humble@gmail.com" target="_blank">wanghua.humble@gmail.com</a>></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">To: </font><font size="1" face="sans-serif">"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Date: </font><font size="1" face="sans-serif">08/13/2015 03:32 PM</font><span class=""><br>
<font size="1" color="#5F5F5F" face="sans-serif">Subject: </font><font size="1" face="sans-serif">Re: [openstack-dev] [magnum]problems for horizontal scale</font><br>
</span></p><hr width="100%" size="2" align="left" noshade style="color:#8091a5"><br>
<br>
<br>
<span class=""><font size="3" face="serif">Hi Kai Qiang Wu,</font><br>
<br>
<font size="3" face="serif">I have some comments in line.</font><br>
<br>
<font size="3" face="serif">On Thu, Aug 13, 2015 at 1:32 PM, Kai Qiang Wu <</font><a href="mailto:wkqwu@cn.ibm.com" target="_blank"><font size="3" color="#0000FF" face="serif"><u>wkqwu@cn.ibm.com</u></font></a><font size="3" face="serif">> wrote:</font>
<ul style="padding-left:9pt"><font size="2" face="sans-serif">Hi Hua,</font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
I have some comments about this:</font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
A><br>
remove heat poller can be a way, but some of its logic needs to make sure it work and performance not burden.<br>
1) for old heat poller it is quick loop, with fixed interval, to make sure stack status update quickly can be reflected in bay status<br>
2) for periodic task running, it seems dynamic loop, and period is long, it was added for some stacks creation timeout, 1) loop exit, this 2) loop can help update the stack and also conductor crash issue</font></ul>
<br>
<font size="3" face="serif"> It is not necessary to remove heat poller, so we can keep it.</font>
<ul style="padding-left:9pt"><font size="3" face="serif"><br>
<br>
</font><font size="2" face="sans-serif"><br>
It would be ideal to put in one place for looping over the stacks, but periodic tasks need to consider if it really just need to loop<br>
IN_PROGRESS status stack ? And what's the interval for loop that ? (60s or short, loop performance)</font></ul>
<br>
<font size="3" face="serif">It is necessary to loop IN_PROGRESS status stack for conductor crash issue.</font>
<ul style="padding-left:9pt"><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
Does heat have other status transition path, like delete_failed --> (status reset) --> become OK. etc.</font></ul>
<br>
<font size="3" face="serif"> It needs to be made sure.</font>
<ul style="padding-left:9pt"><font size="3" face="serif"><br>
<br>
<br>
</font><font size="2" face="sans-serif"><br>
B> For remove db operation in bay_update case. I did not understand your suggestion.<br>
bay_update include update_stack and poll_and_check(it is in heat poller), if you removed heat poller to periodic task(as you said in your 3). It still needs db operations.</font><font size="3" face="serif"><br>
</font></ul>
<br>
<font size="3" face="serif">Race conditions occur in periodic tasks too. If we save the stack params such as node_count in bay_update and race condition occurs, then the node_count in db is wrong and the status is UPDATE_COMPLETE. And there is no way to correct it.</font><br>
<font size="3" face="serif">If we save stack params in periodic tasks and race condition occurs, the node_count in db is still wrong and status is UPDATE_COMPLETE. We can correct it in the next periodic task if race condition does not occur. The solution I proposed can not promise the data in db is always right.</font>
</span><ul style="padding-left:9pt"><font size="3" face="serif"><br>
</font><font size="2" color="#0000FF" face="sans-serif"><i> Yes, it can help some, when you talked periodic task, I checked that,</i></font><br>
<br>
<font size="2" color="#0000FF" face="sans-serif"><i> filters = [bay_status.CREATE_IN_PROGRESS,</i></font></ul>
<font size="2" color="#0000FF" face="sans-serif"><i> bay_status.UPDATE_IN_PROGRESS,</i></font><br>
<font size="2" color="#0000FF" face="sans-serif"><i> bay_status.DELETE_IN_PROGRESS]</i></font><br>
<font size="2" color="#0000FF" face="sans-serif"><i> bays = objects.Bay.list_all(ctx, filters=filters)</i></font><br>
<font size="2" face="sans-serif"> </font><font size="2" color="#001FE2" face="sans-serif"><i> If UPDATE_COMPLETE, I did not find it would sync it in this task. Do you mean add that status check in this periodic task ?</i></font><span class=""><br>
<br>
<font size="2" face="sans-serif"> </font><br>
<font size="2" face="sans-serif"><br>
C> For allow admin user to show stacks in other tenant, it seems OK. Does other projects try this before? Is it reasonable case for customer ?</font><font size="3" face="serif"><br>
<br>
Nova allow admin user to show instances in other tenant. Neutron allow admin user to show ports in other tenant, nova uses it to sync up network info for instance from neutron.</font><br>
</span><font size="3" face="serif"> </font><font size="3" color="#0000FF" face="serif"><i>That would be OK, I think</i></font><div><div class="h5"><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
Thanks</font><font size="3" face="serif"><br>
<br>
</font><font size="2" face="sans-serif"><br>
Best Wishes,<br>
--------------------------------------------------------------------------------<br>
Kai Qiang Wu (吴开强 Kennan）<br>
IBM China System and Technology Lab, Beijing<br>
<br>
E-mail: </font><a href="mailto:wkqwu@cn.ibm.com" target="_blank"><font size="2" color="#0000FF" face="sans-serif"><u>wkqwu@cn.ibm.com</u></font></a><font size="2" face="sans-serif"><br>
Tel: 86-10-82451647<br>
Address: Building 28(Ring Building), ZhongGuanCun Software Park, <br>
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193<br>
--------------------------------------------------------------------------------<br>
Follow your heart. You are miracle! </font><font size="3" face="serif"><br>
<br>
</font><img src="cid:1__=C7BBF433DFB983AD8f9e8a93df938@cn.ibm.com" width="16" height="16" alt="Inactive hide details for 王华 ---08/13/2015 11:31:53 AM---any comments on this? On Wed, Aug 12, 2015 at 2:50 PM, 王华 <wan"><font size="2" color="#424282" face="sans-serif">王华 ---08/13/2015 11:31:53 AM---any comments on this? On Wed, Aug 12, 2015 at 2:50 PM, 王华 <</font><a href="mailto:wanghua.humble@gmail.com" target="_blank"><font size="2" color="#0000FF" face="sans-serif"><u>wanghua.humble@gmail.com</u></font></a><font size="2" color="#424282" face="sans-serif">> wrote:</font><font size="3" face="serif"><br>
</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
From: </font><font size="1" face="sans-serif">王华 <</font><a href="mailto:wanghua.humble@gmail.com" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>wanghua.humble@gmail.com</u></font></a><font size="1" face="sans-serif">></font><font size="1" color="#5F5F5F" face="sans-serif"><br>
To: </font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><font size="1" color="#0000FF" face="sans-serif"><u>openstack-dev@lists.openstack.org</u></font></a><font size="1" color="#5F5F5F" face="sans-serif"><br>
Date: </font><font size="1" face="sans-serif">08/13/2015 11:31 AM</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
Subject: </font><font size="1" face="sans-serif">Re: [openstack-dev] [magnum]problems for horizontal scale</font>
<ul style="padding-left:9pt"><hr width="100%" size="2" align="left" noshade><br>
<font size="3" face="serif"><br>
<br>
<br>
any comments on this?<br>
<br>
On Wed, Aug 12, 2015 at 2:50 PM, 王华 <</font><a href="mailto:wanghua.humble@gmail.com" target="_blank"><font size="3" color="#0000FF" face="serif"><u>wanghua.humble@gmail.com</u></font></a><font size="3" face="serif">> wrote: </font>
<ul style="padding-left:36pt"><font size="3" face="serif">Hi All,<br>
<br>
In order to prevent race conditions due to multiple conductors, my solution is as blew:<br>
1. remove the db operation in bay_update to prevent race conditions.Stack operation is atomic. Db operation is atomic. But the two operations together are not atomic.So the data in the db may be wrong.<br>
2. sync up stack status and stack parameters(now only node_count) from heat by periodic tasks. bay_update can change stack parameters, so we need to sync up them.<br>
3. remove heat poller, because we have periodic tasks.<br>
<br>
To sync up stack parameters from heat, we need to show stacks using admin_context. But heat don't allow to show stacks in other tenant. If we want to show stacks in other tenant, we need to store auth context for every bay. That is a problem. Even if we store the auth context, there is a timeout for token. The best way I think is to let heat allow admin user to show stacks in other tenant. <br>
<br>
Do you have a better solution or any improvement for my solution?<br>
<br>
Regards,<br>
Wanghua</font></ul>
<tt><font size="2">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: </font></tt><a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank"><tt><font size="2" color="#0000FF"><u>OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</u></font></tt></a><tt><font size="2" color="#0000FF"><u><br>
</u></font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><tt><font size="2" color="#0000FF"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</u></font></tt></a><font size="3" face="serif"><br>
</font><br>
<font size="3" face="serif"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: </font><a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank"><font size="3" color="#0000FF" face="serif"><u>OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</u></font></a><font size="3" color="#0000FF" face="serif"><u><br>
</u></font><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><font size="3" color="#0000FF" face="serif"><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</u></font></a><font size="3" face="serif"><br>
</font></ul>
<tt><font size="2">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</font></tt><tt><font size="2"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></font></tt><tt><font size="2"><br>
</font></tt><br>
</div></div><p></p></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>