<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<STYLE type=text/css> <!--@import url(scrollbar.css); --></STYLE>
<META content="text/html; charset=utf-8" http-equiv=Content-Type>
<STYLE> body{FONT-SIZE:12pt; FONT-FAMILY:宋体,serif;} </STYLE>
<META name=GENERATOR content="MSHTML 10.00.9200.16521"><BASE
target=_blank></HEAD>
<BODY
style="BORDER-LEFT-WIDTH: 0px; BORDER-RIGHT-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; MARGIN: 12px; LINE-HEIGHT: 1.3; BORDER-TOP-WIDTH: 0px"
marginheight="0" marginwidth="0">
<DIV><FONT color=#000000 size=3 face=宋体>Hi John,</FONT></DIV>
<DIV> </DIV>
<DIV>I agree with you about that
'terminate should be able to happen at any time.'</DIV>
<DIV> </DIV>
<DIV>And I have checked the terminate_instance and reboot_instance method in
compute manager,</DIV>
<DIV>but they just log a warning while catching the
InstanceNotFound exception, nothing else.</DIV>
<DIV> </DIV>
<DIV>And last but not least, I believe that the
InstanceNotFound exception should not be raised while this race condition
occurs,</DIV>
<DIV>because the bug <A
href="https://bugs.launchpad.net/nova/+bug/1246181">https://bugs.launchpad.net/nova/+bug/1246181</A></DIV>
<DIV>just resulting in an instance going to be a 'running deleted' one
after termination.</DIV>
<DIV> </DIV>
<DIV>I want to re-explain the race condition here:</DIV>
<DIV>1. soft reboot an instance</DIV>
<DIV>2. the 'soft reboot' thread waits for the instance becoming 'shutdown'
state(this may be a long period if the instance doesn't install acpid
service)</DIV>
<DIV>3. terminate the instance during step #2</DIV>
<DIV>4. the 'terminate' thread waits for the instance becoming 'shutdown' state
through a endless loopingcall '_wait_for_destroy', too</DIV>
<DIV>4. if the 'soft reboot' thread finds the instance becomed to 'shutdown'
state firstly, and re-create/restart it before the 'terminate' thread, then the
instance will stick to 'deleting' status and couldn't be deleted again, because
the 'terminate' thread lost itself in the endless loopingcall
'_wait_for_destroy'(the instance will never become to 'shutdown' state) and the
lock in 'terminate_instance', this is the bug <A
href="https://bugs.launchpad.net/nova/+bug/1111213">https://bugs.launchpad.net/nova/+bug/1111213</A> which
has been fixed.</DIV>
<DIV>5. and on the other hand, if the 'terminate' thread finds the instance
becoming to 'shutdown' state firstly in the loopingcall '_wait_for_destroy', and
the 'soft reboot' thread re-create/restart it just before the 'terminate' thread
deletes the instance files in the instance dir(disk, disk.local, libvirt.xml,
console.log and so on), then the 'terminate' thread finishes successfully, the
instance is deleted in the nova db, but it is still running in the hypervisor,
this is the bug I want to fix this time <A
href="https://bugs.launchpad.net/nova/+bug/1246181">https://bugs.launchpad.net/nova/+bug/1246181</A>,
you can find my FIXME comment here: <A
href="https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L924">https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L924</A></DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV align=left><FONT color=#c0c0c0 size=2 face=Verdana>2013-11-12</FONT></DIV>
<DIV align=left><FONT size=2 face=Verdana>
<HR id=SignNameHR style="HEIGHT: 2px; WIDTH: 122px" align=left SIZE=2>
</FONT></DIV>
<DIV align=left><FONT color=#c0c0c0 size=2 face=Verdana><SPAN
id=_FlashSignName>Wangpan</SPAN></FONT></DIV>
<DIV><FONT size=2 face=Verdana>
<HR>
</FONT></DIV>
<DIV><FONT size=2 face=Verdana><STRONG>发件人：</STRONG>John Garbutt
<john@johngarbutt.com></FONT></DIV>
<DIV><FONT size=2
face=Verdana><STRONG>发送时间：</STRONG>2013-11-11 20:11</FONT></DIV>
<DIV><FONT size=2 face=Verdana><STRONG>主题：</STRONG>Re: [openstack-dev] [nova]
How to fix the race condition issue between deleting and soft
reboot?</FONT></DIV>
<DIV><FONT size=2 face=Verdana><STRONG>收件人：</STRONG>"OpenStack Development
Mailing List (not for usage
questions)"<openstack-dev@lists.openstack.org></FONT></DIV>
<DIV><FONT size=2 face=Verdana><STRONG>抄送：</STRONG></FONT></DIV>
<DIV><FONT size=2 face=Verdana></FONT> </DIV>
<DIV><FONT size=2 face=Verdana>
<DIV>It seems we still agreed that terminate should be able to happen at any time. </DIV>
<DIV> </DIV>
<DIV>I thought I remembered some code in the manager that treats </DIV>
<DIV>InstanceNotFound errors differently. </DIV>
<DIV> </DIV>
<DIV>I would rather we ensure InstanceNotFound is raised to indicate we </DIV>
<DIV>have hit this race condition, and let the compute manager unify how we </DIV>
<DIV>deal with that across all sorts of operations. </DIV>
<DIV> </DIV>
<DIV>John </DIV>
<DIV> </DIV>
<DIV>On 11 November 2013 02:57, Wangpan <hzwangpan@corp.netease.com> wrote: </DIV>
<DIV>> Hi all, </DIV>
<DIV>> </DIV>
<DIV>> I want to re-ask this problem after the Hongkong summit, you may have time </DIV>
<DIV>> to discuss this issue now. </DIV>
<DIV>> Thanks a lot! </DIV>
<DIV>> </DIV>
<DIV>> 2013-11-11 </DIV>
<DIV>> ________________________________ </DIV>
<DIV>> Wangpan </DIV>
<DIV>> ________________________________ </DIV>
<DIV>> 发件人："Wangpan"<hzwangpan@corp.netease.com> </DIV>
<DIV>> 发送时间：2013-11-04 12:08 </DIV>
<DIV>> 主题：[openstack-dev] [nova] How to fix the race condition issue between </DIV>
<DIV>> deleting and soft reboot? </DIV>
<DIV>> 收件人："OpenStack Development Mailing List (not for usage </DIV>
<DIV>> questions)"<openstack-dev@lists.openstack.org> </DIV>
<DIV>> 抄送： </DIV>
<DIV>> </DIV>
<DIV>> Hi all, </DIV>
<DIV>> </DIV>
<DIV>> I have a question about fixing a race condition issue between deleting and </DIV>
<DIV>> soft reboot, </DIV>
<DIV>> the issue is that: </DIV>
<DIV>> 1. If we soft reboot an instance, and then delete it, the instance may not </DIV>
<DIV>> be deleted and stand on deleting task state, this is because the bug below, </DIV>
<DIV>> https://bugs.launchpad.net/nova/+bug/1111213 </DIV>
<DIV>> and I have fixed this bug yet several months ago(just for libvirt driver). </DIV>
<DIV>> 2. The other issue is, if the instance is rebooted just before deleting the </DIV>
<DIV>> files under instance dir, then it may become to a running deleted one, and </DIV>
<DIV>> this bug is at below: </DIV>
<DIV>> https://bugs.launchpad.net/nova/+bug/1246181 </DIV>
<DIV>> I want to fix it now, and I need your advice. </DIV>
<DIV>> The commit is here: https://review.openstack.org/#/c/54477/ , you can post </DIV>
<DIV>> your advice on gerrit or mail to me. </DIV>
<DIV>> </DIV>
<DIV>> The ways to fix bug #2 may be these(just for libvirt driver in my mind): </DIV>
<DIV>> 1. Add a lock to reboot operation like the deleting operation, so the reboot </DIV>
<DIV>> operation and the delete operation will be done in sequence. </DIV>
<DIV>> But on the other hand, the soft reboot operation may cost 120s if the </DIV>
<DIV>> instance doesn't support graceful shutdown, I think it is too long for a </DIV>
<DIV>> user to delete an instance, so this may not be the best way. </DIV>
<DIV>> 2. Check the instance state at the last of _cleanup method in libvirt </DIV>
<DIV>> driver, and if it is still running, destroy it again. </DIV>
<DIV>> This way is usable but both Nikola Dipanov and I don't like this 'ugly' way. </DIV>
<DIV>> 3. Check the instance vm state and task state in nova db before booting in </DIV>
<DIV>> reboot, if it is deleted/deleting, stop the reboot process, this will access </DIV>
<DIV>> db at driver level, it is a 'ugly' way, too. </DIV>
<DIV>> </DIV>
<DIV>> Nikola suggests that 'maybe we can leverage task/vm states and refactor how </DIV>
<DIV>> reboot is done so we can back out of a reboot on a delete', but I think we </DIV>
<DIV>> should let user delete an instance at any time and any state, so the delete </DIV>
<DIV>> operation during 'soft reboot' may not be forbidden. </DIV>
<DIV>> </DIV>
<DIV>> Thanks and waiting for your voice! </DIV>
<DIV>> </DIV>
<DIV>> 2013-11-04 </DIV>
<DIV>> ________________________________ </DIV>
<DIV>> Wangpan </DIV>
<DIV>> </DIV>
<DIV>> _______________________________________________ </DIV>
<DIV>> OpenStack-dev mailing list </DIV>
<DIV>> OpenStack-dev@lists.openstack.org </DIV>
<DIV>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev </DIV>
<DIV>> </DIV>
<DIV> </DIV>
<DIV>_______________________________________________ </DIV>
<DIV>OpenStack-dev mailing list </DIV>
<DIV>OpenStack-dev@lists.openstack.org </DIV>
<DIV>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev </DIV></FONT></DIV></BODY></HTML>