<div dir="ltr">Hi Vish,<div>do you have more info about the libvirt deadlocks that you observed?</div><div>Maybe I'm observing the same on SLC6 where I can't even "kill" libvirtd process.</div><div><br></div><div>Belmiro</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 16, 2014 at 12:01 AM, Vishvananda Ishaya <span dir="ltr"><<a href="mailto:vishvananda@gmail.com" target="_blank">vishvananda@gmail.com</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I have seen deadlocks in libvirt that could cause this. When you are in this state, check to see if you can do a virsh list on the node. If not, libvirt is deadlocked, and ubuntu may need to pull in a fix/newer version.<br>
<br>
Vish<br>
<div class="HOEnZb"><div class="h5"><br>
On Dec 12, 2014, at 2:12 PM, pcrews <<a href="mailto:gleebix@gmail.com">gleebix@gmail.com</a>> wrote:<br>
<br>
> On 12/09/2014 03:54 PM, Ken'ichi Ohmichi wrote:<br>
>> Hi,<br>
>><br>
>> This case is always tested by Tempest on the gate.<br>
>><br>
>> <a href="https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_delete_server.py#L152" target="_blank">https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_delete_server.py#L152</a><br>
>><br>
>> So I guess this problem wouldn't happen on the latest version at least.<br>
>><br>
>> Thanks<br>
>> Ken'ichi Ohmichi<br>
>><br>
>> ---<br>
>><br>
>> 2014-12-10 6:32 GMT+09:00 Joe Gordon <<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>>:<br>
>>><br>
>>><br>
>>> On Sat, Dec 6, 2014 at 5:08 PM, Danny Choi (dannchoi) <<a href="mailto:dannchoi@cisco.com">dannchoi@cisco.com</a>><br>
>>> wrote:<br>
>>>><br>
>>>> Hi,<br>
>>>><br>
>>>> I have a VM which is in ERROR state.<br>
>>>><br>
>>>><br>
>>>> +--------------------------------------+----------------------------------------------+--------+------------+-------------+--------------------+<br>
>>>><br>
>>>> | ID | Name<br>
>>>> | Status | Task State | Power State | Networks |<br>
>>>><br>
>>>><br>
>>>> +--------------------------------------+----------------------------------------------+--------+------------+-------------+--------------------+<br>
>>>><br>
>>>> | 1cb5bf96-619c-4174-baae-dd0d8c3d40c5 |<br>
>>>> cirros--1cb5bf96-619c-4174-baae-dd0d8c3d40c5 | ERROR | - | NOSTATE<br>
>>>> | |<br>
>>>><br>
>>>><br>
>>>> I tried in both CLI “nova delete” and Horizon “terminate instance”.<br>
>>>> Both accepted the delete command without any error.<br>
>>>> However, the VM never got deleted.<br>
>>>><br>
>>>> Is there a way to remove the VM?<br>
>>><br>
>>><br>
>>> What version of nova are you using? This is definitely a serious bug, you<br>
>>> should be able to delete an instance in error state. Can you file a bug that<br>
>>> includes steps on how to reproduce the bug along with all relevant logs.<br>
>>><br>
>>> <a href="http://bugs.launchpad.net/nova" target="_blank">bugs.launchpad.net/nova</a><br>
>>><br>
>>>><br>
>>>><br>
>>>> Thanks,<br>
>>>> Danny<br>
>>>><br>
>>>> _______________________________________________<br>
>>>> OpenStack-dev mailing list<br>
>>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> OpenStack-dev mailing list<br>
>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
> Hi,<br>
><br>
> I've encountered this in my own testing and have found that it appears to be tied to libvirt.<br>
><br>
> When I hit this, reset-state as the admin user reports success (and state is set), *but* things aren't really working as advertised and subsequent attempts to do anything with the errant vm's will send them right back into 'FLAIL' / can't delete / endless DELETING mode.<br>
><br>
> restarting libvirt-bin on my machine fixes this - after restart, the deleting vm's are properly wiped without any further user input to nova/horizon and all seems right in the world.<br>
><br>
> using:<br>
> devstack<br>
> ubuntu 14.04<br>
> libvirtd (libvirt) 1.2.2<br>
><br>
> triggered via:<br>
> lots of random create/reboot/resize/delete requests of varying validity and sanity.<br>
><br>
> Am in the process of cleaning up my test code so as not to hurt anyone's brain with the ugly and will file a bug once done, but thought this worth sharing.<br>
><br>
> Thanks,<br>
> Patrick<br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div></div>