<div dir="ltr">Thanks for explanation, Steven.<div>Will try to figure out why it is not working in nova.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 10, 2015 at 4:04 PM, Steven Hardy <span dir="ltr"><<a href="mailto:shardy@redhat.com" target="_blank">shardy@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Feb 10, 2015 at 03:04:39PM +0400, Kairat Kushaev wrote:<br>
> Hi all,<br>
> During the analysis of the following bug:<br>
> <a href="https://bugs.launchpad.net/heat/+bug/1418878" target="_blank">https://bugs.launchpad.net/heat/+bug/1418878</a><br>
> i figured out that orchestration engine doesn't work properly in some<br>
> cases.<br>
</span>> The case is the following:A<br>
<span class="">> trying to delete the same stack with resources n times in series.<br>
> It might happen if the stack deleting takes much time and a user is<br>
> sending<br>
> the second delete request again.<br>
> Orchestration engine behavior is the following:<br>
> 1) When first stack-delete command comes to heat service<br>
> it acquires the stack lock and sends delete request for resources<br>
> to other clients.<br>
> Unfortunately, the command does not start to delete resources from heat<br>
</span>> db.A<br>
<span class="">> 2) At that time second stack-delete command for the same stack<br>
> comes to heat engine. It steals the stack lock, waits 0.2 (hard-coded<br>
</span>> constant!)A<br>
<span class="">> sec to allow previous stack-delete command finish the operations (of<br>
</span>> course,A<br>
<span class="">> the first didn't manage to finish deleting on time). After that engine<br>
</span>> service startsA<br>
> the deleting again:<br>
> A A A - Request resources from heat DB (They exist!)<br>
> A A A - Send requests for delete to other clients (They do not exist<br>
> because ofA<br>
> A A A A point 1).<br>
<br>
This is expected, and the reason for the following error path in most<br>
resource handle_delete paths is to ignore any "do not exist errors":<br>
<br>
self.client_plugin().ignore_not_found(e)<br>
<span class=""><br>
> Finally, we have stack in DELETE_FAILED state because the clients raise<br>
> exceptions during stack delete.<br>
<br>
</span>This is the bug, the exception which is raised isn't getting ignored by the<br>
nova client plugin, which by default only ignores NotFound exceptions:<br>
<br>
<a href="https://github.com/openstack/heat/blob/master/heat/engine/clients/os/nova.py#L85" target="_blank">https://github.com/openstack/heat/blob/master/heat/engine/clients/os/nova.py#L85</a><br>
<br>
In this case, I think the problem is you're getting a Conflict exception<br>
when attempting to re-delete the NovaFloatingIpAssociation:<br>
<br>
<a href="https://github.com/openstack/heat/blob/master/heat/engine/resources/nova_floatingip.py#L148" target="_blank">https://github.com/openstack/heat/blob/master/heat/engine/resources/nova_floatingip.py#L148</a><br>
<br>
So, I think this is probably a bug specific to NovaFloatingIpAssociation<br>
rather than a problem we need to fix accross all resources?<br>
<br>
I'd probably suggest we either add another except clause which catches (and<br>
ignores) this situation, or look at if novaclient is raising the wrong<br>
exception type, as "NotFound" would appear to be a saner error than<br>
"Conflict" when trying to delete a non-existent association?<br>
<br>
Steve<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div>