[openstack-dev] [heat] Repeating stack-delete many times

Kairat Kushaev kkushaev at mirantis.com
Tue Feb 10 13:47:11 UTC 2015


Thanks for explanation, Steven.
Will try to figure out why it is not working in nova.

On Tue, Feb 10, 2015 at 4:04 PM, Steven Hardy <shardy at redhat.com> wrote:

> On Tue, Feb 10, 2015 at 03:04:39PM +0400, Kairat Kushaev wrote:
> >    Hi all,
> >    During the analysis of the following bug:
> >    https://bugs.launchpad.net/heat/+bug/1418878
> >    i figured out that orchestration engine doesn't work properly in some
> >    cases.
> >    The case is the following:A
> >    trying to delete the same stack with resources n times in series.
> >    It might happen if the stack deleting takes much time and a user is
> >    sending
> >    the second delete request again.
> >    Orchestration engine behavior is the following:
> >    1) When first stack-delete command comes to heat service
> >    it acquires the stack lock and sends delete request for resources
> >    to other clients.
> >    Unfortunately, the command does not start to delete resources from
> heat
> >    db.A
> >    2) At that time second stack-delete command for the same stack
> >    comes to heat engine. It steals the stack lock, waits 0.2 (hard-coded
> >    constant!)A
> >    sec to allow previous stack-delete command finish the operations (of
> >    course,A
> >    the first didn't manage to finish deleting on time). After that engine
> >    service startsA
> >    the deleting again:
> >    A  A  A - Request resources from heat DB (They exist!)
> >    A  A  A - Send requests for delete to other clients (They do not exist
> >    because ofA
> >    A  A  A  A  point 1).
>
> This is expected, and the reason for the following error path in most
> resource handle_delete paths is to ignore any "do not exist errors":
>
>   self.client_plugin().ignore_not_found(e)
>
> >    Finally, we have stack in DELETE_FAILED state because the clients
> raise
> >    exceptions during stack delete.
>
> This is the bug, the exception which is raised isn't getting ignored by the
> nova client plugin, which by default only ignores NotFound exceptions:
>
>
> https://github.com/openstack/heat/blob/master/heat/engine/clients/os/nova.py#L85
>
> In this case, I think the problem is you're getting a Conflict exception
> when attempting to re-delete the NovaFloatingIpAssociation:
>
>
> https://github.com/openstack/heat/blob/master/heat/engine/resources/nova_floatingip.py#L148
>
> So, I think this is probably a bug specific to NovaFloatingIpAssociation
> rather than a problem we need to fix accross all resources?
>
> I'd probably suggest we either add another except clause which catches (and
> ignores) this situation, or look at if novaclient is raising the wrong
> exception type, as "NotFound" would appear to be a saner error than
> "Conflict" when trying to delete a non-existent association?
>
> Steve
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150210/e43fb5d7/attachment.html>


More information about the OpenStack-dev mailing list