[openstack-dev] [heat] heat delete woes in Juno

Kevin Benton blak111 at gmail.com
Sun Mar 29 22:27:21 UTC 2015


It's also possible you've come across a deadlock error caused by several
related objects being deleted close together in Neutron. It would be good
to check for tracebacks in the server log (usually
/var/log/neutron/server.log) to see if it's a Neutron bug here.
On Mar 29, 2015 1:38 PM, "Steve Baker" <sbaker at redhat.com> wrote:

>  On 28/03/15 09:05, Matt Fischer wrote:
>
> Pavlo,
>
>  Here is a link to one of the stacks. It is fairly simple just some
> routers/nets/subnets. The description is a bit odd perhaps, but legal. I've
> changed the template to not point at IPs at internal DNS.
>
>  http://paste.ubuntu.com/10690759/
>
>   I've not been able to reproduce with this template running my local
> Juno 2014.2.2 or a recent devstack, but I'm sure you can.
>
>  I created and deleted this in a loop about 5 times and it finally failed
> to delete on the last run. Now that it is stuck in DELETE_FAILED no amount
> of deleting will help. I'm concerned that a template this simple can get
> stuck like this.
>
>   You should be able to delete the underlying resources using the neutron
> command, then delete the stack.
>
>  I will have stack_abandon enabled next week as it just landed in Puppet:
> https://review.openstack.org/#/c/168157/ and will plan on trying that
> then.
>
>   We've needed a series of workarounds for neutron resources as there are
> often implicit dependencies created which are not declared in REST create
> calls.  Its likely you've discovered some more implicit dependencies which
> we need to handle. Could you please raise a bug [1] with the following?
>
> - the above pasted template
> - the heat event-list of the DELETE_FAILED stack
> - the heat stack-show of the DELETE_FAILED stack
>
> [1] https://bugs.launchpad.net/heat/+filebug
>
>
>
>
> On Thu, Mar 26, 2015 at 12:40 PM, Pavlo Shchelokovskyy <
> pshchelokovskyy at mirantis.com> wrote:
>
>> Hi Matt,
>>
>>  if it would be feasible/appropriate, could you provide us with
>> templates for stacks that show this behavior (try to get them with "heat
>> template-show <stack-name-or-id>")? This would help us to test and
>> understand the problem better.
>>
>>  And yes, just the day before I was contacted by one of my colleagues
>> who seems to experience similar problems with Juno-based OpenStack
>> deployment (though I did not had a chance to look through the issue yet).
>>
>>  Best regards,
>>
>>  Pavlo Shchelokovskyy
>> Software Engineer
>> Mirantis Inc
>> www.mirantis.com
>>
>>  On Thu, Mar 26, 2015 at 8:17 PM, Matt Fischer <matt at mattfischer.com>
>> wrote:
>>
>>>  Nobody on the operators list had any ideas on this, so re-posting here.
>>>
>>>  We've been having some issues with heat delete-stack in Juno. The
>>> issues generally fall into three categories:
>>>
>>>  1) it takes multiple calls to heat to delete a stack. Presumably due
>>> to heat being unable to figure out the ordering on deletion and resources
>>> being in use.
>>>
>>>  2) undeleteable stacks. Stacks that refuse to delete, get stuck in
>>> DELETE_FAILED state. In this case, they show up in stack-list and
>>> stack-show, yet resource-list and stack-delete deny their existence. This
>>> means I can't be sure whether they have any real resources very easily.
>>>
>>>  3) As a corollary to item 1, stacks for which heat can never unwind
>>> the dependencies and stay in DELETE_IN_PROGRESS forever.
>>>
>>>  Does anyone have any work-arounds for these or recommendations on
>>> cleanup? My main worry is removing a stack from the database that is still
>>> consuming the customer's resources. I also don't just want to remove stacks
>>> from the database and leave orphaned records in the DB.
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150329/0307fb82/attachment.html>


More information about the OpenStack-dev mailing list