<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div id="yiv6805405000"><div id="yui_3_16_0_1_1413985109205_12106"><div style="color:#000;background-color:#fff;font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;" id="yui_3_16_0_1_1413985109205_12105"><div dir="ltr" id="yiv6805405000yui_3_16_0_1_1413985109205_11009"><span id="yiv6805405000yui_3_16_0_1_1413985109205_11015">Thanks for the background info! I'll see what I can suss out since it's easily (100%) reproducable for me.</span></div><div dir="ltr" id="yiv6805405000yui_3_16_0_1_1413985109205_11009"><span><br></span></div><div dir="ltr" id="yiv6805405000yui_3_16_0_1_1413985109205_11009"><span>I'll keep y'all posted although it may be a day or two before I can get back to it.</span></div><div dir="ltr" id="yiv6805405000yui_3_16_0_1_1413985109205_11009"><br></div><div dir="ltr" id="yiv6805405000yui_3_16_0_1_1413985109205_11009"><span id="yui_3_16_0_1_1413985109205_12224">Ken </span></div><br clear="none"> <div class="qtdSeparateBR"><br><br></div><div class="yiv6805405000yqt8228632494" id="yiv6805405000yqt54822"><div id="yiv6805405000yui_3_16_0_1_1413985109205_10995" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div id="yiv6805405000yui_3_16_0_1_1413985109205_10994" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div dir="ltr" id="yiv6805405000yui_3_16_0_1_1413985109205_10993"> <hr id="yiv6805405000yui_3_16_0_1_1413985109205_11007" size="1"> <font id="yiv6805405000yui_3_16_0_1_1413985109205_10992" size="2" face="Arial"> <b><span style="font-weight:bold;">From:</span></b> Zane Bitter <zbitter@redhat.com><br clear="none"> <b><span style="font-weight:bold;">To:</span></b> Ken Thomas <krt@yahoo-inc.com>; "openstack@lists.openstack.org" <openstack@lists.openstack.org> <br clear="none"> <b><span style="font-weight:bold;">Sent:</span></b> Wednesday, October 22, 2014 6:10 AM<br clear="none"> <b><span style="font-weight:bold;">Subject:</span></b> Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE<br clear="none"> </font> </div> <div class="yiv6805405000y_msg_container" id="yiv6805405000yui_3_16_0_1_1413985109205_10996"><br clear="none">Moving this to the openstack@ list...<br clear="none"><br clear="none">On 21/10/14 18:56, Ken Thomas wrote:<br clear="none">> Greetings all,<br clear="none">> I'm using Heat from Icehouse and I'm hitting a problem that I'm hoping y'all can shed some light on.<br clear="none">> I have no problems doing stack-create. I can watch the MySQL commands go by and see it happily update the stack table so that it eventually shows up as CREATE_COMPLETE. When I delete the stack, everything seems to be working fine. I see the MySQL update that sets the stack to DELETE IN PROGRESS as well as the updates that sets my single resource to DELETE_COMPLETE... but I never see the final update to the stack table to set it to DELETE COMPLETE.<br clear="none"><br clear="none">I don't think you're the first person to report this, but afaik we've <br clear="none">never been able to track it down. There are a bunch of things that Heat <br clear="none">does after deleting the resources, like cleaning up trusts in Keystone, <br clear="none">and perhaps one of those is failing. However, *in theory* there's no way <br clear="none">for the stack to be left in the "DELETE_IN_PROGRESS" state without an <br clear="none">exception being raised that would leave a prominent entry in the logs.<br clear="none"><br clear="none">> One very odd thing that I found in the MySQL logs (snippet below), is a query that includes that stack name but with an extra '*' character append to it. My stack is named 'xyzzy8' but notice the 'xzyzzy8*' in the constraint. That's not going to return anything and I'm wondering if that is what's preventing the final stack DELETE_COMPLTE update from happening?<br clear="none"><br clear="none">The stack name that ends with '*' is the "backup stack", which is used <br clear="none">to hold out-of-date resources during a stack update. When the stack is <br clear="none">deleted, Heat will check for the existence of the backup stack and make <br clear="none">sure any resources left behind in there (from a failed update) are <br clear="none">deleted too. If it doesn't exist, that shouldn't be a problem.<br clear="none"><br clear="none">> There are no errors in the any of the heat logs.<br clear="none">><br clear="none">> Any guidance would be greatly appreciated!<br clear="none"><br clear="none">All I can suggest is to (a) make sure you have the very latest version <br clear="none">of Icehouse and (b) make sure your logging is turned up to 11, then try <br clear="none">again.<br clear="none"><br clear="none">Since you seem very proficient at debugging, if that doesn't work you <br clear="none">could also try adding extra logging into the method <br clear="none">heat.engine.parser.Stack.delete to try to figure out how far it is getting.<br clear="none"><br clear="none">Feel free to raise a bug in Launchpad and attach any logs that might <br clear="none">help us figure out what is happening.<br clear="none"><br clear="none">cheers,<br clear="none">Zane.<div class="yiv6805405000qtdSeparateBR"><br clear="none"><br clear="none"></div><div class="yiv6805405000yqt9561687983" id="yiv6805405000yqtfd89301"><br clear="none"><br clear="none">><br clear="none">> Ken<br clear="none">> SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.updated_at AS stack_updated_atFROM stackWHERE stack.deleted_at IS NULL AND (stack.tenant = 'c6c488223aae4e97bf56dda8cef36b3b' OR stack.stack_user_project_id = 'c6c488223aae4e97bf56dda8cef36b3b') AND stack.name = 'xyzzy8*' AND stack.owner_id = '9a3e56d7-0c10-4c1c-8c54-0e5580cee121'<br clear="none"><br clear="none"></div><br clear="none"><br clear="none"></div> </div> </div></div> </div></div></div></div></body></html>