<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div id="yui_3_16_0_1_1414418113898_7075" dir="ltr"><span id="yui_3_16_0_1_1414418113898_7080">I opened <a href="https://bugs.launchpad.net/heat/+bug/1386213" id="yui_3_16_0_1_1414418113898_7085">https://bugs.launchpad.net/heat/+bug/1386213</a> to track this issue.</span></div><div id="yui_3_16_0_1_1414418113898_7075" dir="ltr"><span><br></span></div><div id="yui_3_16_0_1_1414418113898_7075" dir="ltr">I understand that our situation may be a bit unusual and that changing the schema may not be the optimum solution as it affects everyone. (As I mentioned in the bug, we're doing an internal cloud and we're required to use an internal user validation system that returns cookies; that's why our 'passwords' are much bigger than y'all probably expected.)</div><div id="yui_3_16_0_1_1414418113898_7075" dir="ltr"><br></div><div id="yui_3_16_0_1_1414418113898_7075" dir="ltr">Any suggestions for our specific situation would be greatly appreciated. I'm thinking our best bet at this point is to have a local patch to alter the user_creds.password column.</div><div id="yui_3_16_0_1_1414418113898_7075" dir="ltr"><br></div><div id="yui_3_16_0_1_1414418113898_7075" dir="ltr">Ken</div><br> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id="yui_3_16_0_1_1414418113898_7068"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id="yui_3_16_0_1_1414418113898_7067"> <div dir="ltr" id="yui_3_16_0_1_1414418113898_7066"> <hr size="1" id="yui_3_16_0_1_1414418113898_7081"> <font size="2" face="Arial" id="yui_3_16_0_1_1414418113898_7065"> <b id="yui_3_16_0_1_1414418113898_7139"><span style="font-weight:bold;" id="yui_3_16_0_1_1414418113898_7138">From:</span></b> Ken Thomas <krt@yahoo-inc.com><br> <b id="yui_3_16_0_1_1414418113898_7137"><span style="font-weight: bold;" id="yui_3_16_0_1_1414418113898_7136">To:</span></b> Zane Bitter <zbitter@redhat.com>; "openstack@lists.openstack.org" <openstack@lists.openstack.org> <br> <b id="yui_3_16_0_1_1414418113898_7135"><span style="font-weight: bold;" id="yui_3_16_0_1_1414418113898_7134">Sent:</span></b> Friday, October 24, 2014 3:33 PM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Openstack] [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE<br> </font> </div> <div class="y_msg_container" id="yui_3_16_0_1_1414418113898_7141"><br><div id="yiv8592356032"><div id="yui_3_16_0_1_1414418113898_7143"><div style="color:#000;background-color:#fff;font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;" id="yui_3_16_0_1_1414418113898_7142"><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_26246"><span id="yiv8592356032yui_3_16_0_1_1414101400091_26253">Found it! The user_creds.password column in the database is 255 characters. My value is long enough (800+) that it fills it up and gets truncated. I think that garbling of the data leads to the padding exception.</span></div><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_26246"><span><br clear="none"></span></div><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_26246"><span id="yiv8592356032yui_3_16_0_1_1414101400091_26336">If I modify the database so that the column is longer, then the delete goes through just fine.</span></div><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_26246"><span><br clear="none"></span></div><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_26246"><span>Ken</span></div><br clear="none"> <div class="qtdSeparateBR"><br><br></div><div class="yiv8592356032yqt0611504972" id="yiv8592356032yqt55573"><div id="yiv8592356032yui_3_16_0_1_1414101400091_26250" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div id="yiv8592356032yui_3_16_0_1_1414101400091_26249" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_26248"> <hr id="yiv8592356032yui_3_16_0_1_1414101400091_26247" size="1"> <font id="yiv8592356032yui_3_16_0_1_1414101400091_26337" size="2" face="Arial"> <b><span style="font-weight:bold;">From:</span></b> Ken Thomas <krt@yahoo-inc.com><br clear="none"> <b><span style="font-weight:bold;">To:</span></b> Zane Bitter <zbitter@redhat.com>; "openstack@lists.openstack.org" <openstack@lists.openstack.org> <br clear="none"> <b><span style="font-weight:bold;">Sent:</span></b> Friday, October 24, 2014 2:43 PM<br clear="none"> <b><span style="font-weight:bold;">Subject:</span></b> Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE<br clear="none"> </font> </div> <div class="yiv8592356032y_msg_container" id="yiv8592356032yui_3_16_0_1_1414101400091_26296"><br clear="none"><div id="yiv8592356032"><div id="yiv8592356032yui_3_16_0_1_1414101400091_26295"><div id="yiv8592356032yui_3_16_0_1_1414101400091_26294" style="color:#000;background-color:#fff;font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"><div id="yiv8592356032yui_3_16_0_1_1414101400091_23117">I've got some additional info for y'all on this. The problem is in heat.engine.parser.Stack.delete with this statement:</div><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_23117"><br clear="none"></div><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_23117">
</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23478" style="">user_creds = db_api.user_creds_get(self.user_creds_id)</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23528" style=""><br clear="none"></div><div class="yiv8592356032" dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_23526" style="">In my case, that call is throwing an exception that's not getting caught or logged. The exact exception is:</div><div class="yiv8592356032" dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_23527" style=""><br clear="none"></div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23477" style="">ERROR Incorrect padding</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23464" style="">Traceback (most recent call last):</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23465" style=""> File "/usr/lib/python2.6/site-packages/heat/engine/parser.py", line 670, in delete</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23466" style=""> user_creds = db_api.user_creds_get(self.user_creds_id)</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23467" style=""> File "/usr/lib/python2.6/site-packages/heat/db/api.py", line 170, in user_creds_get</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23468" style=""> return IMPL.user_creds_get(context_id)</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23469" style=""> File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 468, in user_creds_get</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23470" style=""> result['password'] = _decrypt(result['password'], db_result.decrypt_method)</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23471" style=""> File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 174, in _decrypt</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23472" style=""> value = decryptor(enc_value)</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23473" style=""> File "/usr/lib/python2.6/site-packages/heat/common/crypt.py", line 48, in oslo_decrypt_v1</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23474" style=""> auth_info, b64decode=True)</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23530" style=""> File "/usr/lib/python2.6/site-packages/heat/openstack/common/crypto/utils.py", line 158, in decrypt</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23475" style=""> msg = base64.b64decode(msg)</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23476" style=""> File "/usr/lib64/python2.6/base64.py", line 76, in b64decode</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23536" style=""> raise TypeError(msg)</div><div class="yiv8592356032" dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_23397" style="">
</div><div class="yiv8592356032" id="yiv8592356032yui_3_16_0_1_1414101400091_23531" style="">TypeError: Incorrect padding</div><div id="yiv8592356032yui_3_16_0_1_1414101400091_23292"><br clear="none"></div><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_23292">Does this ring any bells? I checked the user_creds table and it looks okay as far as I can tell. Do I have a mismatch between how the value is encoded vs how it's being decoded?</div><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_23292"><br clear="none"></div><div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_23292">Ken</div> <div class="yiv8592356032qtdSeparateBR"><br clear="none"><br clear="none"></div><div class="yiv8592356032yqt8868523620" id="yiv8592356032yqt25547"><div id="yiv8592356032yui_3_16_0_1_1414101400091_23120" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div id="yiv8592356032yui_3_16_0_1_1414101400091_23119" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div dir="ltr" id="yiv8592356032yui_3_16_0_1_1414101400091_23118"> <hr id="yiv8592356032yui_3_16_0_1_1414101400091_23293" size="1"> <font id="yiv8592356032yui_3_16_0_1_1414101400091_23121" size="2" face="Arial"> <b id="yiv8592356032yui_3_16_0_1_1414101400091_23396"><span id="yiv8592356032yui_3_16_0_1_1414101400091_23395" style="font-weight:bold;">From:</span></b> Zane Bitter <zbitter@redhat.com><br clear="none"> <b><span style="font-weight:bold;">To:</span></b> Ken Thomas <krt@yahoo-inc.com>; "openstack@lists.openstack.org" <openstack@lists.openstack.org> <br clear="none"> <b><span style="font-weight:bold;">Sent:</span></b> Wednesday, October 22, 2014 6:10 AM<br clear="none"> <b><span style="font-weight:bold;">Subject:</span></b> Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE<br clear="none"> </font> </div> <div class="yiv8592356032y_msg_container" id="yiv8592356032yui_3_16_0_1_1414101400091_23236"><br clear="none">Moving this to the openstack@ list...<br clear="none"><br clear="none">On 21/10/14 18:56, Ken Thomas wrote:<br clear="none">> Greetings all,<br clear="none">> I'm using Heat from Icehouse and I'm hitting a problem that I'm hoping y'all can shed some light on.<br clear="none">> I have no problems doing stack-create. I can watch the MySQL commands go by and see it happily update the stack table so that it eventually shows up as CREATE_COMPLETE. When I delete the stack, everything seems to be working fine. I see the MySQL update that sets the stack to DELETE IN PROGRESS as well as the updates that sets my single resource to DELETE_COMPLETE... but I never see the final update to the stack table to set it to DELETE COMPLETE.<br clear="none"><br clear="none">I don't think you're the first person to report this, but afaik we've <br clear="none">never been able to track it down. There are a bunch of things that Heat <br clear="none">does after deleting the resources, like cleaning up trusts in Keystone, <br clear="none">and perhaps one of those is failing. However, *in theory* there's no way <br clear="none">for the stack to be left in the "DELETE_IN_PROGRESS" state without an <br clear="none">exception being raised that would leave a prominent entry in the logs.<br clear="none"><br clear="none">> One very odd thing that I found in the MySQL logs (snippet below), is a query that includes that stack name but with an extra '*' character append to it. My stack is named 'xyzzy8' but notice the 'xzyzzy8*' in the constraint. That's not going to return anything and I'm wondering if that is what's preventing the final stack DELETE_COMPLTE update from happening?<br clear="none"><br clear="none">The stack name that ends with '*' is the "backup stack", which is used <br clear="none">to hold out-of-date resources during a stack update. When the stack is <br clear="none">deleted, Heat will check for the existence of the backup stack and make <br clear="none">sure any resources left behind in there (from a failed update) are <br clear="none">deleted too. If it doesn't exist, that shouldn't be a problem.<br clear="none"><br clear="none">> There are no errors in the any of the heat logs.<br clear="none">><br clear="none">> Any guidance would be greatly appreciated!<br clear="none"><br clear="none">All I can suggest is to (a) make sure you have the very latest version <br clear="none">of Icehouse and (b) make sure your logging is turned up to 11, then try <br clear="none">again.<br clear="none"><br clear="none">Since you seem very proficient at debugging, if that doesn't work you <br clear="none">could also try adding extra logging into the method <br clear="none">heat.engine.parser.Stack.delete to try to figure out how far it is getting.<br clear="none"><br clear="none">Feel free to raise a bug in Launchpad and attach any logs that might <br clear="none">help us figure out what is happening.<br clear="none"><br clear="none">cheers,<br clear="none">Zane.<div class="yiv8592356032qtdSeparateBR"><br clear="none"><br clear="none"></div><div class="yiv8592356032yqt0646956842" id="yiv8592356032yqtfd79377"><br clear="none"><br clear="none">><br clear="none">> Ken<br clear="none">> SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.updated_at AS stack_updated_atFROM stackWHERE stack.deleted_at IS NULL AND (stack.tenant = 'c6c488223aae4e97bf56dda8cef36b3b' OR stack.stack_user_project_id = 'c6c488223aae4e97bf56dda8cef36b3b') AND stack.name = 'xyzzy8*' AND stack.owner_id = '9a3e56d7-0c10-4c1c-8c54-0e5580cee121'<br clear="none"><br clear="none"></div><br clear="none"><br clear="none"></div> </div> </div></div> </div></div></div><br clear="none"><br clear="none"></div> </div> </div></div> </div></div></div><br><div class="yqt0611504972" id="yqt09691">_______________________________________________<br clear="none">Mailing list: <a shape="rect" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br clear="none">Post to : <a shape="rect" ymailto="mailto:openstack@lists.openstack.org" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br clear="none">Unsubscribe : <a shape="rect" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br clear="none"></div><br><br></div> </div> </div> </div></body></html>