<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div id="yui_3_16_0_1_1414101400091_33866" dir="ltr"><span id="yui_3_16_0_1_1414101400091_33874">Thanks for the info. I'll take a look into using trusts.</span></div><div id="yui_3_16_0_1_1414101400091_33866" dir="ltr"><span><br></span></div><div id="yui_3_16_0_1_1414101400091_33866" dir="ltr"><span>If there's anything I can do to help out on this, please let me know.</span></div><div id="yui_3_16_0_1_1414101400091_33866" dir="ltr"><span><br></span></div><div id="yui_3_16_0_1_1414101400091_33866" dir="ltr"><span>Ken</span></div><br> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id="yui_3_16_0_1_1414101400091_33861"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id="yui_3_16_0_1_1414101400091_33860"> <div dir="ltr" id="yui_3_16_0_1_1414101400091_33859"> <hr size="1"> <font size="2" face="Arial" id="yui_3_16_0_1_1414101400091_33872"> <b><span style="font-weight:bold;">From:</span></b> Zane Bitter <zbitter@redhat.com><br> <b><span style="font-weight: bold;">To:</span></b> Ken Thomas <krt@yahoo-inc.com>; "openstack@lists.openstack.org" <openstack@lists.openstack.org> <br> <b><span style="font-weight: bold;">Sent:</span></b> Monday, October 27, 2014 7:49 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Openstack] [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE<br> </font> </div> <div class="y_msg_container" id="yui_3_16_0_1_1414101400091_33865"><br>On 27/10/14 10:09, Ken Thomas wrote:<br clear="none">> I opened <a shape="rect" href="https://bugs.launchpad.net/heat/+bug/1386213" target="_blank" id="yui_3_16_0_1_1414101400091_33873">https://bugs.launchpad.net/heat/+bug/1386213 </a>to track this issue.<br clear="none">><br clear="none">> I understand that our situation may be a bit unusual and that changing<br clear="none">> the schema may not be the optimum solution as it affects everyone. (As I<br clear="none">> mentioned in the bug, we're doing an internal cloud and we're required<br clear="none">> to use an internal user validation system that returns cookies; that's<br clear="none">> why our 'passwords' are much bigger than y'all probably expected.)<br clear="none"><br clear="none">At a minimum, we need to handle this error better, so there is <br clear="none">definitely something there for us to fix. Many thanks for tracking this <br clear="none">down, it is much appreciated!<br clear="none"><br clear="none">> Any suggestions for our specific situation would be greatly appreciated.<br clear="none">> I'm thinking our best bet at this point is to have a local patch to<br clear="none">> alter the user_creds.password column.<br clear="none"><br clear="none">Your very _best_ bet is to configure Heat to use Keystone trusts and <br clear="none">therefore not need to store the user's password at all. Steve Hardy has <br clear="none">an excellent blog post on how to do it:<br clear="none"><br clear="none"><a shape="rect" href="http://hardysteven.blogspot.co.uk/2014/04/heat-auth-model-updates-part-1-trusts.html" target="_blank">http://hardysteven.blogspot.co.uk/2014/04/heat-auth-model-updates-part-1-trusts.html</a><br clear="none"><br clear="none">If that's not possible for whatever reason then increasing the column <br clear="none">size in a local patch is probably a decent solution, although it has a <br clear="none">small potential to cause problems with database migrations in the future.<br clear="none"><br clear="none">cheers,<br clear="none">Zane.<br clear="none"><br clear="none">> Ken<br clear="none">><br clear="none">> ------------------------------------------------------------------------<br clear="none">> *From:* Ken Thomas <<a shape="rect" ymailto="mailto:krt@yahoo-inc.com" href="mailto:krt@yahoo-inc.com">krt@yahoo-inc.com</a>><br clear="none">> *To:* Zane Bitter <<a shape="rect" ymailto="mailto:zbitter@redhat.com" href="mailto:zbitter@redhat.com">zbitter@redhat.com</a>>; "<a shape="rect" ymailto="mailto:openstack@lists.openstack.org" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>"<br clear="none">> <<a shape="rect" ymailto="mailto:openstack@lists.openstack.org" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>><br clear="none">> *Sent:* Friday, October 24, 2014 3:33 PM<br clear="none">> *Subject:* Re: [Openstack] [openstack-dev] [Heat] Stack stuck in<br clear="none">> DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE<br clear="none">><br clear="none">> Found it! The user_creds.password column in the database is 255<br clear="none">> characters. My value is long enough (800+) that it fills it up and gets<br clear="none">> truncated. I think that garbling of the data leads to the padding<br clear="none">> exception.<br clear="none">><br clear="none">> If I modify the database so that the column is longer, then the delete<br clear="none">> goes through just fine.<br clear="none">><br clear="none">> Ken<br clear="none">><br clear="none">><br clear="none">><br clear="none">> ------------------------------------------------------------------------<br clear="none">> *From:* Ken Thomas <<a shape="rect" ymailto="mailto:krt@yahoo-inc.com" href="mailto:krt@yahoo-inc.com">krt@yahoo-inc.com</a>><br clear="none">> *To:* Zane Bitter <<a shape="rect" ymailto="mailto:zbitter@redhat.com" href="mailto:zbitter@redhat.com">zbitter@redhat.com</a>>; "<a shape="rect" ymailto="mailto:openstack@lists.openstack.org" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>"<br clear="none">> <<a shape="rect" ymailto="mailto:openstack@lists.openstack.org" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>><br clear="none">> *Sent:* Friday, October 24, 2014 2:43 PM<br clear="none">> *Subject:* Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS<br clear="none">> even though all resources are DELETE_COMPLETE<br clear="none">><br clear="none">> I've got some additional info for y'all on this. The problem is in<br clear="none">> heat.engine.parser.Stack.delete with this statement:<br clear="none">><br clear="none">> user_creds = db_api.user_creds_get(self.user_creds_id)<br clear="none">><br clear="none">> In my case, that call is throwing an exception that's not getting caught<br clear="none">> or logged. The exact exception is:<br clear="none">><br clear="none">> ERROR Incorrect padding<br clear="none">> Traceback (most recent call last):<br clear="none">> File "/usr/lib/python2.6/site-packages/heat/engine/parser.py", line<br clear="none">> 670, in delete<br clear="none">> user_creds = db_api.user_creds_get(self.user_creds_id)<br clear="none">> File "/usr/lib/python2.6/site-packages/heat/db/api.py", line 170, in<br clear="none">> user_creds_get<br clear="none">> return IMPL.user_creds_get(context_id)<br clear="none">> File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py",<br clear="none">> line 468, in user_creds_get<br clear="none">> result['password'] = _decrypt(result['password'],<br clear="none">> db_result.decrypt_method)<br clear="none">> File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py",<br clear="none">> line 174, in _decrypt<br clear="none">> value = decryptor(enc_value)<br clear="none">> File "/usr/lib/python2.6/site-packages/heat/common/crypt.py", line<br clear="none">> 48, in oslo_decrypt_v1<br clear="none">> auth_info, b64decode=True)<br clear="none">> File<br clear="none">> "/usr/lib/python2.6/site-packages/heat/openstack/common/crypto/utils.py", line<br clear="none">> 158, in decrypt<br clear="none">> msg = base64.b64decode(msg)<br clear="none">> File "/usr/lib64/python2.6/base64.py", line 76, in b64decode<br clear="none">> raise TypeError(msg)<br clear="none">> TypeError: Incorrect padding<br clear="none">><br clear="none">> Does this ring any bells? I checked the user_creds table and it looks<br clear="none">> okay as far as I can tell. Do I have a mismatch between how the value is<br clear="none">> encoded vs how it's being decoded?<br clear="none">><br clear="none">> Ken<br clear="none">><br clear="none">><br clear="none">> ------------------------------------------------------------------------<br clear="none">> *From:* Zane Bitter <<a shape="rect" ymailto="mailto:zbitter@redhat.com" href="mailto:zbitter@redhat.com">zbitter@redhat.com</a>><br clear="none">> *To:* Ken Thomas <<a shape="rect" ymailto="mailto:krt@yahoo-inc.com" href="mailto:krt@yahoo-inc.com">krt@yahoo-inc.com</a>>; "<a shape="rect" ymailto="mailto:openstack@lists.openstack.org" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>"<br clear="none">> <<a shape="rect" ymailto="mailto:openstack@lists.openstack.org" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>><br clear="none">> *Sent:* Wednesday, October 22, 2014 6:10 AM<br clear="none">> *Subject:* Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS<br clear="none">> even though all resources are DELETE_COMPLETE<br clear="none">><br clear="none">> Moving this to the openstack@ list...<br clear="none">><br clear="none">> On 21/10/14 18:56, Ken Thomas wrote:<br clear="none">> > Greetings all,<br clear="none">> > I'm using Heat from Icehouse and I'm hitting a problem that I'm<br clear="none">> hoping y'all can shed some light on.<br clear="none">> > I have no problems doing stack-create. I can watch the MySQL commands<br clear="none">> go by and see it happily update the stack table so that it eventually<br clear="none">> shows up as CREATE_COMPLETE. When I delete the stack, everything seems<br clear="none">> to be working fine. I see the MySQL update that sets the stack to<br clear="none">> DELETE IN PROGRESS as well as the updates that sets my single resource<br clear="none">> to DELETE_COMPLETE... but I never see the final update to the stack<br clear="none">> table to set it to DELETE COMPLETE.<br clear="none">><br clear="none">> I don't think you're the first person to report this, but afaik we've<br clear="none">> never been able to track it down. There are a bunch of things that Heat<br clear="none">> does after deleting the resources, like cleaning up trusts in Keystone,<br clear="none">> and perhaps one of those is failing. However, *in theory* there's no way<br clear="none">> for the stack to be left in the "DELETE_IN_PROGRESS" state without an<br clear="none">> exception being raised that would leave a prominent entry in the logs.<br clear="none">><br clear="none">> > One very odd thing that I found in the MySQL logs (snippet below), is<br clear="none">> a query that includes that stack name but with an extra '*' character<br clear="none">> append to it. My stack is named 'xyzzy8' but notice the 'xzyzzy8*' in<br clear="none">> the constraint. That's not going to return anything and I'm wondering if<br clear="none">> that is what's preventing the final stack DELETE_COMPLTE update from<br clear="none">> happening?<br clear="none">><br clear="none">> The stack name that ends with '*' is the "backup stack", which is used<br clear="none">> to hold out-of-date resources during a stack update. When the stack is<br clear="none">> deleted, Heat will check for the existence of the backup stack and make<br clear="none">> sure any resources left behind in there (from a failed update) are<br clear="none">> deleted too. If it doesn't exist, that shouldn't be a problem.<br clear="none">><br clear="none">> > There are no errors in the any of the heat logs.<br clear="none">> ><br clear="none">> > Any guidance would be greatly appreciated!<br clear="none">><br clear="none">> All I can suggest is to (a) make sure you have the very latest version<br clear="none">> of Icehouse and (b) make sure your logging is turned up to 11, then try<br clear="none">> again.<br clear="none">><br clear="none">> Since you seem very proficient at debugging, if that doesn't work you<br clear="none">> could also try adding extra logging into the method<br clear="none">> heat.engine.parser.Stack.delete to try to figure out how far it is getting.<br clear="none">><br clear="none">> Feel free to raise a bug in Launchpad and attach any logs that might<br clear="none">> help us figure out what is happening.<br clear="none">><br clear="none">> cheers,<br clear="none">> Zane.<br clear="none">><br clear="none">><br clear="none">><br clear="none">><br clear="none">> ><br clear="none">> > Ken<br clear="none">> > SELECT stack.status_reason AS stack_status_reason, stack.created_at<br clear="none">> AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action<br clear="none">> AS stack_action, stack.status AS stack_status, stack.id AS stack_id,<br clear="none">> stack.name AS stack_name, stack.raw_template_id AS<br clear="none">> stack_raw_template_id, stack.username AS stack_username, stack.tenant AS<br clear="none">> stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id<br clear="none">> AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout<br clear="none">> AS stack_timeout, stack.disable_rollback AS stack_disable_rollback,<br clear="none">> stack.stack_user_project_id AS stack_stack_user_project_id,<br clear="none">> stack.updated_at AS stack_updated_atFROM stackWHERE stack.deleted_at IS<br clear="none">> NULL AND (stack.tenant = 'c6c488223aae4e97bf56dda8cef36b3b' OR<br clear="none">> stack.stack_user_project_id = 'c6c488223aae4e97bf56dda8cef36b3b') AND<br clear="none">> stack.name = 'xyzzy8*' AND stack.owner_id =<br clear="none">> '9a3e56d7-0c10-4c1c-8c54-0e5580cee121'<br clear="none">><br clear="none">><br clear="none">><br clear="none">><br clear="none">><br clear="none">><br clear="none">> _______________________________________________<br clear="none">> Mailing list: <a shape="rect" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br clear="none">> Post to : <a shape="rect" ymailto="mailto:openstack@lists.openstack.org" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br clear="none">> <mailto:<a shape="rect" ymailto="mailto:openstack@lists.openstack.org" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>><div class="qtdSeparateBR"><br><br></div><div class="yqt2688950118" id="yqtfd66362"><br clear="none">> Unsubscribe : <a shape="rect" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br clear="none">><br clear="none">><br clear="none"><br clear="none"></div><br><br></div> </div> </div> </div></body></html>