Open Stack

Found it! The user_creds.password column in the database is 255 characters. My value is long enough (800+) that it fills it up and gets truncated.  I think that garbling of the data leads to the padding exception.
If I modify the database so that the column is longer, then the delete goes through just fine.
Ken
      From: Ken Thomas <krt at yahoo-inc.com>
 To: Zane Bitter <zbitter at redhat.com>; "openstack at lists.openstack.org" <openstack at lists.openstack.org> 
 Sent: Friday, October 24, 2014 2:43 PM
 Subject: Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE
   
I've got some additional info for y'all on this.  The problem is in heat.engine.parser.Stack.delete with this statement:
user_creds = db_api.user_creds_get(self.user_creds_id)
In my case, that call is throwing an exception that's not getting caught or logged.  The exact exception is:
ERROR Incorrect paddingTraceback (most recent call last):  File "/usr/lib/python2.6/site-packages/heat/engine/parser.py", line 670, in delete    user_creds = db_api.user_creds_get(self.user_creds_id)  File "/usr/lib/python2.6/site-packages/heat/db/api.py", line 170, in user_creds_get    return IMPL.user_creds_get(context_id)  File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 468, in user_creds_get    result['password'] = _decrypt(result['password'], db_result.decrypt_method)  File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 174, in _decrypt    value = decryptor(enc_value)  File "/usr/lib/python2.6/site-packages/heat/common/crypt.py", line 48, in oslo_decrypt_v1    auth_info, b64decode=True)  File "/usr/lib/python2.6/site-packages/heat/openstack/common/crypto/utils.py", line 158, in decrypt    msg = base64.b64decode(msg)  File "/usr/lib64/python2.6/base64.py", line 76, in b64decode    raise TypeError(msg)TypeError: Incorrect padding
Does this ring any bells? I checked the user_creds table and it looks okay as far as I can tell. Do I have a mismatch between how the value is encoded vs how it's being decoded?
Ken 

     From: Zane Bitter <zbitter at redhat.com>
 To: Ken Thomas <krt at yahoo-inc.com>; "openstack at lists.openstack.org" <openstack at lists.openstack.org> 
 Sent: Wednesday, October 22, 2014 6:10 AM
 Subject: Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE
   
Moving this to the openstack@ list...

On 21/10/14 18:56, Ken Thomas wrote:
> Greetings all,
> I'm using Heat from Icehouse and I'm hitting a problem that I'm hoping y'all can shed some light on.
> I have no problems doing stack-create. I can watch the MySQL commands go by and see it happily update the stack table so that it eventually shows up as CREATE_COMPLETE. When I delete the stack, everything seems to be working fine.  I see the MySQL update that sets the stack to DELETE IN PROGRESS as well as the updates that sets my single resource to DELETE_COMPLETE... but I never see the final update to the stack table to set it to DELETE COMPLETE.

I don't think you're the first person to report this, but afaik we've 
never been able to track it down. There are a bunch of things that Heat 
does after deleting the resources, like cleaning up trusts in Keystone, 
and perhaps one of those is failing. However, *in theory* there's no way 
for the stack to be left in the "DELETE_IN_PROGRESS" state without an 
exception being raised that would leave a prominent entry in the logs.

> One very odd thing that I found in the MySQL logs (snippet below), is a query that includes that stack name but with an extra '*' character append to it. My stack is named 'xyzzy8' but notice the 'xzyzzy8*' in the constraint. That's not going to return anything and I'm wondering if that is what's preventing the final stack DELETE_COMPLTE update from happening?

The stack name that ends with '*' is the "backup stack", which is used 
to hold out-of-date resources during a stack update. When the stack is 
deleted, Heat will check for the existence of the backup stack and make 
sure any resources left behind in there (from a failed update) are 
deleted too. If it doesn't exist, that shouldn't be a problem.

> There are no errors in the any of the heat logs.
>
> Any guidance would be greatly appreciated!

All I can suggest is to (a) make sure you have the very latest version 
of Icehouse and (b) make sure your logging is turned up to 11, then try 
again.

Since you seem very proficient at debugging, if that doesn't work you 
could also try adding extra logging into the method 
heat.engine.parser.Stack.delete to try to figure out how far it is getting.

Feel free to raise a bug in Launchpad and attach any logs that might 
help us figure out what is happening.

cheers,
Zane.



>
> Ken
> SELECT stack.status_reason AS stack_status_reason, stack.created_at AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS stack_disable_rollback, stack.stack_user_project_id AS stack_stack_user_project_id, stack.updated_at AS stack_updated_atFROM stackWHERE stack.deleted_at IS NULL AND (stack.tenant = 'c6c488223aae4e97bf56dda8cef36b3b' OR stack.stack_user_project_id = 'c6c488223aae4e97bf56dda8cef36b3b') AND stack.name = 'xyzzy8*' AND stack.owner_id = '9a3e56d7-0c10-4c1c-8c54-0e5580cee121'



   

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20141024/585b0449/attachment.html>