Open Stack

Thanks for the info. I'll take a look into using trusts.
If there's anything I can do to help out on this, please let me know.
Ken
      From: Zane Bitter <zbitter at redhat.com>
 To: Ken Thomas <krt at yahoo-inc.com>; "openstack at lists.openstack.org" <openstack at lists.openstack.org> 
 Sent: Monday, October 27, 2014 7:49 AM
 Subject: Re: [Openstack] [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE
   
On 27/10/14 10:09, Ken Thomas wrote:
> I opened https://bugs.launchpad.net/heat/+bug/1386213 to track this issue.
>
> I understand that our situation may be a bit unusual and that changing
> the schema may not be the optimum solution as it affects everyone. (As I
> mentioned in the bug, we're doing an internal cloud and we're required
> to use an internal user validation system that returns cookies; that's
> why our 'passwords' are much bigger than y'all probably expected.)

At a minimum, we need to handle this error better, so there is 
definitely something there for us to fix. Many thanks for tracking this 
down, it is much appreciated!

> Any suggestions for our specific situation would be greatly appreciated.
> I'm thinking our best bet at this point is to have a local patch to
> alter the user_creds.password column.

Your very _best_ bet is to configure Heat to use Keystone trusts and 
therefore not need to store the user's password at all. Steve Hardy has 
an excellent blog post on how to do it:

http://hardysteven.blogspot.co.uk/2014/04/heat-auth-model-updates-part-1-trusts.html

If that's not possible for whatever reason then increasing the column 
size in a local patch is probably a decent solution, although it has a 
small potential to cause problems with database migrations in the future.

cheers,
Zane.

> Ken
>
> ------------------------------------------------------------------------
> *From:* Ken Thomas <krt at yahoo-inc.com>
> *To:* Zane Bitter <zbitter at redhat.com>; "openstack at lists.openstack.org"
> <openstack at lists.openstack.org>
> *Sent:* Friday, October 24, 2014 3:33 PM
> *Subject:* Re: [Openstack] [openstack-dev] [Heat] Stack stuck in
> DELETE_IN_PROGRESS even though all resources are DELETE_COMPLETE
>
> Found it! The user_creds.password column in the database is 255
> characters. My value is long enough (800+) that it fills it up and gets
> truncated.  I think that garbling of the data leads to the padding
> exception.
>
> If I modify the database so that the column is longer, then the delete
> goes through just fine.
>
> Ken
>
>
>
> ------------------------------------------------------------------------
> *From:* Ken Thomas <krt at yahoo-inc.com>
> *To:* Zane Bitter <zbitter at redhat.com>; "openstack at lists.openstack.org"
> <openstack at lists.openstack.org>
> *Sent:* Friday, October 24, 2014 2:43 PM
> *Subject:* Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS
> even though all resources are DELETE_COMPLETE
>
> I've got some additional info for y'all on this.  The problem is in
> heat.engine.parser.Stack.delete with this statement:
>
> user_creds = db_api.user_creds_get(self.user_creds_id)
>
> In my case, that call is throwing an exception that's not getting caught
> or logged.  The exact exception is:
>
> ERROR Incorrect padding
> Traceback (most recent call last):
>    File "/usr/lib/python2.6/site-packages/heat/engine/parser.py", line
> 670, in delete
>      user_creds = db_api.user_creds_get(self.user_creds_id)
>    File "/usr/lib/python2.6/site-packages/heat/db/api.py", line 170, in
> user_creds_get
>      return IMPL.user_creds_get(context_id)
>    File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py",
> line 468, in user_creds_get
>      result['password'] = _decrypt(result['password'],
> db_result.decrypt_method)
>    File "/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py",
> line 174, in _decrypt
>      value = decryptor(enc_value)
>    File "/usr/lib/python2.6/site-packages/heat/common/crypt.py", line
> 48, in oslo_decrypt_v1
>      auth_info, b64decode=True)
>    File
> "/usr/lib/python2.6/site-packages/heat/openstack/common/crypto/utils.py", line
> 158, in decrypt
>      msg = base64.b64decode(msg)
>    File "/usr/lib64/python2.6/base64.py", line 76, in b64decode
>      raise TypeError(msg)
> TypeError: Incorrect padding
>
> Does this ring any bells? I checked the user_creds table and it looks
> okay as far as I can tell. Do I have a mismatch between how the value is
> encoded vs how it's being decoded?
>
> Ken
>
>
> ------------------------------------------------------------------------
> *From:* Zane Bitter <zbitter at redhat.com>
> *To:* Ken Thomas <krt at yahoo-inc.com>; "openstack at lists.openstack.org"
> <openstack at lists.openstack.org>
> *Sent:* Wednesday, October 22, 2014 6:10 AM
> *Subject:* Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS
> even though all resources are DELETE_COMPLETE
>
> Moving this to the openstack@ list...
>
> On 21/10/14 18:56, Ken Thomas wrote:
>  > Greetings all,
>  > I'm using Heat from Icehouse and I'm hitting a problem that I'm
> hoping y'all can shed some light on.
>  > I have no problems doing stack-create. I can watch the MySQL commands
> go by and see it happily update the stack table so that it eventually
> shows up as CREATE_COMPLETE. When I delete the stack, everything seems
> to be working fine.  I see the MySQL update that sets the stack to
> DELETE IN PROGRESS as well as the updates that sets my single resource
> to DELETE_COMPLETE... but I never see the final update to the stack
> table to set it to DELETE COMPLETE.
>
> I don't think you're the first person to report this, but afaik we've
> never been able to track it down. There are a bunch of things that Heat
> does after deleting the resources, like cleaning up trusts in Keystone,
> and perhaps one of those is failing. However, *in theory* there's no way
> for the stack to be left in the "DELETE_IN_PROGRESS" state without an
> exception being raised that would leave a prominent entry in the logs.
>
>  > One very odd thing that I found in the MySQL logs (snippet below), is
> a query that includes that stack name but with an extra '*' character
> append to it. My stack is named 'xyzzy8' but notice the 'xzyzzy8*' in
> the constraint. That's not going to return anything and I'm wondering if
> that is what's preventing the final stack DELETE_COMPLTE update from
> happening?
>
> The stack name that ends with '*' is the "backup stack", which is used
> to hold out-of-date resources during a stack update. When the stack is
> deleted, Heat will check for the existence of the backup stack and make
> sure any resources left behind in there (from a failed update) are
> deleted too. If it doesn't exist, that shouldn't be a problem.
>
>  > There are no errors in the any of the heat logs.
>  >
>  > Any guidance would be greatly appreciated!
>
> All I can suggest is to (a) make sure you have the very latest version
> of Icehouse and (b) make sure your logging is turned up to 11, then try
> again.
>
> Since you seem very proficient at debugging, if that doesn't work you
> could also try adding extra logging into the method
> heat.engine.parser.Stack.delete to try to figure out how far it is getting.
>
> Feel free to raise a bug in Launchpad and attach any logs that might
> help us figure out what is happening.
>
> cheers,
> Zane.
>
>
>
>
>  >
>  > Ken
>  > SELECT stack.status_reason AS stack_status_reason, stack.created_at
> AS stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action
> AS stack_action, stack.status AS stack_status, stack.id AS stack_id,
> stack.name AS stack_name, stack.raw_template_id AS
> stack_raw_template_id, stack.username AS stack_username, stack.tenant AS
> stack_tenant, stack.parameters AS stack_parameters, stack.user_creds_id
> AS stack_user_creds_id, stack.owner_id AS stack_owner_id, stack.timeout
> AS stack_timeout, stack.disable_rollback AS stack_disable_rollback,
> stack.stack_user_project_id AS stack_stack_user_project_id,
> stack.updated_at AS stack_updated_atFROM stackWHERE stack.deleted_at IS
> NULL AND (stack.tenant = 'c6c488223aae4e97bf56dda8cef36b3b' OR
> stack.stack_user_project_id = 'c6c488223aae4e97bf56dda8cef36b3b') AND
> stack.name = 'xyzzy8*' AND stack.owner_id =
> '9a3e56d7-0c10-4c1c-8c54-0e5580cee121'
>
>
>
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to    : openstack at lists.openstack.org
> <mailto:openstack at lists.openstack.org>


> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>



  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20141027/a37259fe/attachment.html>