Open Stack

Hi everyone,

Please excuse the post bump. I thought this might have gotten buried
in US holiday weekend spam. Does anyone have any ideas on this?

On Fri, Sep 2, 2016 at 4:39 PM, Erik McCormick
<emccormick at cirrusseven.com> wrote:
> Hello all,
>
> I recently found myself with a requirement for Sahara and, for the
> most part, got the initial setup done without incident. I've created
> node and cluster templates, installed images, and gotten to the point
> of creating clusters.
>
> When clusters are created, the structure is set up, but when it
> actually goes to provision things, it fails creating a trust.
>
> 2016-09-01 11:20:13.775 17080 DEBUG
> keystoneclient.auth.identity.v3.base
> [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
> 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
> -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> Making authentication request to
> https://internal-osapi.rwdc1.cirrusseven.net:5000/v3/auth/tokens
> get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py:188
> 2016-09-01 11:20:13.814 17080 DEBUG keystoneclient.session
> [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
> 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
> -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> Request returned failure status:
> 401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:419
> 2016-09-01 11:20:13.815 17080 ERROR sahara.service.trusts
> [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
> 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
> -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> Unable to create trust (reason: The request you have made requires
> authentication. (HTTP 401) (Request-ID:
> req-6099c99a-eb75-4dd3-b335-61f47b16d265))
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops
> [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
> 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
> -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> Error during operating on cluster (reason: Failed to create trust
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Traceback (most
> recent call last):
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 192, in
> wrapper
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     f(cluster_id,
> *args, **kwds)
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 273, in
> _provision_cluster
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     ctx, cluster,
> plugin = _prepare_provisioning(cluster_id)
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 254, in
> _prepare_provisioning
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> _setup_trust_for_cluster(cluster)
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 180, in
> _setup_trust_for_cluster
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> trusts.create_trust_for_cluster(cluster)
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 97,
> in create_trust_for_cluster
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> allow_redelegation=True)
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 75,
> in create_trust
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     raise
> ex.CreationFailed(_('Failed to create trust'))
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] CreationFailed:
> Failed to create trust
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Error ID:
> ec8e5abf-cee8-429b-8258-3a7b98d1e966
> 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> 2016-09-01 11:20:14.097 17080 DEBUG
> keystoneclient.auth.identity.v3.base
> [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
> 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
> -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> Making authentication request to
> https://internal-osapi.rwdc1.cirrusseven.net:5000/v3/auth/tokens
> get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py:188
> 2016-09-01 11:20:14.129 17080 DEBUG keystoneclient.session
> [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
> 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
> -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> Request returned failure status:
> 401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:419
> 2016-09-01 11:20:14.130 17080 ERROR sahara.service.trusts
> [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
> 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
> -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> Unable to create trust (reason: The request you have made requires
> authentication. (HTTP 401) (Request-ID:
> req-ca6ffb9c-ab1b-4c03-af9a-e2570d2fc791))
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops
> [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
> 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
> -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> Error during rollback of cluster (reason: Failed to create trust
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Traceback (most
> recent call last):
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 210, in
> wrapper
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     if
> _rollback_cluster(cluster, ex):
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 238, in
> _rollback_cluster
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> _setup_trust_for_cluster(cluster)
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 180, in
> _setup_trust_for_cluster
> 2016-09-01 11:20:14" .228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> trusts.create_trust_for_cluster(cluster)
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 97,
> in create_trust_for_cluster
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> allow_redelegation=True)
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
> "/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 75,
> in create_trust
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     raise
> ex.CreationFailed(_('Failed to create trust'))
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] CreationFailed:
> Failed to create trust
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Error ID:
> e811d08e-e1a6-450f-aaa7-bb114fc1d2de
> 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
> none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> 2016-09-01 11:20:14.774 17081 DEBUG sahara.utils.api [-]
> Rest.route.decorator.handler, kwargs={'tenant_id':
> u'0d81fa86513d4e3d9ed88cdc8c0b7d73', 'cluster_id':
> u'e10e2581-e9e4-45c7-a267-70359f19b9db'} handler
> /usr/lib/python2.7/site-packages/sahara/utils/api.py:71
> 2016-09-01 11:20:14.829 17080 INFO sahara.utils.cluster
> [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
> 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
> -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
> Cluster status has been changed. New status=Error
>
> Any attempt to delete the cluster results in a similar error and
> leaves the cluster in a "Deleting" state forever.
>
> I assume what it's trying to do is create a trust between the
> requesting user and the Sahara service user (trustor and trustee
> respectively). I have tried doing this manually with the Openstack
> client and it was created just fine. Keystone logs don't shed much
> light on this either. It looks like a normal auth request followed by
> the old "requires authentication" error.
>
> I am using Keystone V3 for everything, and have set
> "use_identity_api_v3 = True" and "auth_version = 3" as well as all the
> usual domain and project settings under keystone_authtoken. I'm a bit
> at a loss where to look next.
>
> Thanks in advance for any help you all can provide.
>
> Cheers,
> Erik