Open Stack

Hello all,

I recently found myself with a requirement for Sahara and, for the
most part, got the initial setup done without incident. I've created
node and cluster templates, installed images, and gotten to the point
of creating clusters.

When clusters are created, the structure is set up, but when it
actually goes to provision things, it fails creating a trust.

2016-09-01 11:20:13.775 17080 DEBUG
keystoneclient.auth.identity.v3.base
[req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
-] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
Making authentication request to
https://internal-osapi.rwdc1.cirrusseven.net:5000/v3/auth/tokens
get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py:188
2016-09-01 11:20:13.814 17080 DEBUG keystoneclient.session
[req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
-] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
Request returned failure status:
401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:419
2016-09-01 11:20:13.815 17080 ERROR sahara.service.trusts
[req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
-] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
Unable to create trust (reason: The request you have made requires
authentication. (HTTP 401) (Request-ID:
req-6099c99a-eb75-4dd3-b335-61f47b16d265))
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops
[req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
-] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
Error during operating on cluster (reason: Failed to create trust
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Traceback (most
recent call last):
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 192, in
wrapper
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     f(cluster_id,
*args, **kwds)
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 273, in
_provision_cluster
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     ctx, cluster,
plugin = _prepare_provisioning(cluster_id)
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 254, in
_prepare_provisioning
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
_setup_trust_for_cluster(cluster)
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 180, in
_setup_trust_for_cluster
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
trusts.create_trust_for_cluster(cluster)
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 97,
in create_trust_for_cluster
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
allow_redelegation=True)
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 75,
in create_trust
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     raise
ex.CreationFailed(_('Failed to create trust'))
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] CreationFailed:
Failed to create trust
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Error ID:
ec8e5abf-cee8-429b-8258-3a7b98d1e966
2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
2016-09-01 11:20:14.097 17080 DEBUG
keystoneclient.auth.identity.v3.base
[req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
-] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
Making authentication request to
https://internal-osapi.rwdc1.cirrusseven.net:5000/v3/auth/tokens
get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py:188
2016-09-01 11:20:14.129 17080 DEBUG keystoneclient.session
[req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
-] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
Request returned failure status:
401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:419
2016-09-01 11:20:14.130 17080 ERROR sahara.service.trusts
[req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
-] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
Unable to create trust (reason: The request you have made requires
authentication. (HTTP 401) (Request-ID:
req-ca6ffb9c-ab1b-4c03-af9a-e2570d2fc791))
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops
[req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
-] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
Error during rollback of cluster (reason: Failed to create trust
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Traceback (most
recent call last):
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 210, in
wrapper
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     if
_rollback_cluster(cluster, ex):
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 238, in
_rollback_cluster
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
_setup_trust_for_cluster(cluster)
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 180, in
_setup_trust_for_cluster
2016-09-01 11:20:14" .228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
trusts.create_trust_for_cluster(cluster)
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 97,
in create_trust_for_cluster
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
allow_redelegation=True)
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]   File
"/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 75,
in create_trust
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]     raise
ex.CreationFailed(_('Failed to create trust'))
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] CreationFailed:
Failed to create trust
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Error ID:
e811d08e-e1a6-450f-aaa7-bb114fc1d2de
2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance:
none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
2016-09-01 11:20:14.774 17081 DEBUG sahara.utils.api [-]
Rest.route.decorator.handler, kwargs={'tenant_id':
u'0d81fa86513d4e3d9ed88cdc8c0b7d73', 'cluster_id':
u'e10e2581-e9e4-45c7-a267-70359f19b9db'} handler
/usr/lib/python2.7/site-packages/sahara/utils/api.py:71
2016-09-01 11:20:14.829 17080 INFO sahara.utils.cluster
[req-41d4dbfd-9fec-468c-8bfa-60b225c16fea
88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - -
-] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db]
Cluster status has been changed. New status=Error

Any attempt to delete the cluster results in a similar error and
leaves the cluster in a "Deleting" state forever.

I assume what it's trying to do is create a trust between the
requesting user and the Sahara service user (trustor and trustee
respectively). I have tried doing this manually with the Openstack
client and it was created just fine. Keystone logs don't shed much
light on this either. It looks like a normal auth request followed by
the old "requires authentication" error.

I am using Keystone V3 for everything, and have set
"use_identity_api_v3 = True" and "auth_version = 3" as well as all the
usual domain and project settings under keystone_authtoken. I'm a bit
at a loss where to look next.

Thanks in advance for any help you all can provide.

Cheers,
Erik