Issue with heat and magnum

feilong feilong at catalyst.net.nz
Wed Sep 2 07:41:07 UTC 2020


Hi Alexander,

Firstly, could you please help understand your Heat version and Magnum
version? Secondly, I don't really think it's related to Magnum. As long
as Heat API get the request from Magnum, Magnum conductor just query
Heat API to sync the status.


On 14/08/20 10:49 pm, Alexander Dibbo - UKRI STFC wrote:
>
> Hi,
>
>  
>
> I am having an issue with magnum creating clusters when I have
> multiple active heat-engine daemons running.
>
>  
>
> I get the following error in the heat engine logs:
>
> 2020-08-14 10:36:30.237 598383 INFO heat.engine.resource
> [req-a2c862eb-370c-4e91-a2c6-dca32c7872ce - - - - -] signal
> SoftwareDeployment "master_config_deployment" [67ba9ce2-aba5-4c15-a7ea
> -6b774659a0e2] Stack
> "kubernetes-test-26-3uzjqqob47fh-kube_masters-mhctjio2b4gh-0-pbhumflm5mn5"
> [dc66e4d9-0c9b-4b18-a2c6-dd9724fa51a9] : Authentication cannot be
> scoped to multiple target
> s. Pick one of: project, domain, trust or unscoped
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource Traceback
> (most recent call last):
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2462,
> in _handle_signal
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource    
> signal_result = self.handle_signal(details)
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/software_deployment.py",
> line 514, in handle_signal
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource    
> timeutils.utcnow().isoformat())
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 788, in
> signal_software_deployment
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource
>     version='1.6')
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 89, in call
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     return
> client.call(ctxt, method, **kwargs)
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line
> 165, in call
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     msg_ctxt
> = self.serializer.serialize_context(ctxt)
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/heat/common/messaging.py", line 46,
> in serialize_context
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     _context
> = ctxt.to_dict()
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/heat/common/context.py", line 185,
> in to_dict
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     'roles':
> self.roles,
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/heat/common/context.py", line 315,
> in roles
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource    
> self._load_keystone_data()
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 292, in
> wrapped_f
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     return
> self.call(f, *args, **kw)
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 358, in call
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     do =
> self.iter(retry_state=retry_state)
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 319, in iter
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     return
> fut.result()
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line
> 422, in result
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     return
> self.__get_result()
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 361, in call
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     result =
> fn(*args, **kwargs)
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/heat/common/context.py", line 306,
> in _load_keystone_data
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     auth_ref
> = self.auth_plugin.get_access(self.keystone_session)
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py",
> line 134, in get_access
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource    
> self.auth_ref = self.get_auth_ref(session)
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py",
> line 208, in get_auth_ref
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource     return
> self._plugin.get_auth_ref(session, **kwargs)
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource   File
> "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py",
> line 144, in get_auth_ref
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource    
> message='Authentication cannot be scoped to multiple'
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource
> AuthorizationFailure: Authentication cannot be scoped to multiple
> targets. Pick one of: project, domain, trust or unscoped
> 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service
> [req-a2c862eb-370c-4e91-a2c6-dca32c7872ce - - - - -] Unhandled error
> in asynchronous task: ResourceFailure: AuthorizationFailure:
> resources.master_config_deployment: Authentication cannot be scoped to
> multiple targets. Pick one of: project, domain, trust or unscoped
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service Traceback
> (most recent call last):
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service   File
> "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 132,
> in log_exceptions
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service     gt.wait()
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service   File
> "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 181,
> in wait
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service     return
> self._exit_event.wait()
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service   File
> "/usr/lib/python2.7/site-packages/eventlet/event.py", line 132, in wait
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service    
> current.throw(*self._exc)
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service   File
> "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 221,
> in main
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service     result =
> function(*args, **kwargs)
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service   File
> "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 123,
> in _start_with_trace
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service     return
> func(*args, **kwargs)
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service   File
> "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1871,
> in _resource_signal
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service    
> needs_metadata_updates = rsrc.signal(details, need_check)
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service   File
> "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2500,
> in signal
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service    
> self._handle_signal(details)
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service   File
> "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2480,
> in _handle_signal
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service     raise failure
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service
> ResourceFailure: AuthorizationFailure:
> resources.master_config_deployment: Authentication cannot be scoped to
> multiple targets. Pi
> ck one of: project, domain, trust or unscoped
> 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service
>
>  
>
> Each of the individual heat-engine daemons create magnum clusters
> correctly when they are the only ones online.
>
>  
>
> Attached are the heat and magnum config files.
>
>  
>
> Any ideas where to look would be appreciated?
>
>  
>
>  
>
> Regards
>
>  
>
> Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader
>
> For STFC Cloud Documentation visit
> https://stfc-cloud-docs.readthedocs.io
> <https://stfc-cloud-docs.readthedocs.io/>
>
> To raise a support ticket with the cloud team please email
> cloud-support at gridpp.rl.ac.uk <mailto:cloud-support at gridpp.rl.ac.uk>
>
> To receive notifications about the service please subscribe to our
> mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD
>
> To receive fast notifications or to discuss usage of the cloud please
> join our Slack: https://stfc-cloud.slack.com/
>
>  
>
> This email and any attachments are intended solely for the use of the
> named recipients. If you are not the intended recipient you must not
> use, disclose, copy or distribute this email or any of its attachments
> and should notify the sender immediately and delete this email from
> your system. UK Research and Innovation (UKRI) has taken every
> reasonable precaution to minimise risk of this email or any
> attachments containing viruses or malware but the recipient should
> carry out its own virus and malware checks before opening the
> attachments. UKRI does not accept any liability for any losses or
> damages which the recipient may sustain due to presence of any
> viruses. Opinions, conclusions or other information in this message
> and attachments that are not related directly to UKRI business are
> solely those of the author and do not represent the views of UKRI.
>
-- 
Cheers & Best regards,
Feilong Wang (王飞龙)
------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: flwang at catalyst.net.nz
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
------------------------------------------------------ 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200902/ccf9084e/attachment-0001.html>


More information about the openstack-discuss mailing list