<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Cascadia Mono";
panose-1:2 11 6 9 2 0 0 2 0 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am having an issue with magnum creating clusters when I have multiple active heat-engine daemons running.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I get the following error in the heat engine logs:<o:p></o:p></p>
<p class="MsoNormal">2020-08-14 10:36:30.237 598383 INFO heat.engine.resource [req-a2c862eb-370c-4e91-a2c6-dca32c7872ce - - - - -] signal SoftwareDeployment "master_config_deployment" [67ba9ce2-aba5-4c15-a7ea<br>
-6b774659a0e2] Stack "kubernetes-test-26-3uzjqqob47fh-kube_masters-mhctjio2b4gh-0-pbhumflm5mn5" [dc66e4d9-0c9b-4b18-a2c6-dd9724fa51a9] : Authentication cannot be scoped to multiple target<br>
s. Pick one of: project, domain, trust or unscoped<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource Traceback (most recent call last):<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2462, in _handle_signal<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource signal_result = self.handle_signal(details)<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/software_deployment.py", line 514, in handle_signal<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource timeutils.utcnow().isoformat())<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 788, in signal_software_deployment<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource version='1.6')<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 89, in call<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs)<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 165, in call<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource msg_ctxt = self.serializer.serialize_context(ctxt)<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/messaging.py", line 46, in serialize_context<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource _context = ctxt.to_dict()<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 185, in to_dict<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource 'roles': self.roles,<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 315, in roles<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource self._load_keystone_data()<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 292, in wrapped_f<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return self.call(f, *args, **kw)<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 358, in call<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource do = self.iter(retry_state=retry_state)<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 319, in iter<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return fut.result()<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return self.__get_result()<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 361, in call<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource result = fn(*args, **kwargs)<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 306, in _load_keystone_data<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource auth_ref = self.auth_plugin.get_access(self.keystone_session)<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource self.auth_ref = self.get_auth_ref(session)<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return self._plugin.get_auth_ref(session, **kwargs)<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 144, in get_auth_ref<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource message='Authentication cannot be scoped to multiple'<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource AuthorizationFailure: Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped<br>
2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service [req-a2c862eb-370c-4e91-a2c6-dca32c7872ce - - - - -] Unhandled error in asynchronous task: ResourceFailure: AuthorizationFailure:
<br>
resources.master_config_deployment: Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service Traceback (most recent call last):<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 132, in log_exceptions<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service gt.wait()<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 181, in wait<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service return self._exit_event.wait()<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 132, in wait<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service current.throw(*self._exc)<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 221, in main<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service result = function(*args, **kwargs)<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 123, in _start_with_trace<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service return func(*args, **kwargs)<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1871, in _resource_signal<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service needs_metadata_updates = rsrc.signal(details, need_check)<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2500, in signal<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service self._handle_signal(details)<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2480, in _handle_signal<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service raise failure<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service ResourceFailure: AuthorizationFailure: resources.master_config_deployment: Authentication cannot be scoped to multiple targets. Pi<br>
ck one of: project, domain, trust or unscoped<br>
2020-08-14 10:36:30.890 598383 ERROR heat.engine.service<span style="font-size:12.0pt;font-family:"Cascadia Mono";mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Each of the individual heat-engine daemons create magnum clusters correctly when they are the only ones online.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Attached are the heat and magnum config files.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any ideas where to look would be appreciated?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Regards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">For STFC Cloud Documentation visit
<a href="https://stfc-cloud-docs.readthedocs.io/"><span style="color:#0563C1">https://stfc-cloud-docs.readthedocs.io</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">To raise a support ticket with the cloud team please email
<a href="mailto:cloud-support@gridpp.rl.ac.uk"><span style="color:#0563C1">cloud-support@gridpp.rl.ac.uk</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">To receive notifications about the service please subscribe to our mailing list at:
<a href="https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD"><span style="color:#0563C1">https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">To receive fast notifications or to discuss usage of the cloud please join our Slack:
<a href="https://stfc-cloud.slack.com/"><span style="color:#0563C1">https://stfc-cloud.slack.com/</span></a>
<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p><span style="font-size: 6pt;">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.</span></p></body>
</html>