<div dir="ltr"><div dir="ltr">Ciao Mark,<div>finally it works! Many many thanks! </div><div>That was the missing piece of the puzzle.</div><div><br></div><div>Just FYI information, from the systemctl status for the heat-container-agent I can still see this repetitive logs:</div><div><div><font face="monospace, monospace">:</font></div><div><font face="monospace, monospace">Feb 21 08:00:40 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping</font></div><div><font face="monospace, monospace">Feb 21 08:01:11 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping</font></div></div><div>:</div><div><br></div><div>This doesn't seem to harm the deployment but I will check further.</div><div><br></div><div>Thanks a lot to everyone!</div><div><br></div><div>/Giuseppe</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 20 Feb 2019 at 20:16, Mark Goddard <<a href="mailto:mark@stackhpc.com">mark@stackhpc.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default"><font face="verdana, sans-serif">Hi, I think we've hit this, and John Garbutt has added the following configuration for Kolla Ansible in</font> /etc/kolla/config/heat.conf:</div><div class="gmail_default"><br></div><div class="gmail_default"><table class="gmail-m_-2698930127641246007gmail-highlight gmail-m_-2698930127641246007gmail-tab-size gmail-m_-2698930127641246007gmail-js-file-line-container" style="box-sizing:border-box;border-collapse:collapse;border-spacing:0px;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px"><tbody style="box-sizing:border-box"><tr style="box-sizing:border-box"><td id="gmail-m_-2698930127641246007gmail-L1" class="gmail-m_-2698930127641246007gmail-blob-num gmail-m_-2698930127641246007gmail-js-line-number" style="box-sizing:border-box;padding:0px 10px;color:rgba(27,31,35,0.3);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;line-height:20px;min-width:50px;text-align:right;vertical-align:top;white-space:nowrap;width:50px"></td><td id="gmail-m_-2698930127641246007gmail-LC1" class="gmail-m_-2698930127641246007gmail-blob-code gmail-m_-2698930127641246007gmail-blob-code-inner gmail-m_-2698930127641246007gmail-js-file-line" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;overflow:visible;white-space:pre-wrap">[DEFAULT]</td></tr><tr style="box-sizing:border-box"><td id="gmail-m_-2698930127641246007gmail-L2" class="gmail-m_-2698930127641246007gmail-blob-num gmail-m_-2698930127641246007gmail-js-line-number" style="box-sizing:border-box;padding:0px 10px;color:rgba(27,31,35,0.3);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;line-height:20px;min-width:50px;text-align:right;vertical-align:top;white-space:nowrap;width:50px"></td><td id="gmail-m_-2698930127641246007gmail-LC2" class="gmail-m_-2698930127641246007gmail-blob-code gmail-m_-2698930127641246007gmail-blob-code-inner gmail-m_-2698930127641246007gmail-js-file-line" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;overflow:visible;white-space:pre-wrap">region_name_for_services=RegionOne
</td></tr></tbody></table><br></div><div class="gmail_default">We'll need a patch in kolla ansible to do that without custom config changes.</div><div class="gmail_default">Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 20 Feb 2019 at 11:05, Bharat Kunwar <<a href="mailto:bharat@stackhpc.com" target="_blank">bharat@stackhpc.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hi Giuseppe,<div><br></div><div>What version of heat are you running? </div><div><br></div><div>Can you check if you have this patch merged? <a href="https://review.openstack.org/579485" target="_blank">https://review.openstack.org/579485</a></div><div><br></div><div><a href="https://review.openstack.org/579485" target="_blank">https://review.openstack.org/579485</a></div><div><br></div><div>Bharat </div><div><br></div><div><div id="gmail-m_-2698930127641246007gmail-m_3103183791145376596AppleMailSignature" dir="ltr">Sent from my iPhone</div><div dir="ltr"><br>On 20 Feb 2019, at 10:38, Giuseppe Sannino <<a href="mailto:km.giuseppesannino@gmail.com" target="_blank">km.giuseppesannino@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Feilong, Bharat,<div>thanks for your answer.</div><div><br></div><div>@Feilong, <br></div><div>From /etc/kolla/heat-engine/heat.conf I see:</div><div><div><font face="monospace, monospace">[clients_keystone]</font></div><div><font face="monospace, monospace">auth_uri = <a href="http://10.1.7.201:5000" target="_blank">http://10.1.7.201:5000</a></font></div></div><div><br></div><div>This should map into auth_url within the k8s master. </div><div>Within the k8s master in /etc/os-collect-config.conf I see:</div><div><br></div><div><div><font face="monospace, monospace">[heat]</font></div><div><font face="monospace, monospace">auth_url = <a href="http://10.1.7.201:5000/v3/" target="_blank">http://10.1.7.201:5000/v3/</a></font></div></div><div><font face="monospace, monospace">:</font></div><div><font face="monospace, monospace">:</font></div><div><div><font face="monospace, monospace">resource_name = kube-master<br></font></div><div><font face="monospace, monospace">region_name = null</font></div></div><div><br></div><div><br></div><div>and from /etc/sysconfig/heat-params (among the others):</div><div><font face="monospace, monospace">:</font></div><div><div><font face="monospace, monospace">REGION_NAME="RegionOne"</font></div></div><div><div><font face="monospace, monospace">:</font></div><div><font face="monospace, monospace">AUTH_URL="<a href="http://10.1.7.201:5000/v3" target="_blank">http://10.1.7.201:5000/v3</a>"</font></div></div><div><br></div><div>This URL corresponds to the "public" Heat endpoint</div><div><div><font face="monospace, monospace">openstack endpoint list | grep heat</font></div><div><font face="monospace, monospace">| 3d5f58c43f6b44f6b54990d6fd9ff55d | RegionOne | heat | orchestration | True | internal | <a href="http://10.1.7.200:8004/v1/%(tenant_id)s" target="_blank">http://10.1.7.200:8004/v1/%(tenant_id)s</a> |</font></div><div><font face="monospace, monospace">| 8c2492cb0ddc48ca94942a4a299a88dc | RegionOne | heat-cfn | cloudformation | True | internal | <a href="http://10.1.7.200:8000/v1" target="_blank">http://10.1.7.200:8000/v1</a> |</font></div><div><font face="monospace, monospace">| b164c4618a784da9ae14da75a6c764a3 | RegionOne | heat | orchestration | True | public | <a href="http://10.1.7.201:8004/v1/%(tenant_id)s" target="_blank">http://10.1.7.201:8004/v1/%(tenant_id)s</a> |</font></div><div><font face="monospace, monospace">| da203f7d337b4587a0f5fc774c993390 | RegionOne | heat | orchestration | True | admin | <a href="http://10.1.7.200:8004/v1/%(tenant_id)s" target="_blank">http://10.1.7.200:8004/v1/%(tenant_id)s</a> |</font></div><div><font face="monospace, monospace">| e0d3743e7c604e5c8aa4684df2d1ce53 | RegionOne | heat-cfn | cloudformation | True | public | <a href="http://10.1.7.201:8000/v1" target="_blank">http://10.1.7.201:8000/v1</a> |</font></div><div><font face="monospace, monospace">| efe0b8418aa24dfca33c243e7eed7e90 | RegionOne | heat-cfn | cloudformation | True | admin | <a href="http://10.1.7.200:8000/v1" target="_blank">http://10.1.7.200:8000/v1</a> |</font></div></div><div><br></div><div>Connectivity tests:</div><div><div><font face="monospace, monospace">[fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ ping 10.1.7.201</font></div><div><font face="monospace, monospace">PING 10.1.7.201 (10.1.7.201) 56(84) bytes of data.</font></div><div><font face="monospace, monospace">64 bytes from <a href="http://10.1.7.201" target="_blank">10.1.7.201</a>: icmp_seq=1 ttl=63 time=0.285 ms</font></div></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">[fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ curl <a href="http://10.1.7.201:5000/v3/" target="_blank">http://10.1.7.201:5000/v3/</a></font></div><div><font face="monospace, monospace">{"version": {"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "<a href="http://10.1.7.201:5000/v3/" target="_blank">http://10.1.7.201:5000/v3/</a>", "rel": "self"}]}}</font></div></div><div><br></div><div><br></div><div>Apparently, I can reach such endpoint from within the k8s master</div><div><br></div><div><br></div><div>@Bharat,<br></div><div>that file seems to be properly conifugured to me as well.</div><div>The problem pointed by "systemctl status heat-container-agent" is with:</div><div><br></div><div><div><span style="font-family:monospace,monospace">Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found</span><br></div><div><font face="monospace, monospace">Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable.</font></div><div><font face="monospace, monospace">Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping</font></div><div><font face="monospace, monospace">Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found</font></div><div><font face="monospace, monospace">Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable.</font></div><div><font face="monospace, monospace">Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping</font></div><div><br></div></div><div><br></div><div>Still no way forward from my side.<br></div><div><br></div><div>/Giuseppe</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 19 Feb 2019 at 22:16, Bharat Kunwar <<a href="mailto:bharat@stackhpc.com" target="_blank">bharat@stackhpc.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">I have the same problem. Weird thing is /etc/sysconfig/heat-params has region_name specified in my case!<br><br><div id="gmail-m_-2698930127641246007gmail-m_3103183791145376596gmail-m_-2506836269617916698AppleMailSignature" dir="ltr">Sent from my iPhone</div><div dir="ltr"><br>On 19 Feb 2019, at 22:00, Feilong Wang <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr">
<p>Can you talk to the Heat API from your master node? <br>
</p>
<p><br>
</p>
<div class="gmail-m_-2698930127641246007gmail-m_3103183791145376596gmail-m_-2506836269617916698moz-cite-prefix">On 20/02/19 6:43 AM, Giuseppe Sannino
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi all...again,
<div>I managed to get over the previous issue by "not
disabling" the TLS in the cluster template.</div>
<div>From the cloud-init-output.log I see:</div>
<div>
<div><font face="monospace, monospace">Cloud-init v. 17.1
running 'modules:final' at Tue, 19 Feb 2019 17:03:53
+0000. Up 38.08 seconds.</font></div>
<div><font face="monospace, monospace">Cloud-init v. 17.1
finished at Tue, 19 Feb 2019 17:13:22 +0000.
Datasource DataSourceEc2. Up 607.13 seconds</font></div>
</div>
<div><br>
</div>
<div>But the cluster creation keeps on failing.</div>
<div><font face="monospace, monospace">From the journalctl
-f I see a possible issue:</font></div>
<div>
<div><font face="monospace, monospace">Feb 19 17:42:38
kube-cluster-tls-6hezqcq4ien3-master-0.novalocal
runc[2723]: publicURL endpoint for orchestration
service in null region not found</font></div>
<div><font face="monospace, monospace">Feb 19 17:42:38
kube-cluster-tls-6hezqcq4ien3-master-0.novalocal
runc[2723]: Source [heat] Unavailable.</font></div>
<div><font face="monospace, monospace">Feb 19 17:42:38
kube-cluster-tls-6hezqcq4ien3-master-0.novalocal
runc[2723]: /var/lib/os-collect-config/local-data not
found. Skipping</font></div>
</div>
<div><br>
</div>
<div>anyone familiar with this problem ?</div>
<div><br>
</div>
<div>Thanks as usual.</div>
<div>/Giuseppe</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 19 Feb 2019 at 17:35,
Giuseppe Sannino <<a href="mailto:km.giuseppesannino@gmail.com" target="_blank">km.giuseppesannino@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi all,
<div>need an help.</div>
<div>I deployed an AIO via Kolla on a baremetal
node. Here some information about the deployment:</div>
<div>---------------</div>
<div>kolla-ansible: 7.0.1</div>
<div>openstack_release: Rocky</div>
<div>kolla_base_distro: centos</div>
<div>kolla_install_type: source</div>
<div>TLS: disabled</div>
<div>--------------- <br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>VMs spawn without issue but I can't make the
"Kubernetes cluster creation" successfully. It
fails due to "Time out"</div>
<div><br>
</div>
<div>I managed to log into Kuber Master and from the
cloud-init-output.log I can see:</div>
<div>
<div><font face="monospace, monospace">+ echo
'Waiting for Kubernetes API...'</font></div>
<div><font face="monospace, monospace">Waiting for
Kubernetes API...</font></div>
<div><font face="monospace, monospace">++ curl
--silent <a href="http://127.0.0.1:8080/healthz" target="_blank">http://127.0.0.1:8080/healthz</a></font></div>
<div><font face="monospace, monospace">+ '[' ok =
'' ']'</font></div>
<div><font face="monospace, monospace">+ sleep 5</font></div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Checking via systemctl and journalctl I see:</div>
<div>
<div><font face="monospace, monospace">[fedora@kube-clsuter-qamdealetlbi-master-0
log]$ systemctl status kube-apiserver</font></div>
<div><font face="monospace, monospace">●
kube-apiserver.service - kubernetes-apiserver</font></div>
<div><font face="monospace, monospace"> Loaded:
loaded
(/etc/systemd/system/kube-apiserver.service;
enabled; vendor preset: disabled)</font></div>
<div><font face="monospace, monospace"> Active:
failed (Result: exit-code) since Tue
2019-02-19 15:31:41 UTC; 45min ago</font></div>
<div><font face="monospace, monospace"> Process:
3796 ExecStart=/usr/bin/runc --systemd-cgroup
run kube-apiserver (code=exited,
status=1/FAILURE)</font></div>
<div><font face="monospace, monospace"> Main PID:
3796 (code=exited, status=1/FAILURE)</font></div>
<div><font face="monospace, monospace"><br>
</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:40
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Main
process exited, code=exited, status=1/FAILURE</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:40
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Failed
with result 'exit-code'.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:41
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Service
RestartSec=100ms expired, scheduling restart.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:41
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Scheduled
restart job, restart counter is at 6.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:41
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: Stopped kubernetes-apiserver.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:41
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Start
request repeated too quickly.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:41
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Failed
with result 'exit-code'.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:41
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: Failed to start
kubernetes-apiserver.</font></div>
</div>
<div><font face="monospace, monospace"><br>
</font></div>
<div>
<div><font face="monospace, monospace">[fedora@kube-clsuter-qamdealetlbi-master-0
log]$ sudo journalctl -u kube-apiserver</font></div>
<div><font face="monospace, monospace">-- Logs
begin at Tue 2019-02-19 15:21:36 UTC, end at
Tue 2019-02-19 16:17:00 UTC. --</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:33
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: Started kubernetes-apiserver.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:34
kube-clsuter-qamdealetlbi-master-0.novalocal
runc[2794]: Flag --insecure-bind-address has
been deprecated, This flag will be removed in
a future version.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:34
kube-clsuter-qamdealetlbi-master-0.novalocal
runc[2794]: Flag --insecure-port has been
deprecated, This flag will be removed in a
future version.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:35
kube-clsuter-qamdealetlbi-master-0.novalocal
runc[2794]: Error: error creating self-signed
certificates: open
/var/run/kubernetes/apiserver.crt: permission
denied</font></div>
</div>
<div><font face="monospace, monospace">:</font></div>
<div><font face="monospace, monospace">:</font></div>
<div><font face="monospace, monospace">:</font></div>
<div>
<div><font face="monospace, monospace">Feb 19
15:31:35
kube-clsuter-qamdealetlbi-master-0.novalocal
runc[2794]: error: error creating self-signed
certificates: open
/var/run/kubernetes/apiserver.crt: permission
denied</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:35
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Main
process exited, code=exited, status=1/FAILURE</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:35
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Failed
with result 'exit-code'.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:35
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Service
RestartSec=100ms expired, scheduling restart.</font></div>
<div><font face="monospace, monospace">Feb 19
15:31:35
kube-clsuter-qamdealetlbi-master-0.novalocal
systemd[1]: kube-apiserver.service: Scheduled
restart job, restart counter is at 1.</font></div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>May I ask for an help on this ?</div>
<div><br>
</div>
<div>Many thanks</div>
<div>/Giuseppe</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<pre class="gmail-m_-2698930127641246007gmail-m_3103183791145376596gmail-m_-2506836269617916698moz-signature" cols="72">--
Cheers & Best regards,
Feilong Wang (王飞龙)
--------------------------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: <a class="gmail-m_-2698930127641246007gmail-m_3103183791145376596gmail-m_-2506836269617916698moz-txt-link-abbreviated" href="mailto:flwang@catalyst.net.nz" target="_blank">flwang@catalyst.net.nz</a>
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
-------------------------------------------------------------------------- </pre>
</div></blockquote></div></blockquote></div>
</div></blockquote></div></div></blockquote></div>
</blockquote></div>