<div dir="ltr"><div class="gmail_default" style=""><font face="verdana, sans-serif">Hi, I think we've hit this, and John Garbutt has added the following configuration for Kolla Ansible in</font> /etc/kolla/config/heat.conf:</div><div class="gmail_default" style=""><br></div><div class="gmail_default" style=""><table class="gmail-highlight gmail-tab-size gmail-js-file-line-container" style="box-sizing:border-box;border-collapse:collapse;border-spacing:0px;color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px"><tbody style="box-sizing:border-box"><tr style="box-sizing:border-box"><td id="gmail-L1" class="gmail-blob-num gmail-js-line-number" style="box-sizing:border-box;padding:0px 10px;color:rgba(27,31,35,0.3);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;line-height:20px;min-width:50px;text-align:right;vertical-align:top;white-space:nowrap;width:50px"></td><td id="gmail-LC1" class="gmail-blob-code gmail-blob-code-inner gmail-js-file-line" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;overflow:visible;white-space:pre">[DEFAULT]</td></tr><tr style="box-sizing:border-box"><td id="gmail-L2" class="gmail-blob-num gmail-js-line-number" style="box-sizing:border-box;padding:0px 10px;color:rgba(27,31,35,0.3);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;line-height:20px;min-width:50px;text-align:right;vertical-align:top;white-space:nowrap;width:50px"></td><td id="gmail-LC2" class="gmail-blob-code gmail-blob-code-inner gmail-js-file-line" style="box-sizing:border-box;padding:0px 10px;line-height:20px;vertical-align:top;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;overflow:visible;white-space:pre">region_name_for_services=RegionOne
</td></tr></tbody></table><br></div><div class="gmail_default" style="">We'll need a patch in kolla ansible to do that without custom config changes.</div><div class="gmail_default" style="">Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 20 Feb 2019 at 11:05, Bharat Kunwar <<a href="mailto:bharat@stackhpc.com">bharat@stackhpc.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hi Giuseppe,<div><br></div><div>What version of heat are you running? </div><div><br></div><div>Can you check if you have this patch merged? <a href="https://review.openstack.org/579485" target="_blank">https://review.openstack.org/579485</a></div><div><br></div><div><a href="https://review.openstack.org/579485" target="_blank">https://review.openstack.org/579485</a></div><div><br></div><div>Bharat </div><div><br></div><div><div id="gmail-m_3103183791145376596AppleMailSignature" dir="ltr">Sent from my iPhone</div><div dir="ltr"><br>On 20 Feb 2019, at 10:38, Giuseppe Sannino <<a href="mailto:km.giuseppesannino@gmail.com" target="_blank">km.giuseppesannino@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Feilong, Bharat,<div>thanks for your answer.</div><div><br></div><div>@Feilong, <br></div><div>From /etc/kolla/heat-engine/heat.conf I see:</div><div><div><font face="monospace, monospace">[clients_keystone]</font></div><div><font face="monospace, monospace">auth_uri = <a href="http://10.1.7.201:5000" target="_blank">http://10.1.7.201:5000</a></font></div></div><div><br></div><div>This should map into auth_url within the k8s master. </div><div>Within the k8s master in /etc/os-collect-config.conf  I see:</div><div><br></div><div><div><font face="monospace, monospace">[heat]</font></div><div><font face="monospace, monospace">auth_url = <a href="http://10.1.7.201:5000/v3/" target="_blank">http://10.1.7.201:5000/v3/</a></font></div></div><div><font face="monospace, monospace">:</font></div><div><font face="monospace, monospace">:</font></div><div><div><font face="monospace, monospace">resource_name = kube-master<br></font></div><div><font face="monospace, monospace">region_name = null</font></div></div><div><br></div><div><br></div><div>and from /etc/sysconfig/heat-params (among the others):</div><div><font face="monospace, monospace">:</font></div><div><div><font face="monospace, monospace">REGION_NAME="RegionOne"</font></div></div><div><div><font face="monospace, monospace">:</font></div><div><font face="monospace, monospace">AUTH_URL="<a href="http://10.1.7.201:5000/v3" target="_blank">http://10.1.7.201:5000/v3</a>"</font></div></div><div><br></div><div>This URL corresponds to the "public" Heat endpoint</div><div><div><font face="monospace, monospace">openstack endpoint list | grep heat</font></div><div><font face="monospace, monospace">| 3d5f58c43f6b44f6b54990d6fd9ff55d | RegionOne | heat         | orchestration   | True    | internal  | <a href="http://10.1.7.200:8004/v1/%(tenant_id)s" target="_blank">http://10.1.7.200:8004/v1/%(tenant_id)s</a>   |</font></div><div><font face="monospace, monospace">| 8c2492cb0ddc48ca94942a4a299a88dc | RegionOne | heat-cfn     | cloudformation  | True    | internal  | <a href="http://10.1.7.200:8000/v1" target="_blank">http://10.1.7.200:8000/v1</a>                 |</font></div><div><font face="monospace, monospace">| b164c4618a784da9ae14da75a6c764a3 | RegionOne | heat         | orchestration   | True    | public    | <a href="http://10.1.7.201:8004/v1/%(tenant_id)s" target="_blank">http://10.1.7.201:8004/v1/%(tenant_id)s</a>   |</font></div><div><font face="monospace, monospace">| da203f7d337b4587a0f5fc774c993390 | RegionOne | heat         | orchestration   | True    | admin     | <a href="http://10.1.7.200:8004/v1/%(tenant_id)s" target="_blank">http://10.1.7.200:8004/v1/%(tenant_id)s</a>   |</font></div><div><font face="monospace, monospace">| e0d3743e7c604e5c8aa4684df2d1ce53 | RegionOne | heat-cfn     | cloudformation  | True    | public    | <a href="http://10.1.7.201:8000/v1" target="_blank">http://10.1.7.201:8000/v1</a>                 |</font></div><div><font face="monospace, monospace">| efe0b8418aa24dfca33c243e7eed7e90 | RegionOne | heat-cfn     | cloudformation  | True    | admin     | <a href="http://10.1.7.200:8000/v1" target="_blank">http://10.1.7.200:8000/v1</a>                 |</font></div></div><div><br></div><div>Connectivity tests:</div><div><div><font face="monospace, monospace">[fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ ping 10.1.7.201</font></div><div><font face="monospace, monospace">PING 10.1.7.201 (10.1.7.201) 56(84) bytes of data.</font></div><div><font face="monospace, monospace">64 bytes from <a href="http://10.1.7.201" target="_blank">10.1.7.201</a>: icmp_seq=1 ttl=63 time=0.285 ms</font></div></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">[fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ curl <a href="http://10.1.7.201:5000/v3/" target="_blank">http://10.1.7.201:5000/v3/</a></font></div><div><font face="monospace, monospace">{"version": {"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "<a href="http://10.1.7.201:5000/v3/" target="_blank">http://10.1.7.201:5000/v3/</a>", "rel": "self"}]}}</font></div></div><div><br></div><div><br></div><div>Apparently, I can reach such endpoint from within the k8s master</div><div><br></div><div><br></div><div>@Bharat,<br></div><div>that file seems to be properly conifugured to me as well.</div><div>The problem pointed by "systemctl status heat-container-agent" is with:</div><div><br></div><div><div><span style="font-family:monospace,monospace">Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found</span><br></div><div><font face="monospace, monospace">Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable.</font></div><div><font face="monospace, monospace">Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping</font></div><div><font face="monospace, monospace">Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found</font></div><div><font face="monospace, monospace">Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable.</font></div><div><font face="monospace, monospace">Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping</font></div><div><br></div></div><div><br></div><div>Still no way forward from my side.<br></div><div><br></div><div>/Giuseppe</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 19 Feb 2019 at 22:16, Bharat Kunwar <<a href="mailto:bharat@stackhpc.com" target="_blank">bharat@stackhpc.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">I have the same problem. Weird thing is /etc/sysconfig/heat-params has region_name specified in my case!<br><br><div id="gmail-m_3103183791145376596gmail-m_-2506836269617916698AppleMailSignature" dir="ltr">Sent from my iPhone</div><div dir="ltr"><br>On 19 Feb 2019, at 22:00, Feilong Wang <<a href="mailto:feilong@catalyst.net.nz" target="_blank">feilong@catalyst.net.nz</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr">
  
    
  
  
    <p>Can you talk to the Heat API from your master node? <br>
    </p>
    <p><br>
    </p>
    <div class="gmail-m_3103183791145376596gmail-m_-2506836269617916698moz-cite-prefix">On 20/02/19 6:43 AM, Giuseppe Sannino
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr">
          <div dir="ltr">Hi all...again,
            <div>I managed to get over the previous issue by "not
              disabling" the TLS in the cluster template.</div>
            <div>From the cloud-init-output.log I see:</div>
            <div>
              <div><font face="monospace, monospace">Cloud-init v. 17.1
                  running 'modules:final' at Tue, 19 Feb 2019 17:03:53
                  +0000. Up 38.08 seconds.</font></div>
              <div><font face="monospace, monospace">Cloud-init v. 17.1
                  finished at Tue, 19 Feb 2019 17:13:22 +0000.
                  Datasource DataSourceEc2.  Up 607.13 seconds</font></div>
            </div>
            <div><br>
            </div>
            <div>But the cluster creation keeps on failing.</div>
            <div><font face="monospace, monospace">From the journalctl
                -f I see a possible issue:</font></div>
            <div>
              <div><font face="monospace, monospace">Feb 19 17:42:38
                  kube-cluster-tls-6hezqcq4ien3-master-0.novalocal
                  runc[2723]: publicURL endpoint for orchestration
                  service in null region not found</font></div>
              <div><font face="monospace, monospace">Feb 19 17:42:38
                  kube-cluster-tls-6hezqcq4ien3-master-0.novalocal
                  runc[2723]: Source [heat] Unavailable.</font></div>
              <div><font face="monospace, monospace">Feb 19 17:42:38
                  kube-cluster-tls-6hezqcq4ien3-master-0.novalocal
                  runc[2723]: /var/lib/os-collect-config/local-data not
                  found. Skipping</font></div>
            </div>
            <div><br>
            </div>
            <div>anyone familiar with this problem ?</div>
            <div><br>
            </div>
            <div>Thanks as usual.</div>
            <div>/Giuseppe</div>
            <div><br>
            </div>
            <div><br>
            </div>
            <div><br>
            </div>
            <div><br>
            </div>
            <div><br>
            </div>
            <div><br>
            </div>
          </div>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">On Tue, 19 Feb 2019 at 17:35,
          Giuseppe Sannino <<a href="mailto:km.giuseppesannino@gmail.com" target="_blank">km.giuseppesannino@gmail.com</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="ltr">
            <div dir="ltr">
              <div dir="ltr">
                <div dir="ltr">
                  <div dir="ltr">Hi all,
                    <div>need an help.</div>
                    <div>I deployed an AIO via Kolla on a baremetal
                      node. Here some information about the deployment:</div>
                    <div>---------------</div>
                    <div>kolla-ansible: 7.0.1</div>
                    <div>openstack_release: Rocky</div>
                    <div>kolla_base_distro: centos</div>
                    <div>kolla_install_type: source</div>
                    <div>TLS: disabled</div>
                    <div>---------------  <br>
                    </div>
                    <div><br>
                    </div>
                    <div><br>
                    </div>
                    <div>VMs spawn without issue but I can't make the
                      "Kubernetes cluster creation" successfully. It
                      fails due to "Time out"</div>
                    <div><br>
                    </div>
                    <div>I managed to log into Kuber Master and from the
                      cloud-init-output.log I can see:</div>
                    <div>
                      <div><font face="monospace, monospace">+ echo
                          'Waiting for Kubernetes API...'</font></div>
                      <div><font face="monospace, monospace">Waiting for
                          Kubernetes API...</font></div>
                      <div><font face="monospace, monospace">++ curl
                          --silent <a href="http://127.0.0.1:8080/healthz" target="_blank">http://127.0.0.1:8080/healthz</a></font></div>
                      <div><font face="monospace, monospace">+ '[' ok =
                          '' ']'</font></div>
                      <div><font face="monospace, monospace">+ sleep 5</font></div>
                    </div>
                    <div><br>
                    </div>
                    <div><br>
                    </div>
                    <div>Checking via systemctl and journalctl I see:</div>
                    <div>
                      <div><font face="monospace, monospace">[fedora@kube-clsuter-qamdealetlbi-master-0
                          log]$ systemctl status kube-apiserver</font></div>
                      <div><font face="monospace, monospace">●
                          kube-apiserver.service - kubernetes-apiserver</font></div>
                      <div><font face="monospace, monospace">   Loaded:
                          loaded
                          (/etc/systemd/system/kube-apiserver.service;
                          enabled; vendor preset: disabled)</font></div>
                      <div><font face="monospace, monospace">   Active:
                          failed (Result: exit-code) since Tue
                          2019-02-19 15:31:41 UTC; 45min ago</font></div>
                      <div><font face="monospace, monospace">  Process:
                          3796 ExecStart=/usr/bin/runc --systemd-cgroup
                          run kube-apiserver (code=exited,
                          status=1/FAILURE)</font></div>
                      <div><font face="monospace, monospace"> Main PID:
                          3796 (code=exited, status=1/FAILURE)</font></div>
                      <div><font face="monospace, monospace"><br>
                        </font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:40
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Main
                          process exited, code=exited, status=1/FAILURE</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:40
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Failed
                          with result 'exit-code'.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:41
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Service
                          RestartSec=100ms expired, scheduling restart.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:41
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Scheduled
                          restart job, restart counter is at 6.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:41
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: Stopped kubernetes-apiserver.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:41
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Start
                          request repeated too quickly.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:41
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Failed
                          with result 'exit-code'.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:41
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: Failed to start
                          kubernetes-apiserver.</font></div>
                    </div>
                    <div><font face="monospace, monospace"><br>
                      </font></div>
                    <div>
                      <div><font face="monospace, monospace">[fedora@kube-clsuter-qamdealetlbi-master-0
                          log]$ sudo journalctl -u kube-apiserver</font></div>
                      <div><font face="monospace, monospace">-- Logs
                          begin at Tue 2019-02-19 15:21:36 UTC, end at
                          Tue 2019-02-19 16:17:00 UTC. --</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:33
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: Started kubernetes-apiserver.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:34
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          runc[2794]: Flag --insecure-bind-address has
                          been deprecated, This flag will be removed in
                          a future version.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:34
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          runc[2794]: Flag --insecure-port has been
                          deprecated, This flag will be removed in a
                          future version.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:35
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          runc[2794]: Error: error creating self-signed
                          certificates: open
                          /var/run/kubernetes/apiserver.crt: permission
                          denied</font></div>
                    </div>
                    <div><font face="monospace, monospace">:</font></div>
                    <div><font face="monospace, monospace">:</font></div>
                    <div><font face="monospace, monospace">:</font></div>
                    <div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:35
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          runc[2794]: error: error creating self-signed
                          certificates: open
                          /var/run/kubernetes/apiserver.crt: permission
                          denied</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:35
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Main
                          process exited, code=exited, status=1/FAILURE</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:35
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Failed
                          with result 'exit-code'.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:35
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Service
                          RestartSec=100ms expired, scheduling restart.</font></div>
                      <div><font face="monospace, monospace">Feb 19
                          15:31:35
                          kube-clsuter-qamdealetlbi-master-0.novalocal
                          systemd[1]: kube-apiserver.service: Scheduled
                          restart job, restart counter is at 1.</font></div>
                    </div>
                    <div><br>
                    </div>
                    <div><br>
                    </div>
                    <div>May I ask for an help on this ?</div>
                    <div><br>
                    </div>
                    <div>Many thanks</div>
                    <div>/Giuseppe</div>
                    <div><br>
                    </div>
                    <div><br>
                    </div>
                    <div><br>
                    </div>
                    <div>
                      <div><br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <pre class="gmail-m_3103183791145376596gmail-m_-2506836269617916698moz-signature" cols="72">-- 
Cheers & Best regards,
Feilong Wang (王飞龙)
--------------------------------------------------------------------------
Senior Cloud Software Engineer
Tel: +64-48032246
Email: <a class="gmail-m_3103183791145376596gmail-m_-2506836269617916698moz-txt-link-abbreviated" href="mailto:flwang@catalyst.net.nz" target="_blank">flwang@catalyst.net.nz</a>
Catalyst IT Limited
Level 6, Catalyst House, 150 Willis Street, Wellington
-------------------------------------------------------------------------- </pre>
  

</div></blockquote></div></blockquote></div>
</div></blockquote></div></div></blockquote></div>