[kolla][magnum] Cluster creation failed due to "Waiting for Kubernetes API..."
Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled --------------- VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out" I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5 Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE) Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver. [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1. May I ask for an help on this ? Many thanks /Giuseppe
Hi all...again, I managed to get over the previous issue by "not disabling" the TLS in the cluster template.
From the cloud-init-output.log I see: Cloud-init v. 17.1 running 'modules:final' at Tue, 19 Feb 2019 17:03:53 +0000. Up 38.08 seconds. Cloud-init v. 17.1 finished at Tue, 19 Feb 2019 17:13:22 +0000. Datasource DataSourceEc2. Up 607.13 seconds
But the cluster creation keeps on failing.
From the journalctl -f I see a possible issue: Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: publicURL endpoint for orchestration service in null region not found Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: Source [heat] Unavailable. Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: /var/lib/os-collect-config/local-data not found. Skipping
anyone familiar with this problem ? Thanks as usual. /Giuseppe On Tue, 19 Feb 2019 at 17:35, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote:
Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled ---------------
VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out"
I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5
Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE)
Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver.
[fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1.
May I ask for an help on this ?
Many thanks /Giuseppe
Can you talk to the Heat API from your master node? On 20/02/19 6:43 AM, Giuseppe Sannino wrote:
Hi all...again, I managed to get over the previous issue by "not disabling" the TLS in the cluster template. From the cloud-init-output.log I see: Cloud-init v. 17.1 running 'modules:final' at Tue, 19 Feb 2019 17:03:53 +0000. Up 38.08 seconds. Cloud-init v. 17.1 finished at Tue, 19 Feb 2019 17:13:22 +0000. Datasource DataSourceEc2. Up 607.13 seconds
But the cluster creation keeps on failing. From the journalctl -f I see a possible issue: Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: publicURL endpoint for orchestration service in null region not found Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: Source [heat] Unavailable. Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: /var/lib/os-collect-config/local-data not found. Skipping
anyone familiar with this problem ?
Thanks as usual. /Giuseppe
On Tue, 19 Feb 2019 at 17:35, Giuseppe Sannino <km.giuseppesannino@gmail.com <mailto:km.giuseppesannino@gmail.com>> wrote:
Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled ---------------
VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out"
I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5
Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE)
Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver.
[fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1.
May I ask for an help on this ?
Many thanks /Giuseppe
-- Cheers & Best regards, Feilong Wang (王飞龙) -------------------------------------------------------------------------- Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
I have the same problem. Weird thing is /etc/sysconfig/heat-params has region_name specified in my case! Sent from my iPhone
On 19 Feb 2019, at 22:00, Feilong Wang <feilong@catalyst.net.nz> wrote:
Can you talk to the Heat API from your master node?
On 20/02/19 6:43 AM, Giuseppe Sannino wrote: Hi all...again, I managed to get over the previous issue by "not disabling" the TLS in the cluster template. From the cloud-init-output.log I see: Cloud-init v. 17.1 running 'modules:final' at Tue, 19 Feb 2019 17:03:53 +0000. Up 38.08 seconds. Cloud-init v. 17.1 finished at Tue, 19 Feb 2019 17:13:22 +0000. Datasource DataSourceEc2. Up 607.13 seconds
But the cluster creation keeps on failing. From the journalctl -f I see a possible issue: Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: publicURL endpoint for orchestration service in null region not found Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: Source [heat] Unavailable. Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: /var/lib/os-collect-config/local-data not found. Skipping
anyone familiar with this problem ?
Thanks as usual. /Giuseppe
On Tue, 19 Feb 2019 at 17:35, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote: Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled ---------------
VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out"
I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5
Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE)
Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver.
[fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1.
May I ask for an help on this ?
Many thanks /Giuseppe
-- Cheers & Best regards, Feilong Wang (王飞龙)
Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
Hi Feilong, Bharat, thanks for your answer. @Feilong,
From /etc/kolla/heat-engine/heat.conf I see: [clients_keystone] auth_uri = http://10.1.7.201:5000
This should map into auth_url within the k8s master. Within the k8s master in /etc/os-collect-config.conf I see: [heat] auth_url = http://10.1.7.201:5000/v3/ : : resource_name = kube-master region_name = null and from /etc/sysconfig/heat-params (among the others): : REGION_NAME="RegionOne" : AUTH_URL="http://10.1.7.201:5000/v3" This URL corresponds to the "public" Heat endpoint openstack endpoint list | grep heat | 3d5f58c43f6b44f6b54990d6fd9ff55d | RegionOne | heat | orchestration | True | internal | http://10.1.7.200:8004/v1/%(tenant_id)s | | 8c2492cb0ddc48ca94942a4a299a88dc | RegionOne | heat-cfn | cloudformation | True | internal | http://10.1.7.200:8000/v1 | | b164c4618a784da9ae14da75a6c764a3 | RegionOne | heat | orchestration | True | public | http://10.1.7.201:8004/v1/%(tenant_id)s | | da203f7d337b4587a0f5fc774c993390 | RegionOne | heat | orchestration | True | admin | http://10.1.7.200:8004/v1/%(tenant_id)s | | e0d3743e7c604e5c8aa4684df2d1ce53 | RegionOne | heat-cfn | cloudformation | True | public | http://10.1.7.201:8000/v1 | | efe0b8418aa24dfca33c243e7eed7e90 | RegionOne | heat-cfn | cloudformation | True | admin | http://10.1.7.200:8000/v1 | Connectivity tests: [fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ ping 10.1.7.201 PING 10.1.7.201 (10.1.7.201) 56(84) bytes of data. 64 bytes from 10.1.7.201: icmp_seq=1 ttl=63 time=0.285 ms [fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ curl http://10.1.7.201:5000/v3/ {"version": {"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "http://10.1.7.201:5000/v3/", "rel": "self"}]}} Apparently, I can reach such endpoint from within the k8s master @Bharat, that file seems to be properly conifugured to me as well. The problem pointed by "systemctl status heat-container-agent" is with: Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping Still no way forward from my side. /Giuseppe On Tue, 19 Feb 2019 at 22:16, Bharat Kunwar <bharat@stackhpc.com> wrote:
I have the same problem. Weird thing is /etc/sysconfig/heat-params has region_name specified in my case!
Sent from my iPhone
On 19 Feb 2019, at 22:00, Feilong Wang <feilong@catalyst.net.nz> wrote:
Can you talk to the Heat API from your master node?
On 20/02/19 6:43 AM, Giuseppe Sannino wrote:
Hi all...again, I managed to get over the previous issue by "not disabling" the TLS in the cluster template. From the cloud-init-output.log I see: Cloud-init v. 17.1 running 'modules:final' at Tue, 19 Feb 2019 17:03:53 +0000. Up 38.08 seconds. Cloud-init v. 17.1 finished at Tue, 19 Feb 2019 17:13:22 +0000. Datasource DataSourceEc2. Up 607.13 seconds
But the cluster creation keeps on failing. From the journalctl -f I see a possible issue: Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: publicURL endpoint for orchestration service in null region not found Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: Source [heat] Unavailable. Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: /var/lib/os-collect-config/local-data not found. Skipping
anyone familiar with this problem ?
Thanks as usual. /Giuseppe
On Tue, 19 Feb 2019 at 17:35, Giuseppe Sannino < km.giuseppesannino@gmail.com> wrote:
Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled ---------------
VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out"
I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5
Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE)
Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver.
[fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1.
May I ask for an help on this ?
Many thanks /Giuseppe
-- Cheers & Best regards, Feilong Wang (王飞龙)
Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
Hi Giuseppe, What version of heat are you running? Can you check if you have this patch merged? https://review.openstack.org/579485 https://review.openstack.org/579485 Bharat Sent from my iPhone
On 20 Feb 2019, at 10:38, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote:
Hi Feilong, Bharat, thanks for your answer.
@Feilong, From /etc/kolla/heat-engine/heat.conf I see: [clients_keystone] auth_uri = http://10.1.7.201:5000
This should map into auth_url within the k8s master. Within the k8s master in /etc/os-collect-config.conf I see:
[heat] auth_url = http://10.1.7.201:5000/v3/ : : resource_name = kube-master region_name = null
and from /etc/sysconfig/heat-params (among the others): : REGION_NAME="RegionOne" : AUTH_URL="http://10.1.7.201:5000/v3"
This URL corresponds to the "public" Heat endpoint openstack endpoint list | grep heat | 3d5f58c43f6b44f6b54990d6fd9ff55d | RegionOne | heat | orchestration | True | internal | http://10.1.7.200:8004/v1/%(tenant_id)s | | 8c2492cb0ddc48ca94942a4a299a88dc | RegionOne | heat-cfn | cloudformation | True | internal | http://10.1.7.200:8000/v1 | | b164c4618a784da9ae14da75a6c764a3 | RegionOne | heat | orchestration | True | public | http://10.1.7.201:8004/v1/%(tenant_id)s | | da203f7d337b4587a0f5fc774c993390 | RegionOne | heat | orchestration | True | admin | http://10.1.7.200:8004/v1/%(tenant_id)s | | e0d3743e7c604e5c8aa4684df2d1ce53 | RegionOne | heat-cfn | cloudformation | True | public | http://10.1.7.201:8000/v1 | | efe0b8418aa24dfca33c243e7eed7e90 | RegionOne | heat-cfn | cloudformation | True | admin | http://10.1.7.200:8000/v1 |
Connectivity tests: [fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ ping 10.1.7.201 PING 10.1.7.201 (10.1.7.201) 56(84) bytes of data. 64 bytes from 10.1.7.201: icmp_seq=1 ttl=63 time=0.285 ms
[fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ curl http://10.1.7.201:5000/v3/ {"version": {"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "http://10.1.7.201:5000/v3/", "rel": "self"}]}}
Apparently, I can reach such endpoint from within the k8s master
@Bharat, that file seems to be properly conifugured to me as well. The problem pointed by "systemctl status heat-container-agent" is with:
Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping
Still no way forward from my side.
/Giuseppe
On Tue, 19 Feb 2019 at 22:16, Bharat Kunwar <bharat@stackhpc.com> wrote: I have the same problem. Weird thing is /etc/sysconfig/heat-params has region_name specified in my case!
Sent from my iPhone
On 19 Feb 2019, at 22:00, Feilong Wang <feilong@catalyst.net.nz> wrote:
Can you talk to the Heat API from your master node?
On 20/02/19 6:43 AM, Giuseppe Sannino wrote: Hi all...again, I managed to get over the previous issue by "not disabling" the TLS in the cluster template. From the cloud-init-output.log I see: Cloud-init v. 17.1 running 'modules:final' at Tue, 19 Feb 2019 17:03:53 +0000. Up 38.08 seconds. Cloud-init v. 17.1 finished at Tue, 19 Feb 2019 17:13:22 +0000. Datasource DataSourceEc2. Up 607.13 seconds
But the cluster creation keeps on failing. From the journalctl -f I see a possible issue: Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: publicURL endpoint for orchestration service in null region not found Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: Source [heat] Unavailable. Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: /var/lib/os-collect-config/local-data not found. Skipping
anyone familiar with this problem ?
Thanks as usual. /Giuseppe
On Tue, 19 Feb 2019 at 17:35, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote: Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled ---------------
VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out"
I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5
Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE)
Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver.
[fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1.
May I ask for an help on this ?
Many thanks /Giuseppe
-- Cheers & Best regards, Feilong Wang (王飞龙)
Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
Hi, I think we've hit this, and John Garbutt has added the following configuration for Kolla Ansible in /etc/kolla/config/heat.conf: [DEFAULT] region_name_for_services=RegionOne We'll need a patch in kolla ansible to do that without custom config changes. Mark On Wed, 20 Feb 2019 at 11:05, Bharat Kunwar <bharat@stackhpc.com> wrote:
Hi Giuseppe,
What version of heat are you running?
Can you check if you have this patch merged? https://review.openstack.org/579485
https://review.openstack.org/579485
Bharat
Sent from my iPhone
On 20 Feb 2019, at 10:38, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote:
Hi Feilong, Bharat, thanks for your answer.
@Feilong, From /etc/kolla/heat-engine/heat.conf I see: [clients_keystone] auth_uri = http://10.1.7.201:5000
This should map into auth_url within the k8s master. Within the k8s master in /etc/os-collect-config.conf I see:
[heat] auth_url = http://10.1.7.201:5000/v3/ : : resource_name = kube-master region_name = null
and from /etc/sysconfig/heat-params (among the others): : REGION_NAME="RegionOne" : AUTH_URL="http://10.1.7.201:5000/v3"
This URL corresponds to the "public" Heat endpoint openstack endpoint list | grep heat | 3d5f58c43f6b44f6b54990d6fd9ff55d | RegionOne | heat | orchestration | True | internal | http://10.1.7.200:8004/v1/%(tenant_id)s | | 8c2492cb0ddc48ca94942a4a299a88dc | RegionOne | heat-cfn | cloudformation | True | internal | http://10.1.7.200:8000/v1 | | b164c4618a784da9ae14da75a6c764a3 | RegionOne | heat | orchestration | True | public | http://10.1.7.201:8004/v1/%(tenant_id)s | | da203f7d337b4587a0f5fc774c993390 | RegionOne | heat | orchestration | True | admin | http://10.1.7.200:8004/v1/%(tenant_id)s | | e0d3743e7c604e5c8aa4684df2d1ce53 | RegionOne | heat-cfn | cloudformation | True | public | http://10.1.7.201:8000/v1 | | efe0b8418aa24dfca33c243e7eed7e90 | RegionOne | heat-cfn | cloudformation | True | admin | http://10.1.7.200:8000/v1 |
Connectivity tests: [fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ ping 10.1.7.201 PING 10.1.7.201 (10.1.7.201) 56(84) bytes of data. 64 bytes from 10.1.7.201: icmp_seq=1 ttl=63 time=0.285 ms
[fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ curl http://10.1.7.201:5000/v3/ {"version": {"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "http://10.1.7.201:5000/v3/", "rel": "self"}]}}
Apparently, I can reach such endpoint from within the k8s master
@Bharat, that file seems to be properly conifugured to me as well. The problem pointed by "systemctl status heat-container-agent" is with:
Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping
Still no way forward from my side.
/Giuseppe
On Tue, 19 Feb 2019 at 22:16, Bharat Kunwar <bharat@stackhpc.com> wrote:
I have the same problem. Weird thing is /etc/sysconfig/heat-params has region_name specified in my case!
Sent from my iPhone
On 19 Feb 2019, at 22:00, Feilong Wang <feilong@catalyst.net.nz> wrote:
Can you talk to the Heat API from your master node?
On 20/02/19 6:43 AM, Giuseppe Sannino wrote:
Hi all...again, I managed to get over the previous issue by "not disabling" the TLS in the cluster template. From the cloud-init-output.log I see: Cloud-init v. 17.1 running 'modules:final' at Tue, 19 Feb 2019 17:03:53 +0000. Up 38.08 seconds. Cloud-init v. 17.1 finished at Tue, 19 Feb 2019 17:13:22 +0000. Datasource DataSourceEc2. Up 607.13 seconds
But the cluster creation keeps on failing. From the journalctl -f I see a possible issue: Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: publicURL endpoint for orchestration service in null region not found Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: Source [heat] Unavailable. Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: /var/lib/os-collect-config/local-data not found. Skipping
anyone familiar with this problem ?
Thanks as usual. /Giuseppe
On Tue, 19 Feb 2019 at 17:35, Giuseppe Sannino < km.giuseppesannino@gmail.com> wrote:
Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled ---------------
VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out"
I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5
Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE)
Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver.
[fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1.
May I ask for an help on this ?
Many thanks /Giuseppe
-- Cheers & Best regards, Feilong Wang (王飞龙)
Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
Ciao Mark, finally it works! Many many thanks! That was the missing piece of the puzzle. Just FYI information, from the systemctl status for the heat-container-agent I can still see this repetitive logs: : Feb 21 08:00:40 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping Feb 21 08:01:11 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping : This doesn't seem to harm the deployment but I will check further. Thanks a lot to everyone! /Giuseppe On Wed, 20 Feb 2019 at 20:16, Mark Goddard <mark@stackhpc.com> wrote:
Hi, I think we've hit this, and John Garbutt has added the following configuration for Kolla Ansible in /etc/kolla/config/heat.conf:
[DEFAULT] region_name_for_services=RegionOne We'll need a patch in kolla ansible to do that without custom config changes. Mark
On Wed, 20 Feb 2019 at 11:05, Bharat Kunwar <bharat@stackhpc.com> wrote:
Hi Giuseppe,
What version of heat are you running?
Can you check if you have this patch merged? https://review.openstack.org/579485
https://review.openstack.org/579485
Bharat
Sent from my iPhone
On 20 Feb 2019, at 10:38, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote:
Hi Feilong, Bharat, thanks for your answer.
@Feilong, From /etc/kolla/heat-engine/heat.conf I see: [clients_keystone] auth_uri = http://10.1.7.201:5000
This should map into auth_url within the k8s master. Within the k8s master in /etc/os-collect-config.conf I see:
[heat] auth_url = http://10.1.7.201:5000/v3/ : : resource_name = kube-master region_name = null
and from /etc/sysconfig/heat-params (among the others): : REGION_NAME="RegionOne" : AUTH_URL="http://10.1.7.201:5000/v3"
This URL corresponds to the "public" Heat endpoint openstack endpoint list | grep heat | 3d5f58c43f6b44f6b54990d6fd9ff55d | RegionOne | heat | orchestration | True | internal | http://10.1.7.200:8004/v1/%(tenant_id)s | | 8c2492cb0ddc48ca94942a4a299a88dc | RegionOne | heat-cfn | cloudformation | True | internal | http://10.1.7.200:8000/v1 | | b164c4618a784da9ae14da75a6c764a3 | RegionOne | heat | orchestration | True | public | http://10.1.7.201:8004/v1/%(tenant_id)s | | da203f7d337b4587a0f5fc774c993390 | RegionOne | heat | orchestration | True | admin | http://10.1.7.200:8004/v1/%(tenant_id)s | | e0d3743e7c604e5c8aa4684df2d1ce53 | RegionOne | heat-cfn | cloudformation | True | public | http://10.1.7.201:8000/v1 | | efe0b8418aa24dfca33c243e7eed7e90 | RegionOne | heat-cfn | cloudformation | True | admin | http://10.1.7.200:8000/v1 |
Connectivity tests: [fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ ping 10.1.7.201 PING 10.1.7.201 (10.1.7.201) 56(84) bytes of data. 64 bytes from 10.1.7.201: icmp_seq=1 ttl=63 time=0.285 ms
[fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ curl http://10.1.7.201:5000/v3/ {"version": {"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "http://10.1.7.201:5000/v3/", "rel": "self"}]}}
Apparently, I can reach such endpoint from within the k8s master
@Bharat, that file seems to be properly conifugured to me as well. The problem pointed by "systemctl status heat-container-agent" is with:
Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping
Still no way forward from my side.
/Giuseppe
On Tue, 19 Feb 2019 at 22:16, Bharat Kunwar <bharat@stackhpc.com> wrote:
I have the same problem. Weird thing is /etc/sysconfig/heat-params has region_name specified in my case!
Sent from my iPhone
On 19 Feb 2019, at 22:00, Feilong Wang <feilong@catalyst.net.nz> wrote:
Can you talk to the Heat API from your master node?
On 20/02/19 6:43 AM, Giuseppe Sannino wrote:
Hi all...again, I managed to get over the previous issue by "not disabling" the TLS in the cluster template. From the cloud-init-output.log I see: Cloud-init v. 17.1 running 'modules:final' at Tue, 19 Feb 2019 17:03:53 +0000. Up 38.08 seconds. Cloud-init v. 17.1 finished at Tue, 19 Feb 2019 17:13:22 +0000. Datasource DataSourceEc2. Up 607.13 seconds
But the cluster creation keeps on failing. From the journalctl -f I see a possible issue: Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: publicURL endpoint for orchestration service in null region not found Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: Source [heat] Unavailable. Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: /var/lib/os-collect-config/local-data not found. Skipping
anyone familiar with this problem ?
Thanks as usual. /Giuseppe
On Tue, 19 Feb 2019 at 17:35, Giuseppe Sannino < km.giuseppesannino@gmail.com> wrote:
Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled ---------------
VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out"
I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5
Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE)
Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver.
[fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1.
May I ask for an help on this ?
Many thanks /Giuseppe
-- Cheers & Best regards, Feilong Wang (王飞龙)
Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
Yes I’ve seen those messages too, I think it’s normal so wouldn’t worry too much. Glad this is sorted! Sent from my iPhone
On 21 Feb 2019, at 11:03, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote:
Ciao Mark, finally it works! Many many thanks! That was the missing piece of the puzzle.
Just FYI information, from the systemctl status for the heat-container-agent I can still see this repetitive logs: : Feb 21 08:00:40 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping Feb 21 08:01:11 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping :
This doesn't seem to harm the deployment but I will check further.
Thanks a lot to everyone!
/Giuseppe
On Wed, 20 Feb 2019 at 20:16, Mark Goddard <mark@stackhpc.com> wrote: Hi, I think we've hit this, and John Garbutt has added the following configuration for Kolla Ansible in /etc/kolla/config/heat.conf:
[DEFAULT] region_name_for_services=RegionOne
We'll need a patch in kolla ansible to do that without custom config changes. Mark
On Wed, 20 Feb 2019 at 11:05, Bharat Kunwar <bharat@stackhpc.com> wrote: Hi Giuseppe,
What version of heat are you running?
Can you check if you have this patch merged? https://review.openstack.org/579485
https://review.openstack.org/579485
Bharat
Sent from my iPhone
On 20 Feb 2019, at 10:38, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote:
Hi Feilong, Bharat, thanks for your answer.
@Feilong, From /etc/kolla/heat-engine/heat.conf I see: [clients_keystone] auth_uri = http://10.1.7.201:5000
This should map into auth_url within the k8s master. Within the k8s master in /etc/os-collect-config.conf I see:
[heat] auth_url = http://10.1.7.201:5000/v3/ : : resource_name = kube-master region_name = null
and from /etc/sysconfig/heat-params (among the others): : REGION_NAME="RegionOne" : AUTH_URL="http://10.1.7.201:5000/v3"
This URL corresponds to the "public" Heat endpoint openstack endpoint list | grep heat | 3d5f58c43f6b44f6b54990d6fd9ff55d | RegionOne | heat | orchestration | True | internal | http://10.1.7.200:8004/v1/%(tenant_id)s | | 8c2492cb0ddc48ca94942a4a299a88dc | RegionOne | heat-cfn | cloudformation | True | internal | http://10.1.7.200:8000/v1 | | b164c4618a784da9ae14da75a6c764a3 | RegionOne | heat | orchestration | True | public | http://10.1.7.201:8004/v1/%(tenant_id)s | | da203f7d337b4587a0f5fc774c993390 | RegionOne | heat | orchestration | True | admin | http://10.1.7.200:8004/v1/%(tenant_id)s | | e0d3743e7c604e5c8aa4684df2d1ce53 | RegionOne | heat-cfn | cloudformation | True | public | http://10.1.7.201:8000/v1 | | efe0b8418aa24dfca33c243e7eed7e90 | RegionOne | heat-cfn | cloudformation | True | admin | http://10.1.7.200:8000/v1 |
Connectivity tests: [fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ ping 10.1.7.201 PING 10.1.7.201 (10.1.7.201) 56(84) bytes of data. 64 bytes from 10.1.7.201: icmp_seq=1 ttl=63 time=0.285 ms
[fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ curl http://10.1.7.201:5000/v3/ {"version": {"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "http://10.1.7.201:5000/v3/", "rel": "self"}]}}
Apparently, I can reach such endpoint from within the k8s master
@Bharat, that file seems to be properly conifugured to me as well. The problem pointed by "systemctl status heat-container-agent" is with:
Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping
Still no way forward from my side.
/Giuseppe
On Tue, 19 Feb 2019 at 22:16, Bharat Kunwar <bharat@stackhpc.com> wrote: I have the same problem. Weird thing is /etc/sysconfig/heat-params has region_name specified in my case!
Sent from my iPhone
On 19 Feb 2019, at 22:00, Feilong Wang <feilong@catalyst.net.nz> wrote:
Can you talk to the Heat API from your master node?
> On 20/02/19 6:43 AM, Giuseppe Sannino wrote: > Hi all...again, > I managed to get over the previous issue by "not disabling" the TLS in the cluster template. > From the cloud-init-output.log I see: > Cloud-init v. 17.1 running 'modules:final' at Tue, 19 Feb 2019 17:03:53 +0000. Up 38.08 seconds. > Cloud-init v. 17.1 finished at Tue, 19 Feb 2019 17:13:22 +0000. Datasource DataSourceEc2. Up 607.13 seconds > > But the cluster creation keeps on failing. > From the journalctl -f I see a possible issue: > Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: publicURL endpoint for orchestration service in null region not found > Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: Source [heat] Unavailable. > Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: /var/lib/os-collect-config/local-data not found. Skipping > > anyone familiar with this problem ? > > Thanks as usual. > /Giuseppe > > > > > > > >> On Tue, 19 Feb 2019 at 17:35, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote: >> Hi all, >> need an help. >> I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: >> --------------- >> kolla-ansible: 7.0.1 >> openstack_release: Rocky >> kolla_base_distro: centos >> kolla_install_type: source >> TLS: disabled >> --------------- >> >> >> VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out" >> >> I managed to log into Kuber Master and from the cloud-init-output.log I can see: >> + echo 'Waiting for Kubernetes API...' >> Waiting for Kubernetes API... >> ++ curl --silent http://127.0.0.1:8080/healthz >> + '[' ok = '' ']' >> + sleep 5 >> >> >> Checking via systemctl and journalctl I see: >> [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver >> ● kube-apiserver.service - kubernetes-apiserver >> Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) >> Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago >> Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) >> Main PID: 3796 (code=exited, status=1/FAILURE) >> >> Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE >> Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. >> Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. >> Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. >> Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. >> Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. >> Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. >> Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver. >> >> [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver >> -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- >> Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. >> Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. >> Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. >> Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied >> : >> : >> : >> Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied >> Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE >> Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. >> Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. >> Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1. >> >> >> May I ask for an help on this ? >> >> Many thanks >> /Giuseppe >> >> >> >> -- Cheers & Best regards, Feilong Wang (王飞龙) -------------------------------------------------------------------------- Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
On Thu, 21 Feb 2019 at 10:03, Giuseppe Sannino <km.giuseppesannino@gmail.com> wrote:
Ciao Mark, finally it works! Many many thanks! That was the missing piece of the puzzle.
Just FYI information, from the systemctl status for the heat-container-agent I can still see this repetitive logs: : Feb 21 08:00:40 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping Feb 21 08:01:11 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping :
This doesn't seem to harm the deployment but I will check further.
Thanks a lot to everyone!
/Giuseppe
Glad to hear it worked for you. I've raised a bug [1] and proposed a fix [2] in kolla ansible. Mark [1] https://bugs.launchpad.net/kolla-ansible/+bug/1817051 [2] https://review.openstack.org/638400
Sounds great!! Thanks again ! /Giuseppe On Thu, 21 Feb 2019 at 12:36, Mark Goddard <mark@stackhpc.com> wrote:
On Thu, 21 Feb 2019 at 10:03, Giuseppe Sannino < km.giuseppesannino@gmail.com> wrote:
Ciao Mark, finally it works! Many many thanks! That was the missing piece of the puzzle.
Just FYI information, from the systemctl status for the heat-container-agent I can still see this repetitive logs: : Feb 21 08:00:40 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping Feb 21 08:01:11 kube-cluster-goddard-lq54faeabuhu-master-0.novalocal runc[2715]: /var/lib/os-collect-config/local-data not found. Skipping :
This doesn't seem to harm the deployment but I will check further.
Thanks a lot to everyone!
/Giuseppe
Glad to hear it worked for you. I've raised a bug [1] and proposed a fix [2] in kolla ansible. Mark
[1] https://bugs.launchpad.net/kolla-ansible/+bug/1817051 [2] https://review.openstack.org/638400
On 20/02/19 2:15 PM, Mark Goddard wrote:
Hi, I think we've hit this, and John Garbutt has added the following configuration for Kolla Ansible in /etc/kolla/config/heat.conf:
[DEFAULT] region_name_for_services=RegionOne
We'll need a patch in kolla ansible to do that without custom config changes. Mark
On Wed, 20 Feb 2019 at 11:05, Bharat Kunwar <bharat@stackhpc.com <mailto:bharat@stackhpc.com>> wrote:
Hi Giuseppe,
What version of heat are you running?
Can you check if you have this patch merged? https://review.openstack.org/579485
This patch caused a regression (in combination with corresponding patches to os-collect-config and heat-agents) due to weird things that happen in os-apply-config (https://bugs.launchpad.net/os-apply-config/+bug/1830967). Details are here: https://storyboard.openstack.org/#!/story/2005797 I've proposed a fix, and once that merges the workaround suggested above will no longer be needed. (Although setting the region name explicitly is a Good Thing to do anyway.) cheers, Zane.
Bharat
Sent from my iPhone
On 20 Feb 2019, at 10:38, Giuseppe Sannino <km.giuseppesannino@gmail.com <mailto:km.giuseppesannino@gmail.com>> wrote:
Hi Feilong, Bharat, thanks for your answer.
@Feilong, From /etc/kolla/heat-engine/heat.conf I see: [clients_keystone] auth_uri = http://10.1.7.201:5000
This should map into auth_url within the k8s master. Within the k8s master in /etc/os-collect-config.conf I see:
[heat] auth_url = http://10.1.7.201:5000/v3/ : : resource_name = kube-master region_name = null
and from /etc/sysconfig/heat-params (among the others): : REGION_NAME="RegionOne" : AUTH_URL="http://10.1.7.201:5000/v3"
This URL corresponds to the "public" Heat endpoint openstack endpoint list | grep heat | 3d5f58c43f6b44f6b54990d6fd9ff55d | RegionOne | heat | orchestration | True | internal | http://10.1.7.200:8004/v1/%(tenant_id)s | | 8c2492cb0ddc48ca94942a4a299a88dc | RegionOne | heat-cfn | cloudformation | True | internal | http://10.1.7.200:8000/v1 | | b164c4618a784da9ae14da75a6c764a3 | RegionOne | heat | orchestration | True | public | http://10.1.7.201:8004/v1/%(tenant_id)s | | da203f7d337b4587a0f5fc774c993390 | RegionOne | heat | orchestration | True | admin | http://10.1.7.200:8004/v1/%(tenant_id)s | | e0d3743e7c604e5c8aa4684df2d1ce53 | RegionOne | heat-cfn | cloudformation | True | public | http://10.1.7.201:8000/v1 | | efe0b8418aa24dfca33c243e7eed7e90 | RegionOne | heat-cfn | cloudformation | True | admin | http://10.1.7.200:8000/v1 |
Connectivity tests: [fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ ping 10.1.7.201 PING 10.1.7.201 (10.1.7.201) 56(84) bytes of data. 64 bytes from 10.1.7.201 <http://10.1.7.201>: icmp_seq=1 ttl=63 time=0.285 ms
[fedora@kube-cluster-fed27-k5di3i7stgks-master-0 ~]$ curl http://10.1.7.201:5000/v3/ {"version": {"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "http://10.1.7.201:5000/v3/", "rel": "self"}]}}
Apparently, I can reach such endpoint from within the k8s master
@Bharat, that file seems to be properly conifugured to me as well. The problem pointed by "systemctl status heat-container-agent" is with:
Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:23 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: publicURL endpoint for orchestration service in null region not found Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: Source [heat] Unavailable. Feb 20 09:33:53 kube-cluster-fed27-k5di3i7stgks-master-0.novalocal runc[2837]: /var/lib/os-collect-config/local-data not found. Skipping
Still no way forward from my side.
/Giuseppe
On Tue, 19 Feb 2019 at 22:16, Bharat Kunwar <bharat@stackhpc.com <mailto:bharat@stackhpc.com>> wrote:
I have the same problem. Weird thing is /etc/sysconfig/heat-params has region_name specified in my case!
Sent from my iPhone
On 19 Feb 2019, at 22:00, Feilong Wang <feilong@catalyst.net.nz <mailto:feilong@catalyst.net.nz>> wrote:
Can you talk to the Heat API from your master node?
On 20/02/19 6:43 AM, Giuseppe Sannino wrote:
Hi all...again, I managed to get over the previous issue by "not disabling" the TLS in the cluster template. From the cloud-init-output.log I see: Cloud-init v. 17.1 running 'modules:final' at Tue, 19 Feb 2019 17:03:53 +0000. Up 38.08 seconds. Cloud-init v. 17.1 finished at Tue, 19 Feb 2019 17:13:22 +0000. Datasource DataSourceEc2. Up 607.13 seconds
But the cluster creation keeps on failing. From the journalctl -f I see a possible issue: Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: publicURL endpoint for orchestration service in null region not found Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: Source [heat] Unavailable. Feb 19 17:42:38 kube-cluster-tls-6hezqcq4ien3-master-0.novalocal runc[2723]: /var/lib/os-collect-config/local-data not found. Skipping
anyone familiar with this problem ?
Thanks as usual. /Giuseppe
On Tue, 19 Feb 2019 at 17:35, Giuseppe Sannino <km.giuseppesannino@gmail.com <mailto:km.giuseppesannino@gmail.com>> wrote:
Hi all, need an help. I deployed an AIO via Kolla on a baremetal node. Here some information about the deployment: --------------- kolla-ansible: 7.0.1 openstack_release: Rocky kolla_base_distro: centos kolla_install_type: source TLS: disabled ---------------
VMs spawn without issue but I can't make the "Kubernetes cluster creation" successfully. It fails due to "Time out"
I managed to log into Kuber Master and from the cloud-init-output.log I can see: + echo 'Waiting for Kubernetes API...' Waiting for Kubernetes API... ++ curl --silent http://127.0.0.1:8080/healthz + '[' ok = '' ']' + sleep 5
Checking via systemctl and journalctl I see: [fedora@kube-clsuter-qamdealetlbi-master-0 log]$ systemctl status kube-apiserver ● kube-apiserver.service - kubernetes-apiserver Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-02-19 15:31:41 UTC; 45min ago Process: 3796 ExecStart=/usr/bin/runc --systemd-cgroup run kube-apiserver (code=exited, status=1/FAILURE) Main PID: 3796 (code=exited, status=1/FAILURE)
Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:40 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 6. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Stopped kubernetes-apiserver. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Start request repeated too quickly. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:41 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Failed to start kubernetes-apiserver.
[fedora@kube-clsuter-qamdealetlbi-master-0 log]$ sudo journalctl -u kube-apiserver -- Logs begin at Tue 2019-02-19 15:21:36 UTC, end at Tue 2019-02-19 16:17:00 UTC. -- Feb 19 15:31:33 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: Started kubernetes-apiserver. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version. Feb 19 15:31:34 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Flag --insecure-port has been deprecated, This flag will be removed in a future version. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: Error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied : : : Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal runc[2794]: error: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: permission denied Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Main process exited, code=exited, status=1/FAILURE Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Failed with result 'exit-code'. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Service RestartSec=100ms expired, scheduling restart. Feb 19 15:31:35 kube-clsuter-qamdealetlbi-master-0.novalocal systemd[1]: kube-apiserver.service: Scheduled restart job, restart counter is at 1.
May I ask for an help on this ?
Many thanks /Giuseppe
-- Cheers & Best regards, Feilong Wang (王飞龙) -------------------------------------------------------------------------- Senior Cloud Software Engineer Tel: +64-48032246 Email:flwang@catalyst.net.nz <mailto:flwang@catalyst.net.nz> Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
participants (5)
-
Bharat Kunwar
-
Feilong Wang
-
Giuseppe Sannino
-
Mark Goddard
-
Zane Bitter