[kuryr] [kuryr-kubernetes] Unable to use kubernetes-related things after rebooting host
Hello kuryr-kubernetes team, currently, I deployed openstack with devstack, and I added "kuryr-kubernetes" to use kubernetes vim. At first it worked fine and no error was occurred, so I made some c-vnfs with kuryr-kubernetes. But after rebooting the host, the systems that worked well before are not working normally at all. First, I deployed an example one ('redis') with kubectl command, (Not using Tacker, but only using kubectl cli) it showed below error, Events: │obeTime":null,"lastTransitionTime":"2019-04-15T02:28:31Z"}],"hostIP":"192.168.1.20","startTime":"2019-04-15T02:28:31Z" Type Reason Age From Message │,"containerStatuses":[{"name":"web-server","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":f ---- ------ ---- ---- ------- │alse,"restartCount":0,"image":"kyleoh95/iistrc-k8s-demo2:0.4","imageID":""}],"qosClass":"Guaranteed"}}]} Normal Scheduled 2m25s default-scheduler Successfully assigned default/redis-master-6fbbc44567-ms8n│Apr 15 11:36:52 master tacker-server[1881]: from (pid=1881) request /usr/local/lib/python2.7/dist-packages/kubernetes k to master │/client/rest.py:219 Warning MissingClusterDNS 2m25s kubelet, master pod: "redis-master-6fbbc44567-ms8nk_default(8a6f9079-5f27-│Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.633 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne 11e9-9b42-a4bf01550f1a)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" pol│tes_driver [-] status: Pending from (pid=1881) create_wait /opt/stack/tacker/tacker/vnfm/infra_drivers/kubernetes/kube icy. Falling back to "Default" policy. │rnetes_driver.py:137 Warning FailedCreatePodSandBox 19s kubelet, master Failed create pod sandbox: rpc error: code = Unknown desc │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne = failed to set up sandbox container "310e18b859e118ad6abcaa60d29a9e332da516ce57b8cdf47456ea4882ca66f5" network for pod│tes_driver [-] VNF initializing status: ['default', 'svc-vdu1-b9e851'] Pending from (pid=1881) create_wait /opt/stack/ "redis-master-6fbbc44567-ms8nk": NetworkPlugin cni failed to set up pod "redis-master-6fbbc44567-ms8nk_default" networ│tacker/tacker/vnfm/infra_drivers/kubernetes/kubernetes_driver.py:141 k: Got invalid status code from CNI daemon.; Traceback (most recent call last): │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 WARNING tacker.vnfm.infra_drivers.kubernetes.kuber File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py", line 80, in run │netes_driver [-] VNF Creation failed: Resource creation is not completed within 500 seconds as creation of stack defau vif = self._add(params) Are there any reason for this error? And how can I resolve this problem? Best Regards, Jaewook. ================================================ *Jaewook Oh* (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618 E-mail : jwoh95@dcn.ssu.ac.kr ================================================
On Mon, 2019-04-15 at 11:44 +0900, Jaewook Oh wrote:
Hello kuryr-kubernetes team,
currently, I deployed openstack with devstack, and I added "kuryr-kubernetes" to use kubernetes vim. At first it worked fine and no error was occurred, so I made some c-vnfs with kuryr-kubernetes. But after rebooting the host, the systems that worked well before are not working normally at all.
First, I deployed an example one ('redis') with kubectl command, (Not using Tacker, but only using kubectl cli) it showed below error,
Events: │obeTime":null,"lastTransitionTime":"2019-04-15T02:28:31Z"}],"hostIP":"192.168.1.20","startTime":"2019-04-15T02:28:31Z" Type Reason Age From Message │,"containerStatuses":[{"name":"web-server","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":f ---- ------ ---- ---- ------- │alse,"restartCount":0,"image":"kyleoh95/iistrc-k8s-demo2:0.4","imageID":""}],"qosClass":"Guaranteed"}}]} Normal Scheduled 2m25s default-scheduler Successfully assigned default/redis-master-6fbbc44567-ms8n│Apr 15 11:36:52 master tacker-server[1881]: from (pid=1881) request /usr/local/lib/python2.7/dist-packages/kubernetes k to master │/client/rest.py:219 Warning MissingClusterDNS 2m25s kubelet, master pod: "redis-master-6fbbc44567-ms8nk_default(8a6f9079-5f27-│Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.633 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne 11e9-9b42-a4bf01550f1a)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" pol│tes_driver [-] status: Pending from (pid=1881) create_wait /opt/stack/tacker/tacker/vnfm/infra_drivers/kubernetes/kube icy. Falling back to "Default" policy. │rnetes_driver.py:137 Warning FailedCreatePodSandBox 19s kubelet, master Failed create pod sandbox: rpc error: code = Unknown desc │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne = failed to set up sandbox container "310e18b859e118ad6abcaa60d29a9e332da516ce57b8cdf47456ea4882ca66f5" network for pod│tes_driver [-] VNF initializing status: ['default', 'svc-vdu1-b9e851'] Pending from (pid=1881) create_wait /opt/stack/ "redis-master-6fbbc44567-ms8nk": NetworkPlugin cni failed to set up pod "redis-master-6fbbc44567-ms8nk_default" networ│tacker/tacker/vnfm/infra_drivers/kubernetes/kubernetes_driver.py:141 k: Got invalid status code from CNI daemon.; Traceback (most recent call last): │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 WARNING tacker.vnfm.infra_drivers.kubernetes.kuber File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py", line 80, in run │netes_driver [-] VNF Creation failed: Resource creation is not completed within 500 seconds as creation of stack defau vif = self._add(params)
"Got invalid status code from CNI daemon" means that you should take a look at kuryr-cni pods logs.
Are there any reason for this error? And how can I resolve this problem?
In general I'm pretty sure DevStack is not designed to be rebootable, so it's probably some ip route being gone or some service that haven't restarted. If you really want to debug this instead of firing a new DevStack, start from checking all "devstack@*" services in systemd.
Best Regards, Jaewook. ================================================ Jaewook Oh (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618 E-mail : jwoh95@dcn.ssu.ac.kr ================================================
Overall and besides of what Michal said, yeah, openstack hasn't been designed to be rebootable. A looong time ago and before the migration to systemd services there was a script that restarted the screen instances [1] That said, not there since 2016 so please restack whenever you reboot. In any case, was there any reason for rebooting? I'd check if just restarting service would be enough for you. Thanks! [1] https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=9ba49cd8263... On Mon, Apr 15, 2019 at 9:58 AM Michał Dulko <mdulko@redhat.com> wrote:
On Mon, 2019-04-15 at 11:44 +0900, Jaewook Oh wrote:
Hello kuryr-kubernetes team,
currently, I deployed openstack with devstack, and I added "kuryr-kubernetes" to use kubernetes vim. At first it worked fine and no error was occurred, so I made some c-vnfs with kuryr-kubernetes. But after rebooting the host, the systems that worked well before are not working normally at all.
First, I deployed an example one ('redis') with kubectl command, (Not using Tacker, but only using kubectl cli) it showed below error,
Events:
│obeTime":null,"lastTransitionTime":"2019-04-15T02:28:31Z"}],"hostIP":"192.168.1.20","startTime":"2019-04-15T02:28:31Z"
Type Reason Age From Message
│,"containerStatuses":[{"name":"web-server","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":f
---- ------ ---- ---- -------
│alse,"restartCount":0,"image":"kyleoh95/iistrc-k8s-demo2:0.4","imageID":""}],"qosClass":"Guaranteed"}}]}
Normal Scheduled 2m25s default-scheduler Successfully assigned default/redis-master-6fbbc44567-ms8n│Apr 15 11:36:52 master tacker-server[1881]: from (pid=1881) request /usr/local/lib/python2.7/dist-packages/kubernetes k to master │/client/rest.py:219 Warning MissingClusterDNS 2m25s kubelet, master pod: "redis-master-6fbbc44567-ms8nk_default(8a6f9079-5f27-│Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.633 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne 11e9-9b42-a4bf01550f1a)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" pol│tes_driver [-] status: Pending from (pid=1881) create_wait /opt/stack/tacker/tacker/vnfm/infra_drivers/kubernetes/kube icy. Falling back to "Default" policy. │rnetes_driver.py:137 Warning FailedCreatePodSandBox 19s kubelet, master Failed create pod sandbox: rpc error: code = Unknown desc │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne = failed to set up sandbox container "310e18b859e118ad6abcaa60d29a9e332da516ce57b8cdf47456ea4882ca66f5" network for pod│tes_driver [-] VNF initializing status: ['default', 'svc-vdu1-b9e851'] Pending from (pid=1881) create_wait /opt/stack/ "redis-master-6fbbc44567-ms8nk": NetworkPlugin cni failed to set up pod "redis-master-6fbbc44567-ms8nk_default" networ│tacker/tacker/vnfm/infra_drivers/kubernetes/kubernetes_driver.py:141 k: Got invalid status code from CNI daemon.; Traceback (most recent call last): │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 WARNING tacker.vnfm.infra_drivers.kubernetes.kuber File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py", line 80, in run │netes_driver [-] VNF Creation failed: Resource creation is not completed within 500 seconds as creation of stack defau vif = self._add(params)
"Got invalid status code from CNI daemon" means that you should take a look at kuryr-cni pods logs.
Are there any reason for this error? And how can I resolve this problem?
In general I'm pretty sure DevStack is not designed to be rebootable, so it's probably some ip route being gone or some service that haven't restarted. If you really want to debug this instead of firing a new DevStack, start from checking all "devstack@*" services in systemd.
Best Regards, Jaewook. ================================================ Jaewook Oh (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618 E-mail : jwoh95@dcn.ssu.ac.kr ================================================
On 2019-04-15 10:18:46 +0200 (+0200), Daniel Mellado Area wrote:
Overall and besides of what Michal said, yeah, openstack hasn't been designed to be rebootable. [...]
To be clear, it's DevStack which is designed not to be rebootable (this is intentional, to further dissuade anyone from trying to use it for a production deployment). There is nothing inherently non-rebootable about OpenStack in general. -- Jeremy Stanley
On 15/4/19 14:30, Jeremy Stanley wrote:
On 2019-04-15 10:18:46 +0200 (+0200), Daniel Mellado Area wrote:
Overall and besides of what Michal said, yeah, openstack hasn't been designed to be rebootable. [...]
To be clear, it's DevStack which is designed not to be rebootable (this is intentional, to further dissuade anyone from trying to use it for a production deployment). There is nothing inherently non-rebootable about OpenStack in general.
My bad on the typo, I meant DevStack when I was typing OpenStack, heh
Thanks for all replying! I thought Devstack was rebootable, cause when I didn't use any kuryr things, the devstack was working normally haha... Anyway I got it :) Thanks again for the information! BR, Jaewook. ================================================ *Jaewook Oh* (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618 E-mail : jwoh95@dcn.ssu.ac.kr ================================================ 2019년 4월 15일 (월) 오후 9:49, Daniel Mellado <dmellado@redhat.com>님이 작성:
On 15/4/19 14:30, Jeremy Stanley wrote:
On 2019-04-15 10:18:46 +0200 (+0200), Daniel Mellado Area wrote:
Overall and besides of what Michal said, yeah, openstack hasn't been designed to be rebootable. [...]
To be clear, it's DevStack which is designed not to be rebootable (this is intentional, to further dissuade anyone from trying to use it for a production deployment). There is nothing inherently non-rebootable about OpenStack in general.
My bad on the typo, I meant DevStack when I was typing OpenStack, heh
My server that I deployed devstack was halted suddenly. I don't know what exact reason is but, that made rebooting devstack. Thanks! ================================================ *Jaewook Oh* (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618 E-mail : jwoh95@dcn.ssu.ac.kr ================================================ 2019년 4월 15일 (월) 오후 5:23, Daniel Mellado Area <dmellado@redhat.com>님이 작성:
Overall and besides of what Michal said, yeah, openstack hasn't been designed to be rebootable. A looong time ago and before the migration to systemd services there was a script that restarted the screen instances [1]
That said, not there since 2016 so please restack whenever you reboot. In any case, was there any reason for rebooting? I'd check if just restarting service would be enough for you.
Thanks!
[1] https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=9ba49cd8263...
On Mon, Apr 15, 2019 at 9:58 AM Michał Dulko <mdulko@redhat.com> wrote:
On Mon, 2019-04-15 at 11:44 +0900, Jaewook Oh wrote:
Hello kuryr-kubernetes team,
currently, I deployed openstack with devstack, and I added "kuryr-kubernetes" to use kubernetes vim. At first it worked fine and no error was occurred, so I made some c-vnfs with kuryr-kubernetes. But after rebooting the host, the systems that worked well before are not working normally at all.
First, I deployed an example one ('redis') with kubectl command, (Not using Tacker, but only using kubectl cli) it showed below error,
Events:
│obeTime":null,"lastTransitionTime":"2019-04-15T02:28:31Z"}],"hostIP":"192.168.1.20","startTime":"2019-04-15T02:28:31Z"
Type Reason Age From Message
│,"containerStatuses":[{"name":"web-server","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":f
---- ------ ---- ---- -------
Normal Scheduled 2m25s default-scheduler Successfully assigned default/redis-master-6fbbc44567-ms8n│Apr 15 11:36:52 master tacker-server[1881]: from (pid=1881) request /usr/local/lib/python2.7/dist-packages/kubernetes k to master │/client/rest.py:219 Warning MissingClusterDNS 2m25s kubelet, master pod: "redis-master-6fbbc44567-ms8nk_default(8a6f9079-5f27-│Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.633 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne 11e9-9b42-a4bf01550f1a)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" pol│tes_driver [-] status: Pending from (pid=1881) create_wait /opt/stack/tacker/tacker/vnfm/infra_drivers/kubernetes/kube icy. Falling back to "Default" policy. │rnetes_driver.py:137 Warning FailedCreatePodSandBox 19s kubelet, master Failed create pod sandbox: rpc error: code = Unknown desc │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne = failed to set up sandbox container "310e18b859e118ad6abcaa60d29a9e332da516ce57b8cdf47456ea4882ca66f5" network for pod│tes_driver [-] VNF initializing status: ['default', 'svc-vdu1-b9e851'] Pending from (pid=1881) create_wait /opt/stack/ "redis-master-6fbbc44567-ms8nk": NetworkPlugin cni failed to set up
│alse,"restartCount":0,"image":"kyleoh95/iistrc-k8s-demo2:0.4","imageID":""}],"qosClass":"Guaranteed"}}]} pod "redis-master-6fbbc44567-ms8nk_default" networ│tacker/tacker/vnfm/infra_drivers/kubernetes/kubernetes_driver.py:141
k: Got invalid status code from CNI daemon.; Traceback (most recent call last): │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 WARNING tacker.vnfm.infra_drivers.kubernetes.kuber File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py", line 80, in run │netes_driver [-] VNF Creation failed: Resource creation is not completed within 500 seconds as creation of stack defau vif = self._add(params)
"Got invalid status code from CNI daemon" means that you should take a look at kuryr-cni pods logs.
Are there any reason for this error? And how can I resolve this problem?
In general I'm pretty sure DevStack is not designed to be rebootable, so it's probably some ip route being gone or some service that haven't restarted. If you really want to debug this instead of firing a new DevStack, start from checking all "devstack@*" services in systemd.
Best Regards, Jaewook. ================================================ Jaewook Oh (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618 E-mail : jwoh95@dcn.ssu.ac.kr ================================================
participants (5)
-
Daniel Mellado
-
Daniel Mellado Area
-
Jaewook Oh
-
Jeremy Stanley
-
Michał Dulko