Open Stack

On Mon, 2019-04-15 at 11:44 +0900, Jaewook Oh wrote:
> Hello kuryr-kubernetes team,
> 
> currently, I deployed openstack with devstack, and I added "kuryr-kubernetes" to use kubernetes vim.
> At first it worked fine and no error was occurred,
> so I made some c-vnfs with kuryr-kubernetes.
> But after rebooting the host, the systems that worked well before are not working normally at all.
> 
> First, I deployed an example one ('redis') with kubectl command, (Not using Tacker, but only using kubectl cli)
> it showed below error,
> 
> Events:                                                                                                                │obeTime":null,"lastTransitionTime":"2019-04-15T02:28:31Z"}],"hostIP":"192.168.1.20","startTime":"2019-04-15T02:28:31Z"
>   Type     Reason                  Age    From               Message                                                   │,"containerStatuses":[{"name":"web-server","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":f
>   ----     ------                  ----   ----               -------                                                   │alse,"restartCount":0,"image":"kyleoh95/iistrc-k8s-demo2:0.4","imageID":""}],"qosClass":"Guaranteed"}}]}
>   Normal   Scheduled               2m25s  default-scheduler  Successfully assigned default/redis-master-6fbbc44567-ms8n│Apr 15 11:36:52 master tacker-server[1881]:  from (pid=1881) request /usr/local/lib/python2.7/dist-packages/kubernetes
> k to master                                                                                                            │/client/rest.py:219
>   Warning  MissingClusterDNS       2m25s  kubelet, master    pod: "redis-master-6fbbc44567-ms8nk_default(8a6f9079-5f27-│Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.633 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne
> 11e9-9b42-a4bf01550f1a)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" pol│tes_driver [-] status: Pending from (pid=1881) create_wait /opt/stack/tacker/tacker/vnfm/infra_drivers/kubernetes/kube
> icy. Falling back to "Default" policy.                                                                                 │rnetes_driver.py:137
>   Warning  FailedCreatePodSandBox  19s    kubelet, master    Failed create pod sandbox: rpc error: code = Unknown desc │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 DEBUG tacker.vnfm.infra_drivers.kubernetes.kuberne
> = failed to set up sandbox container "310e18b859e118ad6abcaa60d29a9e332da516ce57b8cdf47456ea4882ca66f5" network for pod│tes_driver [-] VNF initializing status: ['default', 'svc-vdu1-b9e851'] Pending from (pid=1881) create_wait /opt/stack/
>  "redis-master-6fbbc44567-ms8nk": NetworkPlugin cni failed to set up pod "redis-master-6fbbc44567-ms8nk_default" networ│tacker/tacker/vnfm/infra_drivers/kubernetes/kubernetes_driver.py:141
> k: Got invalid status code from CNI daemon.; Traceback (most recent call last):                                        │Apr 15 11:36:52 master tacker-server[1881]: 2019-04-15 11:36:52.634 WARNING tacker.vnfm.infra_drivers.kubernetes.kuber
>   File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py", line 80, in run                                      │netes_driver [-] VNF Creation failed: Resource creation is not completed within 500 seconds as creation of stack defau
>     vif = self._add(params)

"Got invalid status code from CNI daemon" means that you should take a
look at kuryr-cni pods logs.

> 
> Are there any reason for this error? And how can I resolve this problem?

In general I'm pretty sure DevStack is not designed to be rebootable,
so it's probably some ip route being gone or some service that haven't
restarted. If you really want to debug this instead of firing a new
DevStack, start from checking all "devstack@*" services in systemd.

> Best Regards,
> Jaewook.
> ================================================
> Jaewook Oh (오재욱)
> IISTRC - Internet Infra System Technology Research Center
> 369 Sangdo-ro, Dongjak-gu,
> 06978, Seoul, Republic of Korea
> Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618
> E-mail : jwoh95 at dcn.ssu.ac.kr
> ================================================