[kuryr] [kuryr-kubernetes] Unable to use kubernetes-related things after rebooting host

Daniel Mellado Area dmellado at redhat.com
Mon Apr 15 08:18:46 UTC 2019


Overall and besides of what Michal said, yeah, openstack hasn't been
designed to be rebootable.
A looong time ago and before the migration to systemd services there was a
script that restarted the screen instances [1]

That said, not there since 2016 so please restack whenever you reboot. In
any case, was there any reason for rebooting?
I'd check if just restarting service would be enough for you.

Thanks!

[1]
https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=9ba49cd82631ada84b8458b370ac535985d57725

On Mon, Apr 15, 2019 at 9:58 AM Michał Dulko <mdulko at redhat.com> wrote:

> On Mon, 2019-04-15 at 11:44 +0900, Jaewook Oh wrote:
> > Hello kuryr-kubernetes team,
> >
> > currently, I deployed openstack with devstack, and I added
> "kuryr-kubernetes" to use kubernetes vim.
> > At first it worked fine and no error was occurred,
> > so I made some c-vnfs with kuryr-kubernetes.
> > But after rebooting the host, the systems that worked well before are
> not working normally at all.
> >
> > First, I deployed an example one ('redis') with kubectl command, (Not
> using Tacker, but only using kubectl cli)
> > it showed below error,
> >
> > Events:
>
> │obeTime":null,"lastTransitionTime":"2019-04-15T02:28:31Z"}],"hostIP":"192.168.1.20","startTime":"2019-04-15T02:28:31Z"
> >   Type     Reason                  Age    From               Message
>
>  │,"containerStatuses":[{"name":"web-server","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":f
> >   ----     ------                  ----   ----               -------
>
>  │alse,"restartCount":0,"image":"kyleoh95/iistrc-k8s-demo2:0.4","imageID":""}],"qosClass":"Guaranteed"}}]}
> >   Normal   Scheduled               2m25s  default-scheduler
> Successfully assigned default/redis-master-6fbbc44567-ms8n│Apr 15 11:36:52
> master tacker-server[1881]:  from (pid=1881) request
> /usr/local/lib/python2.7/dist-packages/kubernetes
> > k to master
>                                               │/client/rest.py:219
> >   Warning  MissingClusterDNS       2m25s  kubelet, master    pod:
> "redis-master-6fbbc44567-ms8nk_default(8a6f9079-5f27-│Apr 15 11:36:52
> master tacker-server[1881]: 2019-04-15 11:36:52.633 DEBUG
> tacker.vnfm.infra_drivers.kubernetes.kuberne
> > 11e9-9b42-a4bf01550f1a)". kubelet does not have ClusterDNS IP configured
> and cannot create Pod using "ClusterFirst" pol│tes_driver [-] status:
> Pending from (pid=1881) create_wait
> /opt/stack/tacker/tacker/vnfm/infra_drivers/kubernetes/kube
> > icy. Falling back to "Default" policy.
>                                                │rnetes_driver.py:137
> >   Warning  FailedCreatePodSandBox  19s    kubelet, master    Failed
> create pod sandbox: rpc error: code = Unknown desc │Apr 15 11:36:52 master
> tacker-server[1881]: 2019-04-15 11:36:52.634 DEBUG
> tacker.vnfm.infra_drivers.kubernetes.kuberne
> > = failed to set up sandbox container
> "310e18b859e118ad6abcaa60d29a9e332da516ce57b8cdf47456ea4882ca66f5" network
> for pod│tes_driver [-] VNF initializing status: ['default',
> 'svc-vdu1-b9e851'] Pending from (pid=1881) create_wait /opt/stack/
> >  "redis-master-6fbbc44567-ms8nk": NetworkPlugin cni failed to set up pod
> "redis-master-6fbbc44567-ms8nk_default"
> networ│tacker/tacker/vnfm/infra_drivers/kubernetes/kubernetes_driver.py:141
> > k: Got invalid status code from CNI daemon.; Traceback (most recent call
> last):                                        │Apr 15 11:36:52 master
> tacker-server[1881]: 2019-04-15 11:36:52.634 WARNING
> tacker.vnfm.infra_drivers.kubernetes.kuber
> >   File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py", line
> 80, in run                                      │netes_driver [-] VNF
> Creation failed: Resource creation is not completed within 500 seconds as
> creation of stack defau
> >     vif = self._add(params)
>
> "Got invalid status code from CNI daemon" means that you should take a
> look at kuryr-cni pods logs.
>
> >
> > Are there any reason for this error? And how can I resolve this problem?
>
> In general I'm pretty sure DevStack is not designed to be rebootable,
> so it's probably some ip route being gone or some service that haven't
> restarted. If you really want to debug this instead of firing a new
> DevStack, start from checking all "devstack@*" services in systemd.
>
> > Best Regards,
> > Jaewook.
> > ================================================
> > Jaewook Oh (오재욱)
> > IISTRC - Internet Infra System Technology Research Center
> > 369 Sangdo-ro, Dongjak-gu,
> > 06978, Seoul, Republic of Korea
> > Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618
> > E-mail : jwoh95 at dcn.ssu.ac.kr
> > ================================================
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190415/3fc6f24d/attachment-0001.html>


More information about the openstack-discuss mailing list