[kuryr] [kuryr-kubernetes] Unable to use kubernetes-related things after rebooting host

Jaewook Oh jwoh95 at dcn.ssu.ac.kr
Tue Apr 16 01:21:21 UTC 2019


My server that I deployed devstack was halted suddenly. I don't know what
exact reason is but, that made rebooting devstack.

Thanks!

================================================
*Jaewook Oh* (오재욱)
IISTRC - Internet Infra System Technology Research Center
369 Sangdo-ro, Dongjak-gu,
06978, Seoul, Republic of Korea
Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618
E-mail : jwoh95 at dcn.ssu.ac.kr
================================================


2019년 4월 15일 (월) 오후 5:23, Daniel Mellado Area <dmellado at redhat.com>님이 작성:

> Overall and besides of what Michal said, yeah, openstack hasn't been
> designed to be rebootable.
> A looong time ago and before the migration to systemd services there was a
> script that restarted the screen instances [1]
>
> That said, not there since 2016 so please restack whenever you reboot. In
> any case, was there any reason for rebooting?
> I'd check if just restarting service would be enough for you.
>
> Thanks!
>
> [1]
> https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=9ba49cd82631ada84b8458b370ac535985d57725
>
> On Mon, Apr 15, 2019 at 9:58 AM Michał Dulko <mdulko at redhat.com> wrote:
>
>> On Mon, 2019-04-15 at 11:44 +0900, Jaewook Oh wrote:
>> > Hello kuryr-kubernetes team,
>> >
>> > currently, I deployed openstack with devstack, and I added
>> "kuryr-kubernetes" to use kubernetes vim.
>> > At first it worked fine and no error was occurred,
>> > so I made some c-vnfs with kuryr-kubernetes.
>> > But after rebooting the host, the systems that worked well before are
>> not working normally at all.
>> >
>> > First, I deployed an example one ('redis') with kubectl command, (Not
>> using Tacker, but only using kubectl cli)
>> > it showed below error,
>> >
>> > Events:
>>
>> │obeTime":null,"lastTransitionTime":"2019-04-15T02:28:31Z"}],"hostIP":"192.168.1.20","startTime":"2019-04-15T02:28:31Z"
>> >   Type     Reason                  Age    From               Message
>>
>>  │,"containerStatuses":[{"name":"web-server","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":f
>> >   ----     ------                  ----   ----               -------
>>
>>  │alse,"restartCount":0,"image":"kyleoh95/iistrc-k8s-demo2:0.4","imageID":""}],"qosClass":"Guaranteed"}}]}
>> >   Normal   Scheduled               2m25s  default-scheduler
>> Successfully assigned default/redis-master-6fbbc44567-ms8n│Apr 15 11:36:52
>> master tacker-server[1881]:  from (pid=1881) request
>> /usr/local/lib/python2.7/dist-packages/kubernetes
>> > k to master
>>                                                 │/client/rest.py:219
>> >   Warning  MissingClusterDNS       2m25s  kubelet, master    pod:
>> "redis-master-6fbbc44567-ms8nk_default(8a6f9079-5f27-│Apr 15 11:36:52
>> master tacker-server[1881]: 2019-04-15 11:36:52.633 DEBUG
>> tacker.vnfm.infra_drivers.kubernetes.kuberne
>> > 11e9-9b42-a4bf01550f1a)". kubelet does not have ClusterDNS IP
>> configured and cannot create Pod using "ClusterFirst" pol│tes_driver [-]
>> status: Pending from (pid=1881) create_wait
>> /opt/stack/tacker/tacker/vnfm/infra_drivers/kubernetes/kube
>> > icy. Falling back to "Default" policy.
>>                                                │rnetes_driver.py:137
>> >   Warning  FailedCreatePodSandBox  19s    kubelet, master    Failed
>> create pod sandbox: rpc error: code = Unknown desc │Apr 15 11:36:52 master
>> tacker-server[1881]: 2019-04-15 11:36:52.634 DEBUG
>> tacker.vnfm.infra_drivers.kubernetes.kuberne
>> > = failed to set up sandbox container
>> "310e18b859e118ad6abcaa60d29a9e332da516ce57b8cdf47456ea4882ca66f5" network
>> for pod│tes_driver [-] VNF initializing status: ['default',
>> 'svc-vdu1-b9e851'] Pending from (pid=1881) create_wait /opt/stack/
>> >  "redis-master-6fbbc44567-ms8nk": NetworkPlugin cni failed to set up
>> pod "redis-master-6fbbc44567-ms8nk_default"
>> networ│tacker/tacker/vnfm/infra_drivers/kubernetes/kubernetes_driver.py:141
>> > k: Got invalid status code from CNI daemon.; Traceback (most recent
>> call last):                                        │Apr 15 11:36:52 master
>> tacker-server[1881]: 2019-04-15 11:36:52.634 WARNING
>> tacker.vnfm.infra_drivers.kubernetes.kuber
>> >   File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py", line
>> 80, in run                                      │netes_driver [-] VNF
>> Creation failed: Resource creation is not completed within 500 seconds as
>> creation of stack defau
>> >     vif = self._add(params)
>>
>> "Got invalid status code from CNI daemon" means that you should take a
>> look at kuryr-cni pods logs.
>>
>> >
>> > Are there any reason for this error? And how can I resolve this problem?
>>
>> In general I'm pretty sure DevStack is not designed to be rebootable,
>> so it's probably some ip route being gone or some service that haven't
>> restarted. If you really want to debug this instead of firing a new
>> DevStack, start from checking all "devstack@*" services in systemd.
>>
>> > Best Regards,
>> > Jaewook.
>> > ================================================
>> > Jaewook Oh (오재욱)
>> > IISTRC - Internet Infra System Technology Research Center
>> > 369 Sangdo-ro, Dongjak-gu,
>> > 06978, Seoul, Republic of Korea
>> > Tel : +82-2-820-0841 | Mobile : +82-10-9924-2618
>> > E-mail : jwoh95 at dcn.ssu.ac.kr
>> > ================================================
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190416/97650c97/attachment-0001.html>


More information about the openstack-discuss mailing list