The controller logs are from an hour later than the CNI one. The issues seems not to be present. Isn't your controller still restarting? If so try to use -p option on `kubectl logs` to get logs from previous run. On Wed, 2020-02-12 at 09:38 +0000, VeeraReddy wrote:

Hi Mdulko, Below are log files:

Controller Log:http://paste.openstack.org/show/789457/ cni : http://paste.openstack.org/show/789456/ kubelet : http://paste.openstack.org/show/789453/

Unable to create kubelet interface in node, so not able to reach cluster (i.e 10.0.0.129)

Please let me know the issue

Regards, Veera.

On Tuesday, 11 February, 2020, 03:29:10 pm IST, <mdulko@redhat.com> wrote:

Hi,

So from this run you need the kuryr-controller logs. Apparently the pod never got annotated with an information about the VIF.

Thanks, Michał

On Fri, 2020-02-07 at 05:14 +0000, VeeraReddy wrote:

...

Hi mdulko, Thanks for your support.

As you mention i removed readinessProbe and livenessProbe from Kuryr pod definitions. Still i am facing issue , unable to create pod.

Attached kubelet and kuryr-cni logs.

Regards, Veera.

On Thursday, 6 February, 2020, 05:19:12 pm IST, <mdulko@redhat.com> wrote:

Hm, nothing too troubling there too, besides Kubernetes not answering on /healthz endpoint. Are those full logs, including the moment you tried spawning a container there? It seems like you only pasted the fragments with tracebacks regarding failures to read /healthz endpoint of kube-apiserver. That is another problem you should investigate - that causes Kuryr pods to restart.

At first I'd disable the healthchecks (remove readinessProbe and livenessProbe from Kuryr pod definitions) and try to get fresh set of logs.

On Thu, 2020-02-06 at 10:46 +0000, VeeraReddy wrote:

...

Hi mdulko, Please find kuryr-cni logs http://paste.openstack.org/show/789209/

Regards, Veera.

On Thursday, 6 February, 2020, 04:08:35 pm IST, <mdulko@redhat.com> wrote:

Hi,

The logs you provided doesn't seem to indicate any issues. Please provide logs of kuryr-daemon (kuryr-cni pod).

Thanks, Michał

On Thu, 2020-02-06 at 07:09 +0000, VeeraReddy wrote:

...

Hi, I am trying to run kubelet in arm64 platform 1. Generated kuryr-cni successfullly. using kur-cni Dockerfile 2. Generated kuryr-cni-arm64 container. 3. my kube-kuryr-arm64.yml (http://paste.openstack.org/show/789208/)

My master node in x86 installed successfully using devstack

While running kubelet in arm platform , not able to create kubelet interface (kubelet logs: http://paste.openstack.org/show/789206/)

COntroller logs: http://paste.openstack.org/show/789209/

Please help me to fix the issue

Veera.

Regards, Veera.

We're using 2.9.5 on the x86_64 gate and that works fine. I'm not sure if downgrading could help. This is a Neutron issue and I don't have much experience on such a low level. You can try asking on IRC, e.g. on #openstack-neutron. On Thu, 2020-02-13 at 06:52 +0000, VeeraReddy wrote:

Thanks mdulko, Issue is in openvswitch, iam getting following error in switchd logs ./ovs-vswitchd.log:2020-02-12T12:21:18.177Z|00493|connmgr|INFO|br-int<->unix#50: sending NXBAC_CT_DATAPATH_SUPPORT error reply to OFPT_BUNDLE_ADD_MESSAGE message

Do we need to patch openvswitch to support above flow?

My ovs version

[root@node-2088 ~]# ovs-vsctl --version ovs-vsctl (Open vSwitch) 2.11.0 DB Schema 7.16.1

Regards, Veera.

On Wednesday, 12 February, 2020, 05:37:31 pm IST, <mdulko@redhat.com> wrote:

The controller logs are from an hour later than the CNI one. The issues seems not to be present.

Isn't your controller still restarting? If so try to use -p option on `kubectl logs` to get logs from previous run.

On Wed, 2020-02-12 at 09:38 +0000, VeeraReddy wrote:

...

Hi Mdulko, Below are log files:

Controller Log:http://paste.openstack.org/show/789457/ cni : http://paste.openstack.org/show/789456/ kubelet : http://paste.openstack.org/show/789453/

Unable to create kubelet interface in node, so not able to reach cluster (i.e 10.0.0.129)

Please let me know the issue

Regards, Veera.

On Tuesday, 11 February, 2020, 03:29:10 pm IST, <mdulko@redhat.com> wrote:

Hi,

So from this run you need the kuryr-controller logs. Apparently the pod never got annotated with an information about the VIF.

Thanks, Michał

On Fri, 2020-02-07 at 05:14 +0000, VeeraReddy wrote:

...

Hi mdulko, Thanks for your support.

As you mention i removed readinessProbe and livenessProbe from Kuryr pod definitions. Still i am facing issue , unable to create pod.

Attached kubelet and kuryr-cni logs.

Regards, Veera.

On Thursday, 6 February, 2020, 05:19:12 pm IST, <mdulko@redhat.com> wrote:

Hm, nothing too troubling there too, besides Kubernetes not answering on /healthz endpoint. Are those full logs, including the moment you tried spawning a container there? It seems like you only pasted the fragments with tracebacks regarding failures to read /healthz endpoint of kube-apiserver. That is another problem you should investigate - that causes Kuryr pods to restart.

At first I'd disable the healthchecks (remove readinessProbe and livenessProbe from Kuryr pod definitions) and try to get fresh set of logs.

On Thu, 2020-02-06 at 10:46 +0000, VeeraReddy wrote:

...

Hi mdulko, Please find kuryr-cni logs http://paste.openstack.org/show/789209/

Regards, Veera.

On Thursday, 6 February, 2020, 04:08:35 pm IST, <mdulko@redhat.com> wrote:

Hi,

The logs you provided doesn't seem to indicate any issues. Please provide logs of kuryr-daemon (kuryr-cni pod).

Thanks, Michał

On Thu, 2020-02-06 at 07:09 +0000, VeeraReddy wrote:

...

Hi, I am trying to run kubelet in arm64 platform 1. Generated kuryr-cni successfullly. using kur-cni Dockerfile 2. Generated kuryr-cni-arm64 container. 3. my kube-kuryr-arm64.yml (http://paste.openstack.org/show/789208/)

My master node in x86 installed successfully using devstack

While running kubelet in arm platform , not able to create kubelet interface (kubelet logs: http://paste.openstack.org/show/789206/)

COntroller logs: http://paste.openstack.org/show/789209/

Please help me to fix the issue

Veera.

Regards, Veera.

On Mon, 2020-02-17 at 11:19 +0000, VeeraReddy wrote:

Hi mdulko, Thanks you very much, I am able to launch Container and VM side by side in same network on arm64 platform. Issue was openvswitch kernel Module(Enabled "geneve module" in openvswitch).

Now i am able to ping from container to VM but unable to ping Vice Versa (VM to container). And also, I am able ping VM to VM (Both spawned from OpenStack dashboard).

This should depend entirely on your network topology. If Pod->VM traffic works, then subnets seems to be routed correctly, so I'd expect it's the security groups that are blocking traffic. Please check that. Please note that we test VM->Pod and Pod->VM on the gate [1]. [1] https://github.com/openstack/kuryr-tempest-plugin/blob/master/kuryr_tempest_...

Is there any configuration to enable traffic from VM to Container?

Regards, Veera.

On Thursday, 13 February, 2020, 02:42:21 pm IST, <mdulko@redhat.com> wrote:

We're using 2.9.5 on the x86_64 gate and that works fine. I'm not sure if downgrading could help. This is a Neutron issue and I don't have much experience on such a low level. You can try asking on IRC, e.g. on #openstack-neutron.

On Thu, 2020-02-13 at 06:52 +0000, VeeraReddy wrote:

...

Thanks mdulko, Issue is in openvswitch, iam getting following error in switchd logs ./ovs-vswitchd.log:2020-02-12T12:21:18.177Z|00493|connmgr|INFO|br-int<->unix#50: sending NXBAC_CT_DATAPATH_SUPPORT error reply to OFPT_BUNDLE_ADD_MESSAGE message

Do we need to patch openvswitch to support above flow?

My ovs version

[root@node-2088 ~]# ovs-vsctl --version ovs-vsctl (Open vSwitch) 2.11.0 DB Schema 7.16.1

Regards, Veera.

On Wednesday, 12 February, 2020, 05:37:31 pm IST, <mdulko@redhat.com> wrote:

The controller logs are from an hour later than the CNI one. The issues seems not to be present.

Isn't your controller still restarting? If so try to use -p option on `kubectl logs` to get logs from previous run.

On Wed, 2020-02-12 at 09:38 +0000, VeeraReddy wrote:

...

Hi Mdulko, Below are log files:

Controller Log:http://paste.openstack.org/show/789457/ cni : http://paste.openstack.org/show/789456/ kubelet : http://paste.openstack.org/show/789453/

Unable to create kubelet interface in node, so not able to reach cluster (i.e 10.0.0.129)

Please let me know the issue

Regards, Veera.

On Tuesday, 11 February, 2020, 03:29:10 pm IST, <mdulko@redhat.com> wrote:

Hi,

So from this run you need the kuryr-controller logs. Apparently the pod never got annotated with an information about the VIF.

Thanks, Michał

On Fri, 2020-02-07 at 05:14 +0000, VeeraReddy wrote:

...

Hi mdulko, Thanks for your support.

As you mention i removed readinessProbe and livenessProbe from Kuryr pod definitions. Still i am facing issue , unable to create pod.

Attached kubelet and kuryr-cni logs.

Regards, Veera.

On Thursday, 6 February, 2020, 05:19:12 pm IST, <mdulko@redhat.com> wrote:

Hm, nothing too troubling there too, besides Kubernetes not answering on /healthz endpoint. Are those full logs, including the moment you tried spawning a container there? It seems like you only pasted the fragments with tracebacks regarding failures to read /healthz endpoint of kube-apiserver. That is another problem you should investigate - that causes Kuryr pods to restart.

At first I'd disable the healthchecks (remove readinessProbe and livenessProbe from Kuryr pod definitions) and try to get fresh set of logs.

On Thu, 2020-02-06 at 10:46 +0000, VeeraReddy wrote:

...

Hi mdulko, Please find kuryr-cni logs http://paste.openstack.org/show/789209/

Regards, Veera.

On Thursday, 6 February, 2020, 04:08:35 pm IST, <mdulko@redhat.com> wrote:

Hi,

The logs you provided doesn't seem to indicate any issues. Please provide logs of kuryr-daemon (kuryr-cni pod).

Thanks, Michał

On Thu, 2020-02-06 at 07:09 +0000, VeeraReddy wrote:

...

Hi, I am trying to run kubelet in arm64 platform 1. Generated kuryr-cni successfullly. using kur-cni Dockerfile 2. Generated kuryr-cni-arm64 container. 3. my kube-kuryr-arm64.yml (http://paste.openstack.org/show/789208/)

My master node in x86 installed successfully using devstack

While running kubelet in arm platform , not able to create kubelet interface (kubelet logs: http://paste.openstack.org/show/789206/)

COntroller logs: http://paste.openstack.org/show/789209/

Please help me to fix the issue

Veera.

Regards, Veera.