[kuryr][kuryr-kubernetes] does kuryr-kubernetes support dynamic subnet by pod namespace or annotation?
Hi, Folks
We need to create containers on baremetal for different tenants, so pod belongs to tenant VPC or tenant subnet, can we specify subnet by namespace or annotation in pod spec? I don’t mean multiple VIFs by additional subnets, I just need single OVS port for pod.
Hi Yi Yang,
On Tue, Oct 22, 2019 at 4:43 AM Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com wrote:
Hi, Folks
We need to create containers on baremetal for different tenants, so pod belongs to tenant VPC or tenant subnet, can we specify subnet by namespace or annotation in pod spec? I don’t mean multiple VIFs by additional subnets, I just need single OVS port for pod.
There is a namespace handler (and namespace_subnet driver) that creates a different subnet/network per K8s namespace, but those networks are created by that handle in the same tenant account (kuryr is single tenant).
On Tue, 2019-10-22 at 13:20 +0200, Luis Tomas Bolivar wrote:
Hi Yi Yang,
On Tue, Oct 22, 2019 at 4:43 AM Yi Yang (杨燚)-云服务集团 < yangyi01@inspur.com> wrote:
Hi, Folks
We need to create containers on baremetal for different tenants, so pod belongs to tenant VPC or tenant subnet, can we specify subnet by namespace or annotation in pod spec? I don’t mean multiple VIFs by additional subnets, I just need single OVS port for pod.
There is a namespace handler (and namespace_subnet driver) that creates a different subnet/network per K8s namespace, but those networks are created by that handle in the same tenant account (kuryr is single tenant).
Adding to Luis reply:
The short answer is no, but we're totally open to such a contribution in this area. We thought about it but it was never a priority, so there have never been enough resources to get it done properly.
The long answer is that it should be pretty easy to implement by having your own PodSubnetsDriver, very similar to the default [1], that would do that logic. While we would totally welcome such implementation upstream, you can also easily keep it in another Python package and use entrypoints to configure Kuryr-Kubernetes with it.
[1] https://github.com/openstack/kuryr-kubernetes/blob/5fa529efa46695ae2f29a9ad9...
Thanks Michal, got it, I'll try it, I'd like to upstream it if possible.
-----邮件原件----- 发件人: Michał Dulko [mailto:mdulko@redhat.com] 发送时间: 2019年10月22日 20:44 收件人: Luis Tomas Bolivar ltomasbo@redhat.com; Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: [kuryr][kuryr-kubernetes] does kuryr-kubernetes support dynamic subnet by pod namespace or annotation?
On Tue, 2019-10-22 at 13:20 +0200, Luis Tomas Bolivar wrote:
Hi Yi Yang,
On Tue, Oct 22, 2019 at 4:43 AM Yi Yang (杨燚)-云服务集团 < yangyi01@inspur.com> wrote:
Hi, Folks
We need to create containers on baremetal for different tenants, so pod belongs to tenant VPC or tenant subnet, can we specify subnet by namespace or annotation in pod spec? I don’t mean multiple VIFs by additional subnets, I just need single OVS port for pod.
There is a namespace handler (and namespace_subnet driver) that creates a different subnet/network per K8s namespace, but those networks are created by that handle in the same tenant account (kuryr is single tenant).
Adding to Luis reply:
The short answer is no, but we're totally open to such a contribution in this area. We thought about it but it was never a priority, so there have never been enough resources to get it done properly.
The long answer is that it should be pretty easy to implement by having your own PodSubnetsDriver, very similar to the default [1], that would do that logic. While we would totally welcome such implementation upstream, you can also easily keep it in another Python package and use entrypoints to configure Kuryr-Kubernetes with it.
[1] https://github.com/openstack/kuryr-kubernetes/blob/5fa529efa46695ae2f29a9ad9...
Thanks Luis, what if I have created network and subnet with network name and subnet name the namespace driver will create? I just want to check if it can use an existing tenant network and subnet which can be specified by namespace or annotation.
发件人: Luis Tomas Bolivar [mailto:ltomasbo@redhat.com] 发送时间: 2019年10月22日 19:21 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: [kuryr][kuryr-kubernetes] does kuryr-kubernetes support dynamic subnet by pod namespace or annotation?
Hi Yi Yang,
On Tue, Oct 22, 2019 at 4:43 AM Yi Yang (杨燚)-云服务集团 <yangyi01@inspur.com mailto:yangyi01@inspur.com > wrote:
Hi, Folks
We need to create containers on baremetal for different tenants, so pod belongs to tenant VPC or tenant subnet, can we specify subnet by namespace or annotation in pod spec? I don’t mean multiple VIFs by additional subnets, I just need single OVS port for pod.
There is a namespace handler (and namespace_subnet driver) that creates a different subnet/network per K8s namespace, but those networks are created by that handle in the same tenant account (kuryr is single tenant).
Oh, I actually should have thought about it. So if you'll precreate the network, subnet and a KuryrNet Custom Resource [1] it should actually work. The definition of KuryrNet can be find here [2], fields are pretty self-explanatory. Please note that you also need to link KuryrNet to the namespace by adding an annotation to the namespace:
"openstack.org/kuryr-net-crd": "ns-<namespace-name>"
Also, just for safety, make sure the KuryrNet itself is named "ns- <namespace-name>" - I'm not sure if some code isn't looking it up by name.
Please note that this was never tested, so maybe there's something I don't see that might prevent it from working.
[1] https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-r... [2] https://github.com/openstack/kuryr-kubernetes/blob/a85a7bc8b1761eb748ccf1643...
On Tue, 2019-10-22 at 14:11 +0000, Yi Yang (杨燚)-云服务集团 wrote:
Thanks Luis, what if I have created network and subnet with network name and subnet name the namespace driver will create? I just want to check if it can use an existing tenant network and subnet which can be specified by namespace or annotation.
发件人: Luis Tomas Bolivar [mailto:ltomasbo@redhat.com] 发送时间: 2019年10月22日 19:21 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: [kuryr][kuryr-kubernetes] does kuryr-kubernetes support dynamic subnet by pod namespace or annotation?
Hi Yi Yang,
On Tue, Oct 22, 2019 at 4:43 AM Yi Yang (杨燚)-云服务集团 < yangyi01@inspur.com> wrote:
Hi, Folks
We need to create containers on baremetal for different tenants, so pod belongs to tenant VPC or tenant subnet, can we specify subnet by namespace or annotation in pod spec? I don’t mean multiple VIFs by additional subnets, I just need single OVS port for pod.
There is a namespace handler (and namespace_subnet driver) that creates a different subnet/network per K8s namespace, but those networks are created by that handle in the same tenant account (kuryr is single tenant).
-- LUIS TOMÁS BOLÍVAR Senior Software Engineer Red Hat Madrid, Spain ltomasbo@redhat.com
Hi, Michal
I tried it, but it can't work, it is also so even for the network kuryr created by namespace driver, here is some information:
I created namespace by "kubectl create namespace kuryrns1"
yangyi@cmp001:~$ kubectl get ns NAME STATUS AGE default Active 48d kube-node-lease Active 48d kube-public Active 48d kube-system Active 48d kuryrns1 Active 52m
My kuryr conf is below:
yangyi@cmp001:~$ grep "^[^#]" /etc/kuryr/kuryr.conf [DEFAULT] bindir = /home/yangyi/kuryr-k8s-controller/env/libexec/kuryr deployment_type = baremetal log_file = /var/log/kuryr.log [binding] [cache_defaults] [cni_daemon] [cni_health_server] [health_server] [ingress] [kubernetes] api_root = https://10.110.21.64:6443 ssl_client_crt_file = /etc/kubernetes/pki/kuryr.crt ssl_client_key_file = /etc/kubernetes/pki/kuryr.key ssl_ca_crt_file = /etc/kubernetes/pki/ca.crt pod_subnets_driver = namespace enabled_handlers = vif,namespace,kuryrnet [kuryr-kubernetes] [namespace_handler_caching] [namespace_sg] [namespace_subnet] pod_router = 46fc6730-a7f9-45f7-b98b-f682c436e85c pod_subnet_pool = 581daf0e-e661-4fb8-b8d6-b7b11d0b43ab [neutron] auth_url = http://10.110.28.20:35357/v3 auth_type = password password = HAOQNs07Ci9c0DvB project_domain_id = default project_name = admin region_name = SDNRegion tenant_name = admin user_domain_id = default username = admin [neutron_defaults] project = 852d281e70b34b5398c1c5534124952e pod_subnet = b1fa2198-2ecd-41ce-bd06-93ddb2742586 pod_security_groups = d89787f5-b892-487f-b682-88742007f49f ovs_bridge = br-int service_subnet = 58b322fd-19e4-47db-b2fe-5cffd528af05 network_device_mtu = 1450 [node_driver_caching] [np_handler_caching] [octavia_defaults] [pod_ip_caching] [pod_vif_nested] [pool_manager] [sriov] [subnet_caching] [vif_handler_caching] [vif_pool] yangyi@cmp001:~$
KuryrNet has been created automatically by namespace creation:
yangyi@cmp001:~$ kubectl get KuryrNet/ns-kuryrns1 -o yaml apiVersion: openstack.org/v1 kind: KuryrNet metadata: annotations: namespaceName: kuryrns1 creationTimestamp: "2019-10-29T02:58:01Z" generation: 2 name: ns-kuryrns1 resourceVersion: "5926221" selfLink: /apis/openstack.org/v1/kuryrnets/ns-kuryrns1 uid: df5850a5-dc57-4243-b01e-be1c24d788fc spec: netId: 2dcc6969-7923-460e-8ede-17985cdf2b80 populated: true routerId: 46fc6730-a7f9-45f7-b98b-f682c436e85c subnetCIDR: 10.254.0.0/24 subnetId: a46861d3-eccf-4573-8c22-5412cc9d64f0 yangyi@cmp001:~$
But when I created deployment under kuryrns1 namespace, it never succeeded. I found CNI daemon is broken.
It is so before kubectl apply -f deploy.yaml.
yangyi@cmp004:~$ sudo ps aux | grep kuryr root 15339 0.0 0.0 51420 3852 pts/9 S 03:35 0:00 sudo -E kuryr-daemon --config-file /etc/kuryr/kuryr.conf -d root 15340 1.2 0.0 271028 101408 ? Ssl 03:35 0:01 kuryr-daemon: master process [/home/yangyi/kuryr-k8s-cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d] root 15352 0.0 0.0 268948 92016 ? S 03:35 0:00 kuryr-daemon: master process [/home/yangyi/kuryr-k8s-cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d] root 15357 0.0 0.0 426944 94624 ? Sl 03:35 0:00 kuryr-daemon: watcher worker(0) root 15362 0.0 0.0 353220 93084 ? Sl 03:35 0:00 kuryr-daemon: server worker(0) root 15366 0.0 0.0 353212 92260 ? Sl 03:35 0:00 kuryr-daemon: health worker(0)
It is so after kubectl apply -f deploy.yaml.
yangyi@cmp004:~$ sudo ps aux | grep kuryr root 15339 0.0 0.0 51420 3852 pts/9 S 03:35 0:00 sudo -E kuryr-daemon --config-file /etc/kuryr/kuryr.conf -d root 15340 0.2 0.0 271028 101408 ? Ssl 03:35 0:01 kuryr-daemon: master process [/home/yangyi/kuryr-k8s-cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d] root 15352 0.0 0.0 342680 92016 ? Sl 03:35 0:00 kuryr-daemon: master process [/home/yangyi/kuryr-k8s-cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d] root 15357 0.0 0.0 427200 95028 ? Sl 03:35 0:00 kuryr-daemon: watcher worker(0) root 15362 0.0 0.0 353220 93108 ? Sl 03:35 0:00 kuryr-daemon: server worker(0) root 15366 0.0 0.0 353212 92260 ? Sl 03:35 0:00 kuryr-daemon: health worker(0) root 16426 0.1 0.0 0 0 ? Z 03:39 0:00 [kuryr-daemon: s] <defunct> root 16729 0.0 0.0 428232 94988 ? S 03:40 0:00 kuryr-daemon: server worker(0) root 16813 0.0 0.0 429768 97480 ? S 03:40 0:00 kuryr-daemon: server worker(0) yangyi 17700 0.0 0.0 12944 1012 pts/0 R+ 03:42 0:00 grep --color=auto kuryr
I can see port is indeed created.
yangyi@cmp001:~$ openstack port list --network ns/kuryrns1-net +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | 2dd5f11f-5fc6-45ee-8f9b-8037019572cd | | fa:16:3e:af:4e:f1 | ip_address='10.254.0.3', subnet_id='a46861d3-eccf-4573-8c22-5412cc9d64f0' | ACTIVE | | c7ff9e5c-1110-4dfa-983d-2f04bf7d2794 | | fa:16:3e:10:64:c7 | ip_address='10.254.0.1', subnet_id='a46861d3-eccf-4573-8c22-5412cc9d64f0' | ACTIVE | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ yangyi@cmp001:~$
kuryr log indicated cni is defunct and is restarted.
yangyi@cmp004:~$ grep is_alive /var/log/kuryr.log 2019-10-29 03:40:00.492 16426 DEBUG kuryr_kubernetes.cni.binding.bridge [-] Reporting Driver not healthy. is_alive /home/yangyi/kuryr-k8s-cni/kuryr-kubernetes/kuryr_kubernetes/cni/binding/bridge.py:119 yangyi@cmp004:~$
Can you give me some advice or hints about how I can troubleshoot such an issue?
-----邮件原件----- 发件人: Michał Dulko [mailto:mdulko@redhat.com] 发送时间: 2019年10月22日 23:29 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com; ltomasbo@redhat.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: 答复: [kuryr][kuryr-kubernetes] does kuryr-kubernetes support dynamic subnet by pod namespace or annotation?
Oh, I actually should have thought about it. So if you'll precreate the network, subnet and a KuryrNet Custom Resource [1] it should actually work. The definition of KuryrNet can be find here [2], fields are pretty self-explanatory. Please note that you also need to link KuryrNet to the namespace by adding an annotation to the namespace:
"openstack.org/kuryr-net-crd": "ns-<namespace-name>"
Also, just for safety, make sure the KuryrNet itself is named "ns- <namespace-name>" - I'm not sure if some code isn't looking it up by name.
Please note that this was never tested, so maybe there's something I don't see that might prevent it from working.
[1] https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-r... [2] https://github.com/openstack/kuryr-kubernetes/blob/a85a7bc8b1761eb748ccf1643...
See answers inline.
On Tue, 2019-10-29 at 04:08 +0000, Yi Yang (杨燚)-云服务集团 wrote:
Hi, Michal
I tried it, but it can't work, it is also so even for the network kuryr created by namespace driver, here is some information:
I created namespace by "kubectl create namespace kuryrns1"
The correct order to have "predefined" subnets would be to start with KuryrNet creation. But okay.
yangyi@cmp001:~$ kubectl get ns NAME STATUS AGE default Active 48d kube-node-lease Active 48d kube-public Active 48d kube-system Active 48d kuryrns1 Active 52m
My kuryr conf is below:
yangyi@cmp001:~$ grep "^[^#]" /etc/kuryr/kuryr.conf [DEFAULT] bindir = /home/yangyi/kuryr-k8s-controller/env/libexec/kuryr deployment_type = baremetal log_file = /var/log/kuryr.log [binding] [cache_defaults] [cni_daemon] [cni_health_server] [health_server] [ingress] [kubernetes] api_root = https://10.110.21.64:6443 ssl_client_crt_file = /etc/kubernetes/pki/kuryr.crt ssl_client_key_file = /etc/kubernetes/pki/kuryr.key ssl_ca_crt_file = /etc/kubernetes/pki/ca.crt pod_subnets_driver = namespace enabled_handlers = vif,namespace,kuryrnet [kuryr-kubernetes] [namespace_handler_caching] [namespace_sg] [namespace_subnet] pod_router = 46fc6730-a7f9-45f7-b98b-f682c436e85c pod_subnet_pool = 581daf0e-e661-4fb8-b8d6-b7b11d0b43ab [neutron] auth_url = http://10.110.28.20:35357/v3 auth_type = password password = HAOQNs07Ci9c0DvB project_domain_id = default project_name = admin region_name = SDNRegion tenant_name = admin user_domain_id = default username = admin [neutron_defaults] project = 852d281e70b34b5398c1c5534124952e pod_subnet = b1fa2198-2ecd-41ce-bd06-93ddb2742586 pod_security_groups = d89787f5-b892-487f-b682-88742007f49f ovs_bridge = br-int service_subnet = 58b322fd-19e4-47db-b2fe-5cffd528af05 network_device_mtu = 1450 [node_driver_caching] [np_handler_caching] [octavia_defaults] [pod_ip_caching] [pod_vif_nested] [pool_manager] [sriov] [subnet_caching] [vif_handler_caching] [vif_pool] yangyi@cmp001:~$
KuryrNet has been created automatically by namespace creation:
yangyi@cmp001:~$ kubectl get KuryrNet/ns-kuryrns1 -o yaml apiVersion: openstack.org/v1 kind: KuryrNet metadata: annotations: namespaceName: kuryrns1 creationTimestamp: "2019-10-29T02:58:01Z" generation: 2 name: ns-kuryrns1 resourceVersion: "5926221" selfLink: /apis/openstack.org/v1/kuryrnets/ns-kuryrns1 uid: df5850a5-dc57-4243-b01e-be1c24d788fc spec: netId: 2dcc6969-7923-460e-8ede-17985cdf2b80 populated: true routerId: 46fc6730-a7f9-45f7-b98b-f682c436e85c subnetCIDR: 10.254.0.0/24 subnetId: a46861d3-eccf-4573-8c22-5412cc9d64f0 yangyi@cmp001:~$
But when I created deployment under kuryrns1 namespace, it never succeeded. I found CNI daemon is broken.
It is so before kubectl apply -f deploy.yaml.
yangyi@cmp004:~$ sudo ps aux | grep kuryr root 15339 0.0 0.0 51420 3852 pts/9 S 03:35 0:00 sudo -E kuryr-daemon --config-file /etc/kuryr/kuryr.conf -d root 15340 1.2 0.0 271028 101408 ? Ssl 03:35 0:01 kuryr-daemon: master process [/home/yangyi/kuryr-k8s- cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d] root 15352 0.0 0.0 268948 92016 ? S 03:35 0:00 kuryr-daemon: master process [/home/yangyi/kuryr-k8s- cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d] root 15357 0.0 0.0 426944 94624 ? Sl 03:35 0:00 kuryr-daemon: watcher worker(0) root 15362 0.0 0.0 353220 93084 ? Sl 03:35 0:00 kuryr-daemon: server worker(0) root 15366 0.0 0.0 353212 92260 ? Sl 03:35 0:00 kuryr-daemon: health worker(0)
It is so after kubectl apply -f deploy.yaml.
yangyi@cmp004:~$ sudo ps aux | grep kuryr root 15339 0.0 0.0 51420 3852 pts/9 S 03:35 0:00 sudo -E kuryr-daemon --config-file /etc/kuryr/kuryr.conf -d root 15340 0.2 0.0 271028 101408 ? Ssl 03:35 0:01 kuryr-daemon: master process [/home/yangyi/kuryr-k8s- cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d] root 15352 0.0 0.0 342680 92016 ? Sl 03:35 0:00 kuryr-daemon: master process [/home/yangyi/kuryr-k8s- cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d] root 15357 0.0 0.0 427200 95028 ? Sl 03:35 0:00 kuryr-daemon: watcher worker(0) root 15362 0.0 0.0 353220 93108 ? Sl 03:35 0:00 kuryr-daemon: server worker(0) root 15366 0.0 0.0 353212 92260 ? Sl 03:35 0:00 kuryr-daemon: health worker(0) root 16426 0.1 0.0 0 0 ? Z 03:39 0:00 [kuryr-daemon: s] <defunct> root 16729 0.0 0.0 428232 94988 ? S 03:40 0:00 kuryr-daemon: server worker(0) root 16813 0.0 0.0 429768 97480 ? S 03:40 0:00 kuryr-daemon: server worker(0) yangyi 17700 0.0 0.0 12944 1012 pts/0 R+ 03:42 0:00 grep --color=auto kuryr
Defunct processes are just zombie processes. I see that as well in our setups, it's probably a bug in pyroute2. It does not seem to affect Kuryr though.
I can see port is indeed created.
yangyi@cmp001:~$ openstack port list --network ns/kuryrns1-net
+--------------------------------------+------+-------------------+
--------+--------+
ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+
--------+--------+
2dd5f11f-5fc6-45ee-8f9b-8037019572cd | | fa:16:3e:af:4e:f1 | ip_address='10.254.0.3', subnet_id='a46861d3-eccf-4573-8c22- 5412cc9d64f0' | ACTIVE | c7ff9e5c-1110-4dfa-983d-2f04bf7d2794 | | fa:16:3e:10:64:c7 | ip_address='10.254.0.1', subnet_id='a46861d3-eccf-4573-8c22- 5412cc9d64f0' | ACTIVE |
+--------------------------------------+------+-------------------+
--------+--------+ yangyi@cmp001:~$
kuryr log indicated cni is defunct and is restarted.
yangyi@cmp004:~$ grep is_alive /var/log/kuryr.log 2019-10-29 03:40:00.492 16426 DEBUG kuryr_kubernetes.cni.binding.bridge [-] Reporting Driver not healthy. is_alive /home/yangyi/kuryr-k8s-cni/kuryr- kubernetes/kuryr_kubernetes/cni/binding/bridge.py:119 yangyi@cmp004:~$
Can you give me some advice or hints about how I can troubleshoot such an issue?
You'd need longer log, above that message you should see some logs explaining the culprit.
-----邮件原件----- 发件人: Michał Dulko [mailto:mdulko@redhat.com] 发送时间: 2019年10月22日 23:29 收件人: Yi Yang (杨燚)-云服务集团 yangyi01@inspur.com; ltomasbo@redhat.com 抄送: openstack-discuss@lists.openstack.org 主题: Re: 答复: [kuryr][kuryr-kubernetes] does kuryr-kubernetes support dynamic subnet by pod namespace or annotation?
Oh, I actually should have thought about it. So if you'll precreate the network, subnet and a KuryrNet Custom Resource [1] it should actually work. The definition of KuryrNet can be find here [2], fields are pretty self-explanatory. Please note that you also need to link KuryrNet to the namespace by adding an annotation to the namespace:
"openstack.org/kuryr-net-crd": "ns-<namespace-name>"
Also, just for safety, make sure the KuryrNet itself is named "ns- <namespace-name>" - I'm not sure if some code isn't looking it up by name.
Please note that this was never tested, so maybe there's something I don't see that might prevent it from working.
[1] https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-r... [2] https://github.com/openstack/kuryr-kubernetes/blob/a85a7bc8b1761eb748ccf1643...
participants (4)
-
Luis Tomas Bolivar
-
Michał Dulko
-
Yi Yang (杨燚)-云服务集团
-
Yi Yang (杨�D)-云服务集团