Open Stack

Hi, Michal

I tried it, but it can't work, it is also so even for the network kuryr created by namespace driver, here is some information:

I created namespace by "kubectl create namespace kuryrns1"

yangyi at cmp001:~$ kubectl get ns
NAME              STATUS   AGE
default           Active   48d
kube-node-lease   Active   48d
kube-public       Active   48d
kube-system       Active   48d
kuryrns1          Active   52m

My kuryr conf is below:

yangyi at cmp001:~$ grep "^[^#]" /etc/kuryr/kuryr.conf
[DEFAULT]
bindir = /home/yangyi/kuryr-k8s-controller/env/libexec/kuryr
deployment_type = baremetal
log_file = /var/log/kuryr.log
[binding]
[cache_defaults]
[cni_daemon]
[cni_health_server]
[health_server]
[ingress]
[kubernetes]
api_root = https://10.110.21.64:6443
ssl_client_crt_file = /etc/kubernetes/pki/kuryr.crt
ssl_client_key_file = /etc/kubernetes/pki/kuryr.key
ssl_ca_crt_file = /etc/kubernetes/pki/ca.crt
pod_subnets_driver = namespace
enabled_handlers = vif,namespace,kuryrnet
[kuryr-kubernetes]
[namespace_handler_caching]
[namespace_sg]
[namespace_subnet]
pod_router = 46fc6730-a7f9-45f7-b98b-f682c436e85c
pod_subnet_pool = 581daf0e-e661-4fb8-b8d6-b7b11d0b43ab
[neutron]
auth_url = http://10.110.28.20:35357/v3
auth_type = password
password = HAOQNs07Ci9c0DvB
project_domain_id = default
project_name = admin
region_name = SDNRegion
tenant_name = admin
user_domain_id = default
username = admin
[neutron_defaults]
project = 852d281e70b34b5398c1c5534124952e
pod_subnet = b1fa2198-2ecd-41ce-bd06-93ddb2742586
pod_security_groups = d89787f5-b892-487f-b682-88742007f49f
ovs_bridge = br-int
service_subnet = 58b322fd-19e4-47db-b2fe-5cffd528af05
network_device_mtu = 1450
[node_driver_caching]
[np_handler_caching]
[octavia_defaults]
[pod_ip_caching]
[pod_vif_nested]
[pool_manager]
[sriov]
[subnet_caching]
[vif_handler_caching]
[vif_pool]
yangyi at cmp001:~$

KuryrNet has been created automatically by namespace creation:

yangyi at cmp001:~$ kubectl get KuryrNet/ns-kuryrns1 -o yaml
apiVersion: openstack.org/v1
kind: KuryrNet
metadata:
  annotations:
    namespaceName: kuryrns1
  creationTimestamp: "2019-10-29T02:58:01Z"
  generation: 2
  name: ns-kuryrns1
  resourceVersion: "5926221"
  selfLink: /apis/openstack.org/v1/kuryrnets/ns-kuryrns1
  uid: df5850a5-dc57-4243-b01e-be1c24d788fc
spec:
  netId: 2dcc6969-7923-460e-8ede-17985cdf2b80
  populated: true
  routerId: 46fc6730-a7f9-45f7-b98b-f682c436e85c
  subnetCIDR: 10.254.0.0/24
  subnetId: a46861d3-eccf-4573-8c22-5412cc9d64f0
yangyi at cmp001:~$

But when I created deployment under kuryrns1 namespace, it never succeeded. I found CNI daemon is broken.

It is so before kubectl apply -f deploy.yaml.

yangyi at cmp004:~$ sudo ps aux | grep kuryr
root     15339  0.0  0.0  51420  3852 pts/9    S    03:35   0:00 sudo -E kuryr-daemon --config-file /etc/kuryr/kuryr.conf -d
root     15340  1.2  0.0 271028 101408 ?       Ssl  03:35   0:01 kuryr-daemon: master process [/home/yangyi/kuryr-k8s-cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d]
root     15352  0.0  0.0 268948 92016 ?        S    03:35   0:00 kuryr-daemon: master process [/home/yangyi/kuryr-k8s-cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d]
root     15357  0.0  0.0 426944 94624 ?        Sl   03:35   0:00 kuryr-daemon: watcher worker(0)
root     15362  0.0  0.0 353220 93084 ?        Sl   03:35   0:00 kuryr-daemon: server worker(0)
root     15366  0.0  0.0 353212 92260 ?        Sl   03:35   0:00 kuryr-daemon: health worker(0)

It is so after kubectl apply -f deploy.yaml.

yangyi at cmp004:~$ sudo ps aux | grep kuryr
root     15339  0.0  0.0  51420  3852 pts/9    S    03:35   0:00 sudo -E kuryr-daemon --config-file /etc/kuryr/kuryr.conf -d
root     15340  0.2  0.0 271028 101408 ?       Ssl  03:35   0:01 kuryr-daemon: master process [/home/yangyi/kuryr-k8s-cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d]
root     15352  0.0  0.0 342680 92016 ?        Sl   03:35   0:00 kuryr-daemon: master process [/home/yangyi/kuryr-k8s-cni/env/bin/kuryr-daemon --config-file /etc/kuryr kuryr.conf -d]
root     15357  0.0  0.0 427200 95028 ?        Sl   03:35   0:00 kuryr-daemon: watcher worker(0)
root     15362  0.0  0.0 353220 93108 ?        Sl   03:35   0:00 kuryr-daemon: server worker(0)
root     15366  0.0  0.0 353212 92260 ?        Sl   03:35   0:00 kuryr-daemon: health worker(0)
root     16426  0.1  0.0      0     0 ?        Z    03:39   0:00 [kuryr-daemon: s] <defunct>
root     16729  0.0  0.0 428232 94988 ?        S    03:40   0:00 kuryr-daemon: server worker(0)
root     16813  0.0  0.0 429768 97480 ?        S    03:40   0:00 kuryr-daemon: server worker(0)
yangyi   17700  0.0  0.0  12944  1012 pts/0    R+   03:42   0:00 grep --color=auto kuryr

I can see port is indeed created.

yangyi at cmp001:~$ openstack port list --network ns/kuryrns1-net
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+
| ID                                   | Name | MAC Address       | Fixed IP Addresses                                                        | Status |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+
| 2dd5f11f-5fc6-45ee-8f9b-8037019572cd |      | fa:16:3e:af:4e:f1 | ip_address='10.254.0.3', subnet_id='a46861d3-eccf-4573-8c22-5412cc9d64f0' | ACTIVE |
| c7ff9e5c-1110-4dfa-983d-2f04bf7d2794 |      | fa:16:3e:10:64:c7 | ip_address='10.254.0.1', subnet_id='a46861d3-eccf-4573-8c22-5412cc9d64f0' | ACTIVE |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+
yangyi at cmp001:~$

kuryr log indicated cni is defunct and is restarted.

yangyi at cmp004:~$ grep is_alive /var/log/kuryr.log
2019-10-29 03:40:00.492 16426 DEBUG kuryr_kubernetes.cni.binding.bridge [-] Reporting Driver not healthy. is_alive /home/yangyi/kuryr-k8s-cni/kuryr-kubernetes/kuryr_kubernetes/cni/binding/bridge.py:119
yangyi at cmp004:~$

Can you give me some advice or hints about how I can troubleshoot such an issue?

-----邮件原件-----
发件人: Michał Dulko [mailto:mdulko at redhat.com] 
发送时间: 2019年10月22日 23:29
收件人: Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com>; ltomasbo at redhat.com
抄送: openstack-discuss at lists.openstack.org
主题: Re: 答复: [kuryr][kuryr-kubernetes] does kuryr-kubernetes support dynamic subnet by pod namespace or annotation?

Oh, I actually should have thought about it. So if you'll precreate the network, subnet and a KuryrNet Custom Resource [1] it should actually work. The definition of KuryrNet can be find here [2], fields are pretty self-explanatory. Please note that you also need to link KuryrNet to the namespace by adding an annotation to the namespace:

 "openstack.org/kuryr-net-crd": "ns-<namespace-name>"

Also, just for safety, make sure the KuryrNet itself is named "ns- <namespace-name>" - I'm not sure if some code isn't looking it up by name.

Please note that this was never tested, so maybe there's something I don't see that might prevent it from working.

[1] https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
[2] https://github.com/openstack/kuryr-kubernetes/blob/a85a7bc8b1761eb748ccf16430fe77587bc764c2/kubernetes_crds/kuryrnet.yaml
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3600 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20191029/499eb14e/attachment.bin>