November 2021 - openstack-discuss

[kolla][release][Release-job-failures] Release of openstack/kolla for ref refs/tags/10.4.0 failed
by Előd Illés 24 Nov '21

24 Nov '21

Hi Kolla team, The last release of kolla (10.4.0 [1]) had release job failures (see below the attached, forwarded mail). The two job that failed are: - kolla-publish-centos8-source-dockerhub https://zuul.opendev.org/t/openstack/build/feb51c87f74c4dadb4984733d8b3793c : FAILURE in 2h 12m 37s - kolla-publish-centos8-binary-dockerhub https://zuul.opendev.org/t/openstack/build/ebce7219ff4b44368869bc1f3172d688 : FAILURE in 1h 46m 56s (non-voting) Could you please have a look at these? Thanks in advance, Előd (elodilles @ #openstack-release) [1] https://review.opendev.org/c/openstack/releases/+/818882 -------- Forwarded Message -------- Subject: [Release-job-failures] Release of openstack/kolla for ref refs/tags/10.4.0 failed Date: Tue, 23 Nov 2021 17:22:02 +0000 From: zuul(a)openstack.org Reply-To: openstack-discuss(a)lists.openstack.org To: release-job-failures(a)lists.openstack.org Build failed. - openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/4834324fec464b309952930a2c221336 : SUCCESS in 1m 45s - release-openstack-python https://zuul.opendev.org/t/openstack/build/e3f27c5f3f9b4b289dc73570243507ff : SUCCESS in 3m 06s - announce-release https://zuul.opendev.org/t/openstack/build/1d759f048a1145af9b253b6e5d69b4f4 : SUCCESS in 4m 23s - propose-update-constraints https://zuul.opendev.org/t/openstack/build/3a600e6a4c494a829bfcd45c91931067 : SUCCESS in 3m 42s - kolla-publish-centos8-source-dockerhub https://zuul.opendev.org/t/openstack/build/feb51c87f74c4dadb4984733d8b3793c : FAILURE in 2h 12m 37s - kolla-publish-centos8-binary-dockerhub https://zuul.opendev.org/t/openstack/build/ebce7219ff4b44368869bc1f3172d688 : FAILURE in 1h 46m 56s (non-voting) - kolla-publish-debian-source-dockerhub https://zuul.opendev.org/t/openstack/build/e076910efb394683b98106d5648c2ab5 : SUCCESS in 1h 54m 47s (non-voting) - kolla-publish-debian-source-aarch64-dockerhub https://zuul.opendev.org/t/openstack/build/e201c6f797ea4248b933df8c53d65fa8 : SUCCESS in 2h 03m 51s (non-voting) - kolla-publish-debian-binary-dockerhub https://zuul.opendev.org/t/openstack/build/d0cf2b3aa48e49d8a5aa46719b84c926 : SUCCESS in 1h 18m 03s (non-voting) - kolla-publish-ubuntu-source-dockerhub https://zuul.opendev.org/t/openstack/build/17644441b45f4964b52a1e26e8ef3f19 : SUCCESS in 2h 01m 47s - kolla-publish-ubuntu-binary-dockerhub https://zuul.opendev.org/t/openstack/build/63d1810841da4d9c96d4755177761f4d : SUCCESS in 1h 34m 09s (non-voting) _______________________________________________ Release-job-failures mailing list Release-job-failures(a)lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures

1 0

[neutron] default QOS on L3 gateway
by Damien Rannou 24 Nov '21

24 Nov '21

Hello We are currently playing with QOS on L3 agent, mostly for SNAT, but it can apply also on FIP. Everything is working properly, but I’m wondering if there is a way to define a « default » QOS that would be applied on Router creation, but also when the user is setting « no_qos » on his router. On a public cloud environnement, we cannot let the customers without any QOS limitation. Thanks ! Damien

3 3

[kolla-ansible][wallaby][magnum][Kubernetes] Cannot auto-scale workers
by wodel youchi 24 Nov '21

24 Nov '21

Hi, I have a new kolla-ansible deployment with wallaby. I have created a kubernetes cluster using calico (flannel didn't work for me). I configured an autoscale test to see if it works. - pods autoscale is working. - worker nodes autoscale is not working. This is my deployment file :*cat php-apache.yaml* apiVersion: apps/v1 kind: Deployment metadata: name: php-apache-deployment spec: selector: matchLabels: app: php-apache replicas: 2 template: metadata: labels: app: php-apache spec: containers: - name: php-apache image: k8s.gcr.io/hpa-example ports: - containerPort: 80 resources: limits: cpu: 500m requests: cpu: 200m --- apiVersion: v1 kind: Service metadata: name: php-apache-service labels: app: php-apache spec: ports: - port: 80 targetPort: 80 protocol: TCP selector: app: php-apache type: LoadBalancer This is my HPA file :*cat php-apache-hpa.yaml* apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: php-apache-hpa namespace: default labels: service: php-apache-service spec: minReplicas: 2 maxReplicas: 30 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: php-apache-deployment metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 30 # en pourcentage This is my load program : kubectl run -i --tty load-generator-1 --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://ip_load_balancer; done" Here are the output of my kub cluster before the test : [kube8@cdndeployer ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE php-apache-deployment-5b65bbc75c-95k6k 1/1 Running 0 24m php-apache-deployment-5b65bbc75c-mv5h6 1/1 Running 0 24m [kube8@cdndeployer ~]$ kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE php-apache-hpa Deployment/php-apache-deployment *0%/30%* 2 15 2 24m [kube8@cdndeployer ~]$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 13h php-apache-service LoadBalancer 10.254.3.54 *xx.xx.xx.213* 80:31763/TCP 25m When I apply the load : - pods autoscale creates new pods, then some of them get in the state of : *pending * [kube8@cdndeployer ~]$ kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE php-apache-hpa Deployment/php-apache-deployment *155%/30%* 2 15 4 27m [kube8@cdndeployer ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE load-generator-1 1/1 Running 0 97s load-generator-2 1/1 Running 0 94s php-apache-deployment-5b65bbc75c-95k6k 1/1 Running 0 28m *php-apache-deployment-5b65bbc75c-cjkwk 0/1 Pending 0 33s* *php-apache-deployment-5b65bbc75c-cn5rt 0/1 Pending 0 33s* *php-apache-deployment-5b65bbc75c-cxctx 0/1 Pending 0 48s* php-apache-deployment-5b65bbc75c-fffnc 1/1 Running 0 64s php-apache-deployment-5b65bbc75c-hbfw8 0/1 Pending 0 33s php-apache-deployment-5b65bbc75c-l8496 1/1 Running 0 48s php-apache-deployment-5b65bbc75c-mv5h6 1/1 Running 0 28m php-apache-deployment-5b65bbc75c-qddrb 1/1 Running 0 48s php-apache-deployment-5b65bbc75c-dd5r5 0/1 Pending 0 48s php-apache-deployment-5b65bbc75c-tr65j 1/1 Running 0 64s 2 - The cluster is unable to create more pods/workers and I get this error message from the pending pods kubectl describe pod php-apache-deployment-5b65bbc75c-dd5r5 Name: php-apache-deployment-5b65bbc75c-dd5r5 Namespace: default Priority: 0 Node: <none> Labels: app=php-apache pod-template-hash=5b65bbc75c Annotations: kubernetes.io/psp: magnum.privileged *Status: Pending* IP: IPs: <none> Controlled By: ReplicaSet/php-apache-deployment-5b65bbc75c Containers: php-apache: Image: k8s.gcr.io/hpa-example Port: 80/TCP Host Port: 0/TCP Limits: cpu: 500m Requests: cpu: 200m Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-4fsgh (ro) Conditions: Type Status PodScheduled False Volumes: default-token-4fsgh: Type: Secret (a volume populated by a Secret) SecretName: default-token-4fsgh Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- * Warning FailedScheduling 2m48s default-scheduler 0/4 nodes are available: 2 Insufficient cpu, 2 node(s) had taint {node-role.kubernetes.io/master <http://node-role.kubernetes.io/master>: }, that the pod didn't tolerate.* * Warning FailedScheduling 2m48s default-scheduler 0/4 nodes are available: 2 Insufficient cpu, 2 node(s) had taint {node-role.kubernetes.io/master <http://node-role.kubernetes.io/master>: }, that the pod didn't tolerate.* * Normal NotTriggerScaleUp 2m42s cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 in backoff after failed scale-u**p* I have this error message from the autoscaller pod *cluster-autoscaler*-f4bd5f674-b9692 : I1123 00:50:27.714801 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.709µs I1123 00:51:34.181145 1 scale_up.go:658] Scale-up: setting group default-worker size to 3 *W1123 00:51:34.381953 1 clusterstate.go:281] Disabling scale-up for node group default-worker until 2021-11-23 00:56:34.180840351 +0000 UTC m=+47174.376164120; errorClass=Other; errorCode=cloudProviderError* *E1123 00:51:34.382081 1 static_autoscaler.go:415] Failed to scale up: failed to increase node group size: could not check current nodegroup size: could not get cluster: Get https://dash.cdn.domaine.tld:9511/v1/clusters/b4a6b3eb-fcf3-416f-b740-11a08… <https://dash.cdn.cerist.dz:9511/v1/clusters/b4a6b3eb-fcf3-416f-b740-11a083d…>: dial tcp: lookup dash.cdn.domaine. <http://dash.cdn.cerist.dz>tld on 10.254.0.10:53 <http://10.254.0.10:53>: no such host* W1123 00:51:44.392523 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:51:54.410273 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:52:04.422128 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:52:14.434278 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:52:24.442480 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff I1123 00:52:27.715019 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I did some tests on the DNS pod and : kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 13h default php-apache-service LoadBalancer 10.254.3.54 xx.xx.xx.213 80:31763/TCP 19m kube-system dashboard-metrics-scraper ClusterIP 10.254.19.191 <none> 8000/TCP 13h *kube-system kube-dns ClusterIP 10.254.0.10 <none> 53/UDP,53/TCP,9153/TCP 13h* kube-system kubernetes-dashboard ClusterIP 10.254.132.17 <none> 443/TCP 13h kube-system magnum-metrics-server ClusterIP 10.254.235.147 <none> 443/TCP 13h I have noticed this behaviour about the horizon url, sometimes the dns pod responds sometimes it does not !!!!! [root@k8multiclustercalico-ve5t6uuoo245-master-0 ~]# *dig @10.254.0.10 <http://10.254.0.10> dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld* ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc33 <<>> @10.254.0.10 dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 5646 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;dash.cd <http://dash.cdn.cerist.dz>n.domaine.tld. IN A ;; AUTHORITY SECTION: *cdn. <http://cdn.cerist.dz>domaine.tld. 30 IN SOA cdn. <http://cdn.cerist.dz>domaine.tld. root.cdn. <http://root.cdn.cerist.dz>domaine.tld. 2021100900 604800 86400 2419200 604800* ;; Query time: 84 msec ;; SERVER: 10.254.0.10#53(10.254.0.10) ;; WHEN: Tue Nov 23 01:08:03 UTC 2021 ;; MSG SIZE rcvd: 12 2 secondes later [root@k8multiclustercalico-ve5t6uuoo245-master-0 ~]# *dig @10.254.0.10 <http://10.254.0.10> dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld* ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc33 <<>> @10.254.0.10 dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7653 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld. IN A ;; ANSWER SECTION: *dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld. 30 IN A xx.xx.xx.129* ;; Query time: 2 msec ;; SERVER: 10.254.0.10#53(10.254.0.10) ;; WHEN: Tue Nov 23 01:08:21 UTC 2021 ;; MSG SIZE rcvd: 81 In the log of the dns pod I have this kubectl logs *kube-dns-autoscaler*-75859754fd-q8z4w -n kube-system *E1122 20:56:09.944449 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource E1122 20:56:19.945294 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource E1122 20:56:29.944245 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource E1122 20:56:39.946346 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource* *E1122 20:56:49.944693 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource* I don't have experience on kubernetes yet, could someone help me debug this? Regards.

1 0

[horizon] Cancelling Today's weekly meeting
by vishal manchanda 23 Nov '21

23 Nov '21

Hello Team, Since there are no agenda items [1] to discuss for today's horizon weekly meeting. Also, Today is a holiday for me or maybe for others as well So let's cancel today's weekly meeting. Next weekly meeting will be on 01 December. Thanks & Regards, Vishal Manchanda [1] https://etherpad.opendev.org/p/horizon-release-priorities

1 0

[Glance] No weekly meeting
by Abhishek Kekane 23 Nov '21

23 Nov '21

Hello Team, As this is the holiday season and ThanksGiving is on our weekly meeting day, we are not meeting this week. Next meeting will be on 02 December. Thanks & Best Regards, Abhishek Kekane

1 0

[neutron] Neutron OVN+QoS Support
by Erlon Cruz 23 Nov '21

23 Nov '21

4 10

[Neutron][OVN] networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports failed
by Tony Liu 23 Nov '21

23 Nov '21

Hi, Is this known issue? Any filed bug or fix to it? ======================================= 2021-11-23 23:04:20.943 40 ERROR futurist.periodics [-] Failed to call periodic 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports' (it runs every 600.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2021-11-23 23:04:20.943 40 ERROR futurist.periodics Traceback (most recent call last): 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction 2021-11-23 23:04:20.943 40 ERROR futurist.periodics yield self._nested_txns_map[cur_thread_id] 2021-11-23 23:04:20.943 40 ERROR futurist.periodics KeyError: 140347734591640 2021-11-23 23:04:20.943 40 ERROR futurist.periodics 2021-11-23 23:04:20.943 40 ERROR futurist.periodics During handling of the above exception, another exception occurred: 2021-11-23 23:04:20.943 40 ERROR futurist.periodics 2021-11-23 23:04:20.943 40 ERROR futurist.periodics Traceback (most recent call last): 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 290, in run 2021-11-23 23:04:20.943 40 ERROR futurist.periodics work() 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 64, in __call__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 178, in decorator 2021-11-23 23:04:20.943 40 ERROR futurist.periodics return f(*args, **kwargs) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 667, in check_for_mcast_flood_reports 2021-11-23 23:04:20.943 40 ERROR futurist.periodics for port in self._nb_idl.lsp_list().execute(check_error=True): 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute 2021-11-23 23:04:20.943 40 ERROR futurist.periodics t.add(self) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics next(self.gen) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 183, in transaction 2021-11-23 23:04:20.943 40 ERROR futurist.periodics yield t 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics next(self.gen) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction 2021-11-23 23:04:20.943 40 ERROR futurist.periodics del self._nested_txns_map[cur_thread_id] 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics self.result = self.commit() 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2021-11-23 23:04:20.943 40 ERROR futurist.periodics raise result.ex 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run 2021-11-23 23:04:20.943 40 ERROR futurist.periodics txn.results.put(txn.do_commit()) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit 2021-11-23 23:04:20.943 40 ERROR futurist.periodics raise RuntimeError(msg) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2021-11-23 23:04:20.943 40 ERROR futurist.periodics ======================================= Thanks! Tony

2 2

[cinder] spec opportunities and deadline
by Brian Rosmaita 23 Nov '21

23 Nov '21

To anyone working on a cinder spec for Yoga: This is a reminder that all Cinder Specs for features to be implemented in Yoga must be approved by Friday 17 December 2021 (23:59 UTC). There are two upcoming opportunities to get feedback on your spec proposal and/or ask questions about your spec from the cinder team: 1. Tomorrow's cinder weekly meeting (Wednesday 24 November, 1400 UTC) is being held in videoconference; if you'd like a discussion, please put a topic on the weekly agenda (which also has connection details): https://etherpad.opendev.org/p/cinder-yoga-meetings 2. The cinder yoga R-17 virtual midcycle meeting is being held next week on Wednesday 1 December 1400-1600 UTC; if you'd like to discuss your proposal, please add it to the planning etherpad: https://etherpad.opendev.org/p/cinder-yoga-midcycles cheers, brian

1 0

[kolla-ansible][wallaby][glance] Problem with image list after reconfigure
by Franck VEDEL 23 Nov '21

23 Nov '21

Hello everyone I have a strange problem and I haven't found the solution yet. Following a certificate update I had to do a "kolla-ansible -t multinode reconfigure ». Well, after several attempts (it is not easy to use certificates with Kolla-ansible, and from my advice, not documented enough for beginners), I have my new functional certificates. Perfect ... well almost. I am trying to create a new instance to check general operation. ERROR. Okay, I look in the logs and I see that Cinder is having problems creating volumes with an error that I never had ("TypeError: 'NoneType' object is not iterable). I dig and then I wonder if it is not the Glance images which cannot be used, while they are present (openstack image list is OK). I create an empty volume: it works. I am creating a volume from an image: Failed. However, I have my list of ten images in glance. I create a new image and create a volume with this new image: it works. I create an instance with this new image: OK. What is the problem ? The images present before the "reconfigure" are listed, visible in horizon for example, but unusable. Is there a way to fix this, or do we have to reinstall them all? Thanks in advance for your help if this problem speaks to you. Franck VEDEL Dép. Réseaux Informatiques & Télécoms IUT1 - Univ GRENOBLE Alpes 0476824462 Stages, Alternance, Emploi. http://www.rtgrenoble.fr <http://www.rtgrenoble.fr/>

4 13

Cinder NFS Encryption
by Hitesh Mathur 23 Nov '21

23 Nov '21

Hi, I am not able to find whether Cinder support NFS data-in-transit encryption or not. Can you please provide the information on this and how to use it ? -- Regards Hitesh

2 1