November 2021 - openstack-discuss

[release] Release countdown for week R-18, Nov 22-26
by Előd Illés 25 Nov '21

25 Nov '21

Development Focus ---------------------------- We are now past the Yoga-1 milestone. Teams should currentlybe focused on feature development! General Information ---------------------------- Our next milestone in this development cycle will be Yoga-2, on 06 January, 2022. This milestone is when we freeze the list of deliverables that will be included in the Yogafinal release, so if you plan to introduce new deliverables in this release, please propose a change to add an empty deliverable file in the deliverables/yogadirectory of the openstack/releases repository. Now is also generally a good time to look at bugfixes that were introduced in the master branch that might make sense to be backported and released in a stable release. If you have any question around the OpenStack release process, feel free to ask on this mailing-list or on the #openstack-release channel on IRC. Upcoming Deadlines & Dates ----------------------------------------- Yoga-2 Milestone:06 January, 2022 Yoga final release: March 30, 2022 Előd Illés irc: elodilles

1 0

[kolla] Proposing Michal Arbet as Kolla core
by Michał Nasiadka 25 Nov '21

25 Nov '21

Hi, I would like to propose adding Michal Arbet (kevko) to the kolla-core and kolla-ansible-core groups. Michal did some great work on ProxySQL, is a consistent maintainer of Debian related images and has provided some helpful reviews. Cores - please reply +1/-1 before the end of Friday 26th November. Thanks, Michal

5 5

[neutron] Averaga number of rechecks in Neutron comparing to other projects
by Slawek Kaplonski 25 Nov '21

25 Nov '21

Hi neutrinos, As we discussed during the last PTG, I spent some time today to get average number of rechecks which we need to do in the last PS of the change before it's merged. In theory that number should be close to 0 as patch should be merged with first CI run when it's approved by reviewers :) All data are only from the master branch. I didn't check the same for stable branches. File with graph and raw data in csv format are in the attachments. Basically my conclusion is that Neutron's CI is really bad in that. We have to recheck many, many times before patches will be merged. We really need to think about how to improve that in Neutron. -- Slawek Kaplonski Principal Software Engineer Red Hat

2 2

[Triple 0] Undercloud deployment Getting failed
by Lokendra Rathour 25 Nov '21

25 Nov '21

Hi Team getting strange error when installing triple O Train on Centos 8.4 '--volume', '/var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--rm', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/container-puppet-zaqar.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', 'quay.io/tripleotraincentos8/centos-binary-zaqar-wsgi:current-tripleo'] run failed after Error: container_linux.go:370: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied", " attempt(s): 3", "2021-11-23 11:00:38,384 WARNING: 58791 -- Retrying running container: zaqar", "2021-11-23 11:00:38,384 ERROR: 58791 -- Failed running container for zaqar", "2021-11-23 11:00:38,385 INFO: 58791 -- Finished processing puppet configs for zaqar", "2021-11-23 11:00:38,385 ERROR: 58782 -- ERROR configuring crond", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring glance_api", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring heat_api", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring heat", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring ironic_api", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring ironic", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring ironic_inspector", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring neutron", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring iscsid", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring keystone", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring memcached", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring mistral", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring mysql", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring nova", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring rabbitmq", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring placement", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring swift", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring swift_ringbuilder", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring zaqar" ], "stderr_lines": [], "_ansible_no_log": false, "attempts": 15 } ] ] Not cleaning working directory /home/stack/tripleo-heat-installer-templates Not cleaning ansible directory /home/stack/undercloud-ansible-mie5k51_ Install artifact is located at /home/stack/undercloud-install-20211123110040.tar.bzip2 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Deployment Failed! This issue is recurring multiple times please advise. -- ~ Lokendra

2 2

[all][tc] Technical Committee next weekly meeting on Nov 25th at 1500 UTC
by Ghanshyam Mann 25 Nov '21

25 Nov '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Nov 25th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Nov 24th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[ops][nova][victoria] QEMU/KVM libvirt Authentication Failed
by DHilsbos＠performair.com 25 Nov '21

25 Nov '21

All; I feel like I've dealt with this issue before, but I can't find any records of it. I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again. I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1]. When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com 1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#…

3 6

[all][tc] Yoga testing runtime update. Action needed if py3.9 job failing for your project
by Ghanshyam Mann 25 Nov '21

25 Nov '21

Hello Everyone. As discussed in TC PTG, we have updated the testing runtime for the Yoga cycle [1]. Changes are: * Add Debian 11 as the tested distro * Change centos stream 8 -> centos stream 9 * Bump lowest python version to test to 3.8 and highest to python 3.9 ** This removes python 3.6 from testing. I pushed the job template update[2] which will make py3.9 unit test job voting (which is non-voting currently). I do not see any projects failing consistently on py3.9[3] but still, I will keep it as -W until early next week (23rd Nov). If any project needs time to fix the failing py3.9 job, please do it before 23rd Nov. [1] https://governance.openstack.org/tc/reference/runtimes/yoga.html [2] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/818609 [3] https://zuul.openstack.org/builds?job_name=openstack-tox-py39&branch=master… -gmann

1 1

[keystone][policy][ussuri] why I can create a domain
by Piotr Misiak 24 Nov '21

24 Nov '21

Hi, Maybe a stupid question but I'm really confused. In my Ussuri cloud Keystone has a following policy for create_domain action (this is a default policy from Keystone code): "identity:create_domain": "role:admin and system_scope:all" I have a user which has "admin" role assigned in project "admin" in domain "default" - AKA cloud admin. The user does not have any roles assigned on system scope. Could someone please explain why this user is able to create a domain in the cloud? Looking at the policy rule he shouldn't or maybe I'm reading it in a wrong way? Is there any "backward compatibility" casting "cloud admin" role to "system_scope:all"? Please help Thanks Piotr

2 1

[largescale-sig] Next meeting: Nov 24th, 15utc
by Thierry Carrez 24 Nov '21

24 Nov '21

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20211124T15 We will be discussing our upcoming OpenInfra Live! episode on Dec 9. Feel free to add topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[all][sphinx] "rST localisation for language "foo" not found" error
by Stephen Finucane 24 Nov '21

24 Nov '21

We've seen issues in some projects, whereby attempting to build translations as part of the 'openstack-tox-docs' zuul job [1] can result in the following warning: Warning, treated as error: ../doc/source/contributor/contributing.rst::rST localisation for language "id" not found. Based on [2], this issue appears to have been introduced by a recent version bump of docutils, which was in turn introduced by the introduction of Sphinx 4.x. We don't have a resolution for this bug yet but we'll update when we do. Hopefully this email will improve the "Google'ability" of this bug. Stephen [1] https://opendev.org/openstack/openstackdocstheme/src/commit/08461c5311aa692… [2] https://lists.freebsd.org/pipermail/freebsd-ports/2021-April/120848.html

2 2

[cinder] Bug deputy report for week of 11-24-2021
by Sofia Enriquez 24 Nov '21

24 Nov '21

This is a bug report from 11-17-2021 to 11-24-2021. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium The next 3 bugs are related to multipath config everywhere <https://review.opendev.org/c/openstack/cinder/+/818994>: - https://bugs.launchpad.net/cinder/+bug/1951982 "On encrypted volume cloning rekeying doesn't use multipathing". Assigned to Gorka Eguileor. - https://bugs.launchpad.net/cinder/+bug/1951981 "Kaminario not using multipath for creating volume". Assigned to Gorka Eguileor. - https://bugs.launchpad.net/cinder/+bug/1951977 "Cinder backup - The multipath flag is not set on a create_backup operation". Assigned to Gorka Eguileor. Low - https://bugs.launchpad.net/cinder/+bug/1951250 "[Storwize] Fix multiple lsiogrp,lsvdisk calls in Retype". Unassigned. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com> -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[kolla] Transition kolla/kolla-ansible Rocky/Stein to EOL
by Michał Nasiadka 24 Nov '21

24 Nov '21

Hello, We've kept stable/rocky and stable/stein branches of kolla and kolla-ansible around under extended maintenance for a while now. The coordinated Rocky release was 30th Aug, 2018 (Kolla on 22nd Oct, 2018) and Stein was 10th Apr, 2019 (Kolla on 17th Jul, 2019). The last time a change was backported to these branches has been a while: - kolla stable/rocky: Sep 4, 2020 stable/stein: Mar 19, 2021 - kolla-ansible stable/rocky: Sep 13, 2020 stable/stein: May 28,2021 There is no CI running for those projects periodically, last CI jobs have been run: - kolla: stable/rocky: Jul 8th 2019 stable/stein: Dec 10th 2020 - kolla-ansible stable/rocky: Sep 14th 2019 stable/stein: Dec 10th 2020 All changes to those branches have been abandoned long time (and nothing new proposed for at least a year): https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla <https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla> https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla <https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla> https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla-an… <https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla-an…> https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla-an… <https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla-an…> In addition to that - kolla-ansible CI jobs have incorrect Zuul configurations (see http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025797… <http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025797…>) - and on one of the Kolla weekly meetings it was decided to mark those branches as EOL. So, I'd like to request that these branches be moved to end-of-life. Patch proposed to add EOL tags to both kolla and kolla-ansible: https://review.opendev.org/c/openstack/releases/+/818920 <https://review.opendev.org/c/openstack/releases/+/818920> Best regards, Michal

1 0

[kolla][release][Release-job-failures] Release of openstack/kolla for ref refs/tags/10.4.0 failed
by Előd Illés 24 Nov '21

24 Nov '21

Hi Kolla team, The last release of kolla (10.4.0 [1]) had release job failures (see below the attached, forwarded mail). The two job that failed are: - kolla-publish-centos8-source-dockerhub https://zuul.opendev.org/t/openstack/build/feb51c87f74c4dadb4984733d8b3793c : FAILURE in 2h 12m 37s - kolla-publish-centos8-binary-dockerhub https://zuul.opendev.org/t/openstack/build/ebce7219ff4b44368869bc1f3172d688 : FAILURE in 1h 46m 56s (non-voting) Could you please have a look at these? Thanks in advance, Előd (elodilles @ #openstack-release) [1] https://review.opendev.org/c/openstack/releases/+/818882 -------- Forwarded Message -------- Subject: [Release-job-failures] Release of openstack/kolla for ref refs/tags/10.4.0 failed Date: Tue, 23 Nov 2021 17:22:02 +0000 From: zuul(a)openstack.org Reply-To: openstack-discuss(a)lists.openstack.org To: release-job-failures(a)lists.openstack.org Build failed. - openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/4834324fec464b309952930a2c221336 : SUCCESS in 1m 45s - release-openstack-python https://zuul.opendev.org/t/openstack/build/e3f27c5f3f9b4b289dc73570243507ff : SUCCESS in 3m 06s - announce-release https://zuul.opendev.org/t/openstack/build/1d759f048a1145af9b253b6e5d69b4f4 : SUCCESS in 4m 23s - propose-update-constraints https://zuul.opendev.org/t/openstack/build/3a600e6a4c494a829bfcd45c91931067 : SUCCESS in 3m 42s - kolla-publish-centos8-source-dockerhub https://zuul.opendev.org/t/openstack/build/feb51c87f74c4dadb4984733d8b3793c : FAILURE in 2h 12m 37s - kolla-publish-centos8-binary-dockerhub https://zuul.opendev.org/t/openstack/build/ebce7219ff4b44368869bc1f3172d688 : FAILURE in 1h 46m 56s (non-voting) - kolla-publish-debian-source-dockerhub https://zuul.opendev.org/t/openstack/build/e076910efb394683b98106d5648c2ab5 : SUCCESS in 1h 54m 47s (non-voting) - kolla-publish-debian-source-aarch64-dockerhub https://zuul.opendev.org/t/openstack/build/e201c6f797ea4248b933df8c53d65fa8 : SUCCESS in 2h 03m 51s (non-voting) - kolla-publish-debian-binary-dockerhub https://zuul.opendev.org/t/openstack/build/d0cf2b3aa48e49d8a5aa46719b84c926 : SUCCESS in 1h 18m 03s (non-voting) - kolla-publish-ubuntu-source-dockerhub https://zuul.opendev.org/t/openstack/build/17644441b45f4964b52a1e26e8ef3f19 : SUCCESS in 2h 01m 47s - kolla-publish-ubuntu-binary-dockerhub https://zuul.opendev.org/t/openstack/build/63d1810841da4d9c96d4755177761f4d : SUCCESS in 1h 34m 09s (non-voting) _______________________________________________ Release-job-failures mailing list Release-job-failures(a)lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures

1 0

[neutron] default QOS on L3 gateway
by Damien Rannou 24 Nov '21

24 Nov '21

Hello We are currently playing with QOS on L3 agent, mostly for SNAT, but it can apply also on FIP. Everything is working properly, but I’m wondering if there is a way to define a « default » QOS that would be applied on Router creation, but also when the user is setting « no_qos » on his router. On a public cloud environnement, we cannot let the customers without any QOS limitation. Thanks ! Damien

3 3

[kolla-ansible][wallaby][magnum][Kubernetes] Cannot auto-scale workers
by wodel youchi 24 Nov '21

24 Nov '21

Hi, I have a new kolla-ansible deployment with wallaby. I have created a kubernetes cluster using calico (flannel didn't work for me). I configured an autoscale test to see if it works. - pods autoscale is working. - worker nodes autoscale is not working. This is my deployment file :*cat php-apache.yaml* apiVersion: apps/v1 kind: Deployment metadata: name: php-apache-deployment spec: selector: matchLabels: app: php-apache replicas: 2 template: metadata: labels: app: php-apache spec: containers: - name: php-apache image: k8s.gcr.io/hpa-example ports: - containerPort: 80 resources: limits: cpu: 500m requests: cpu: 200m --- apiVersion: v1 kind: Service metadata: name: php-apache-service labels: app: php-apache spec: ports: - port: 80 targetPort: 80 protocol: TCP selector: app: php-apache type: LoadBalancer This is my HPA file :*cat php-apache-hpa.yaml* apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: php-apache-hpa namespace: default labels: service: php-apache-service spec: minReplicas: 2 maxReplicas: 30 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: php-apache-deployment metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 30 # en pourcentage This is my load program : kubectl run -i --tty load-generator-1 --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://ip_load_balancer; done" Here are the output of my kub cluster before the test : [kube8@cdndeployer ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE php-apache-deployment-5b65bbc75c-95k6k 1/1 Running 0 24m php-apache-deployment-5b65bbc75c-mv5h6 1/1 Running 0 24m [kube8@cdndeployer ~]$ kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE php-apache-hpa Deployment/php-apache-deployment *0%/30%* 2 15 2 24m [kube8@cdndeployer ~]$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 13h php-apache-service LoadBalancer 10.254.3.54 *xx.xx.xx.213* 80:31763/TCP 25m When I apply the load : - pods autoscale creates new pods, then some of them get in the state of : *pending * [kube8@cdndeployer ~]$ kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE php-apache-hpa Deployment/php-apache-deployment *155%/30%* 2 15 4 27m [kube8@cdndeployer ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE load-generator-1 1/1 Running 0 97s load-generator-2 1/1 Running 0 94s php-apache-deployment-5b65bbc75c-95k6k 1/1 Running 0 28m *php-apache-deployment-5b65bbc75c-cjkwk 0/1 Pending 0 33s* *php-apache-deployment-5b65bbc75c-cn5rt 0/1 Pending 0 33s* *php-apache-deployment-5b65bbc75c-cxctx 0/1 Pending 0 48s* php-apache-deployment-5b65bbc75c-fffnc 1/1 Running 0 64s php-apache-deployment-5b65bbc75c-hbfw8 0/1 Pending 0 33s php-apache-deployment-5b65bbc75c-l8496 1/1 Running 0 48s php-apache-deployment-5b65bbc75c-mv5h6 1/1 Running 0 28m php-apache-deployment-5b65bbc75c-qddrb 1/1 Running 0 48s php-apache-deployment-5b65bbc75c-dd5r5 0/1 Pending 0 48s php-apache-deployment-5b65bbc75c-tr65j 1/1 Running 0 64s 2 - The cluster is unable to create more pods/workers and I get this error message from the pending pods kubectl describe pod php-apache-deployment-5b65bbc75c-dd5r5 Name: php-apache-deployment-5b65bbc75c-dd5r5 Namespace: default Priority: 0 Node: <none> Labels: app=php-apache pod-template-hash=5b65bbc75c Annotations: kubernetes.io/psp: magnum.privileged *Status: Pending* IP: IPs: <none> Controlled By: ReplicaSet/php-apache-deployment-5b65bbc75c Containers: php-apache: Image: k8s.gcr.io/hpa-example Port: 80/TCP Host Port: 0/TCP Limits: cpu: 500m Requests: cpu: 200m Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-4fsgh (ro) Conditions: Type Status PodScheduled False Volumes: default-token-4fsgh: Type: Secret (a volume populated by a Secret) SecretName: default-token-4fsgh Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- * Warning FailedScheduling 2m48s default-scheduler 0/4 nodes are available: 2 Insufficient cpu, 2 node(s) had taint {node-role.kubernetes.io/master <http://node-role.kubernetes.io/master>: }, that the pod didn't tolerate.* * Warning FailedScheduling 2m48s default-scheduler 0/4 nodes are available: 2 Insufficient cpu, 2 node(s) had taint {node-role.kubernetes.io/master <http://node-role.kubernetes.io/master>: }, that the pod didn't tolerate.* * Normal NotTriggerScaleUp 2m42s cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 in backoff after failed scale-u**p* I have this error message from the autoscaller pod *cluster-autoscaler*-f4bd5f674-b9692 : I1123 00:50:27.714801 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.709µs I1123 00:51:34.181145 1 scale_up.go:658] Scale-up: setting group default-worker size to 3 *W1123 00:51:34.381953 1 clusterstate.go:281] Disabling scale-up for node group default-worker until 2021-11-23 00:56:34.180840351 +0000 UTC m=+47174.376164120; errorClass=Other; errorCode=cloudProviderError* *E1123 00:51:34.382081 1 static_autoscaler.go:415] Failed to scale up: failed to increase node group size: could not check current nodegroup size: could not get cluster: Get https://dash.cdn.domaine.tld:9511/v1/clusters/b4a6b3eb-fcf3-416f-b740-11a08… <https://dash.cdn.cerist.dz:9511/v1/clusters/b4a6b3eb-fcf3-416f-b740-11a083d…>: dial tcp: lookup dash.cdn.domaine. <http://dash.cdn.cerist.dz>tld on 10.254.0.10:53 <http://10.254.0.10:53>: no such host* W1123 00:51:44.392523 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:51:54.410273 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:52:04.422128 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:52:14.434278 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:52:24.442480 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff I1123 00:52:27.715019 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I did some tests on the DNS pod and : kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 13h default php-apache-service LoadBalancer 10.254.3.54 xx.xx.xx.213 80:31763/TCP 19m kube-system dashboard-metrics-scraper ClusterIP 10.254.19.191 <none> 8000/TCP 13h *kube-system kube-dns ClusterIP 10.254.0.10 <none> 53/UDP,53/TCP,9153/TCP 13h* kube-system kubernetes-dashboard ClusterIP 10.254.132.17 <none> 443/TCP 13h kube-system magnum-metrics-server ClusterIP 10.254.235.147 <none> 443/TCP 13h I have noticed this behaviour about the horizon url, sometimes the dns pod responds sometimes it does not !!!!! [root@k8multiclustercalico-ve5t6uuoo245-master-0 ~]# *dig @10.254.0.10 <http://10.254.0.10> dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld* ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc33 <<>> @10.254.0.10 dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 5646 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;dash.cd <http://dash.cdn.cerist.dz>n.domaine.tld. IN A ;; AUTHORITY SECTION: *cdn. <http://cdn.cerist.dz>domaine.tld. 30 IN SOA cdn. <http://cdn.cerist.dz>domaine.tld. root.cdn. <http://root.cdn.cerist.dz>domaine.tld. 2021100900 604800 86400 2419200 604800* ;; Query time: 84 msec ;; SERVER: 10.254.0.10#53(10.254.0.10) ;; WHEN: Tue Nov 23 01:08:03 UTC 2021 ;; MSG SIZE rcvd: 12 2 secondes later [root@k8multiclustercalico-ve5t6uuoo245-master-0 ~]# *dig @10.254.0.10 <http://10.254.0.10> dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld* ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc33 <<>> @10.254.0.10 dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7653 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld. IN A ;; ANSWER SECTION: *dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld. 30 IN A xx.xx.xx.129* ;; Query time: 2 msec ;; SERVER: 10.254.0.10#53(10.254.0.10) ;; WHEN: Tue Nov 23 01:08:21 UTC 2021 ;; MSG SIZE rcvd: 81 In the log of the dns pod I have this kubectl logs *kube-dns-autoscaler*-75859754fd-q8z4w -n kube-system *E1122 20:56:09.944449 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource E1122 20:56:19.945294 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource E1122 20:56:29.944245 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource E1122 20:56:39.946346 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource* *E1122 20:56:49.944693 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource* I don't have experience on kubernetes yet, could someone help me debug this? Regards.

1 0

[horizon] Cancelling Today's weekly meeting
by vishal manchanda 24 Nov '21

24 Nov '21

Hello Team, Since there are no agenda items [1] to discuss for today's horizon weekly meeting. Also, Today is a holiday for me or maybe for others as well So let's cancel today's weekly meeting. Next weekly meeting will be on 01 December. Thanks & Regards, Vishal Manchanda [1] https://etherpad.opendev.org/p/horizon-release-priorities

1 0

[Glance] No weekly meeting
by Abhishek Kekane 24 Nov '21

24 Nov '21

Hello Team, As this is the holiday season and ThanksGiving is on our weekly meeting day, we are not meeting this week. Next meeting will be on 02 December. Thanks & Best Regards, Abhishek Kekane

1 0

[neutron] Neutron OVN+QoS Support
by Erlon Cruz 24 Nov '21

24 Nov '21

4 10

[Neutron][OVN] networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports failed
by Tony Liu 24 Nov '21

24 Nov '21

Hi, Is this known issue? Any filed bug or fix to it? ======================================= 2021-11-23 23:04:20.943 40 ERROR futurist.periodics [-] Failed to call periodic 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports' (it runs every 600.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2021-11-23 23:04:20.943 40 ERROR futurist.periodics Traceback (most recent call last): 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction 2021-11-23 23:04:20.943 40 ERROR futurist.periodics yield self._nested_txns_map[cur_thread_id] 2021-11-23 23:04:20.943 40 ERROR futurist.periodics KeyError: 140347734591640 2021-11-23 23:04:20.943 40 ERROR futurist.periodics 2021-11-23 23:04:20.943 40 ERROR futurist.periodics During handling of the above exception, another exception occurred: 2021-11-23 23:04:20.943 40 ERROR futurist.periodics 2021-11-23 23:04:20.943 40 ERROR futurist.periodics Traceback (most recent call last): 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 290, in run 2021-11-23 23:04:20.943 40 ERROR futurist.periodics work() 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 64, in __call__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 178, in decorator 2021-11-23 23:04:20.943 40 ERROR futurist.periodics return f(*args, **kwargs) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 667, in check_for_mcast_flood_reports 2021-11-23 23:04:20.943 40 ERROR futurist.periodics for port in self._nb_idl.lsp_list().execute(check_error=True): 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute 2021-11-23 23:04:20.943 40 ERROR futurist.periodics t.add(self) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics next(self.gen) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 183, in transaction 2021-11-23 23:04:20.943 40 ERROR futurist.periodics yield t 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics next(self.gen) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction 2021-11-23 23:04:20.943 40 ERROR futurist.periodics del self._nested_txns_map[cur_thread_id] 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics self.result = self.commit() 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2021-11-23 23:04:20.943 40 ERROR futurist.periodics raise result.ex 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run 2021-11-23 23:04:20.943 40 ERROR futurist.periodics txn.results.put(txn.do_commit()) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit 2021-11-23 23:04:20.943 40 ERROR futurist.periodics raise RuntimeError(msg) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2021-11-23 23:04:20.943 40 ERROR futurist.periodics ======================================= Thanks! Tony

2 2

[cinder] spec opportunities and deadline
by Brian Rosmaita 24 Nov '21

24 Nov '21

To anyone working on a cinder spec for Yoga: This is a reminder that all Cinder Specs for features to be implemented in Yoga must be approved by Friday 17 December 2021 (23:59 UTC). There are two upcoming opportunities to get feedback on your spec proposal and/or ask questions about your spec from the cinder team: 1. Tomorrow's cinder weekly meeting (Wednesday 24 November, 1400 UTC) is being held in videoconference; if you'd like a discussion, please put a topic on the weekly agenda (which also has connection details): https://etherpad.opendev.org/p/cinder-yoga-meetings 2. The cinder yoga R-17 virtual midcycle meeting is being held next week on Wednesday 1 December 1400-1600 UTC; if you'd like to discuss your proposal, please add it to the planning etherpad: https://etherpad.opendev.org/p/cinder-yoga-midcycles cheers, brian

1 0

[kolla-ansible][wallaby][glance] Problem with image list after reconfigure
by Franck VEDEL 23 Nov '21

23 Nov '21

Hello everyone I have a strange problem and I haven't found the solution yet. Following a certificate update I had to do a "kolla-ansible -t multinode reconfigure ». Well, after several attempts (it is not easy to use certificates with Kolla-ansible, and from my advice, not documented enough for beginners), I have my new functional certificates. Perfect ... well almost. I am trying to create a new instance to check general operation. ERROR. Okay, I look in the logs and I see that Cinder is having problems creating volumes with an error that I never had ("TypeError: 'NoneType' object is not iterable). I dig and then I wonder if it is not the Glance images which cannot be used, while they are present (openstack image list is OK). I create an empty volume: it works. I am creating a volume from an image: Failed. However, I have my list of ten images in glance. I create a new image and create a volume with this new image: it works. I create an instance with this new image: OK. What is the problem ? The images present before the "reconfigure" are listed, visible in horizon for example, but unusable. Is there a way to fix this, or do we have to reinstall them all? Thanks in advance for your help if this problem speaks to you. Franck VEDEL Dép. Réseaux Informatiques & Télécoms IUT1 - Univ GRENOBLE Alpes 0476824462 Stages, Alternance, Emploi. http://www.rtgrenoble.fr <http://www.rtgrenoble.fr/>

4 13

Cinder NFS Encryption
by Hitesh Mathur 23 Nov '21

23 Nov '21

Hi, I am not able to find whether Cinder support NFS data-in-transit encryption or not. Can you please provide the information on this and how to use it ? -- Regards Hitesh

2 1

[magnum] [victoria] [fedora-coreos]
by Namrata Sitlani 23 Nov '21

23 Nov '21

Hello All, We run release Victoria(11.1.1) and we deployed Kubernetes version 1.18.20 on Magnum with Fedora CoreOS version 33 and recently we ran into issues with Cinder CSI plugin. Multiple master nodes break the CSI plugin, but if I use a single master everything works fine. In my understanding it could be related to Fedora CoreOS version. It is not clear which versions are supported. Can somebody give me information on which version is supported? Thanks, Namrata Sitlani

1 0

[openstack-helm] No Meeting Tomorrow
by Gage Hugo 23 Nov '21

23 Nov '21

Hey team, Since there are no agenda items [0] for the IRC meeting tomorrow, the meeting is cancelled. Our next meeting will be November 30th. [0] https://etherpad.opendev.org/p/openstack-helm-weekly-meeting

1 0

[all][refstack][neutron][kolla][ironic][heat][trove][senlin][barbican][manila] Fixing Zuul Config Errors
by Clark Boylan 23 Nov '21

23 Nov '21

Hello, The OpenStack tenant in Zuul currently has 134 config errors. You can find these errors at https://zuul.opendev.org/t/openstack/config-errors or by clicking the blue bell icon in the top right of https://zuul.opendev.org/t/openstack/status. The vast majority of these errors appear related to project renames that have been requested of OpenDev or project retirements. Can you please look into fixing these as they can be an attractive nuisance when debugging Zuul problems (they also indicate that a number of your jobs are probably not working). Project renames creating issues: * openstack/python-tempestconf -> osf/python-tempestconf -> openinfra/python-tempestconf * openstack/refstack -> osf/refstack -> openinfra/refstack * x/tap-as-a-service -> openstack/tap-as-a-service * openstack/networking-l2gw -> x/networking-l2gw Project retirements creating issues: * openstack/neutron-lbaas * recordsansible/ara Projects whose configs have errors: * openinfra/python-tempestconf * openstack/heat * openstack/ironic * openstack/kolla-ansible * openstack/kuryr-kubernetes * openstack/murano-apps * openstack/networking-midonet * openstack/networking-odl * openstack/neutron * openstack/neutron-fwaas * openstack/python-troveclient * openstack/senlin * openstack/tap-as-a-service * openstack/zaqar * x/vmware-nsx * openinfra/openstackid * openstack/barbican * openstack/cookbook-openstack-application-catalog * openstack/heat-dashboard * openstack/manila-ui * openstack/python-manilaclient Let us know if we can help decipher any errors, Clark

6 8