- openstack-discuss - lists.openstack.org

[neutron] Drivers meeting - Friday 26.11.2021 - cancelled
by Lajos Katona 25 Nov '21

25 Nov '21

Hi Neutron Drivers! Due to the lack of agenda, let's cancel tomorrow's drivers meeting. See You on the meeting next week. Lajos Katona (lajoskatona)

1 0

[release] Release countdown for week R-18, Nov 22-26
by Előd Illés 25 Nov '21

25 Nov '21

Development Focus ---------------------------- We are now past the Yoga-1 milestone. Teams should currentlybe focused on feature development! General Information ---------------------------- Our next milestone in this development cycle will be Yoga-2, on 06 January, 2022. This milestone is when we freeze the list of deliverables that will be included in the Yogafinal release, so if you plan to introduce new deliverables in this release, please propose a change to add an empty deliverable file in the deliverables/yogadirectory of the openstack/releases repository. Now is also generally a good time to look at bugfixes that were introduced in the master branch that might make sense to be backported and released in a stable release. If you have any question around the OpenStack release process, feel free to ask on this mailing-list or on the #openstack-release channel on IRC. Upcoming Deadlines & Dates ----------------------------------------- Yoga-2 Milestone:06 January, 2022 Yoga final release: March 30, 2022 Előd Illés irc: elodilles

1 0

[kolla] Proposing Michal Arbet as Kolla core
by Michał Nasiadka 25 Nov '21

25 Nov '21

Hi, I would like to propose adding Michal Arbet (kevko) to the kolla-core and kolla-ansible-core groups. Michal did some great work on ProxySQL, is a consistent maintainer of Debian related images and has provided some helpful reviews. Cores - please reply +1/-1 before the end of Friday 26th November. Thanks, Michal

5 5

[neutron] Averaga number of rechecks in Neutron comparing to other projects
by Slawek Kaplonski 25 Nov '21

25 Nov '21

Hi neutrinos, As we discussed during the last PTG, I spent some time today to get average number of rechecks which we need to do in the last PS of the change before it's merged. In theory that number should be close to 0 as patch should be merged with first CI run when it's approved by reviewers :) All data are only from the master branch. I didn't check the same for stable branches. File with graph and raw data in csv format are in the attachments. Basically my conclusion is that Neutron's CI is really bad in that. We have to recheck many, many times before patches will be merged. We really need to think about how to improve that in Neutron. -- Slawek Kaplonski Principal Software Engineer Red Hat

2 2

[Triple 0] Undercloud deployment Getting failed
by Lokendra Rathour 25 Nov '21

25 Nov '21

Hi Team getting strange error when installing triple O Train on Centos 8.4 '--volume', '/var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--rm', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/container-puppet-zaqar.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', 'quay.io/tripleotraincentos8/centos-binary-zaqar-wsgi:current-tripleo'] run failed after Error: container_linux.go:370: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied", " attempt(s): 3", "2021-11-23 11:00:38,384 WARNING: 58791 -- Retrying running container: zaqar", "2021-11-23 11:00:38,384 ERROR: 58791 -- Failed running container for zaqar", "2021-11-23 11:00:38,385 INFO: 58791 -- Finished processing puppet configs for zaqar", "2021-11-23 11:00:38,385 ERROR: 58782 -- ERROR configuring crond", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring glance_api", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring heat_api", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring heat", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring ironic_api", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring ironic", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring ironic_inspector", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring neutron", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring iscsid", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring keystone", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring memcached", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring mistral", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring mysql", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring nova", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring rabbitmq", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring placement", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring swift", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring swift_ringbuilder", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring zaqar" ], "stderr_lines": [], "_ansible_no_log": false, "attempts": 15 } ] ] Not cleaning working directory /home/stack/tripleo-heat-installer-templates Not cleaning ansible directory /home/stack/undercloud-ansible-mie5k51_ Install artifact is located at /home/stack/undercloud-install-20211123110040.tar.bzip2 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Deployment Failed! This issue is recurring multiple times please advise. -- ~ Lokendra

2 2

[all][tc] Technical Committee next weekly meeting on Nov 25th at 1500 UTC
by Ghanshyam Mann 24 Nov '21

24 Nov '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Nov 25th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Nov 24th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[ops][nova][victoria] QEMU/KVM libvirt Authentication Failed
by DHilsbos＠performair.com 24 Nov '21

24 Nov '21

All; I feel like I've dealt with this issue before, but I can't find any records of it. I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again. I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1]. When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com 1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#…

3 6

[all][tc] Yoga testing runtime update. Action needed if py3.9 job failing for your project
by Ghanshyam Mann 24 Nov '21

24 Nov '21

Hello Everyone. As discussed in TC PTG, we have updated the testing runtime for the Yoga cycle [1]. Changes are: * Add Debian 11 as the tested distro * Change centos stream 8 -> centos stream 9 * Bump lowest python version to test to 3.8 and highest to python 3.9 ** This removes python 3.6 from testing. I pushed the job template update[2] which will make py3.9 unit test job voting (which is non-voting currently). I do not see any projects failing consistently on py3.9[3] but still, I will keep it as -W until early next week (23rd Nov). If any project needs time to fix the failing py3.9 job, please do it before 23rd Nov. [1] https://governance.openstack.org/tc/reference/runtimes/yoga.html [2] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/818609 [3] https://zuul.openstack.org/builds?job_name=openstack-tox-py39&branch=master… -gmann

1 1

[keystone][policy][ussuri] why I can create a domain
by Piotr Misiak 24 Nov '21

24 Nov '21

Hi, Maybe a stupid question but I'm really confused. In my Ussuri cloud Keystone has a following policy for create_domain action (this is a default policy from Keystone code): "identity:create_domain": "role:admin and system_scope:all" I have a user which has "admin" role assigned in project "admin" in domain "default" - AKA cloud admin. The user does not have any roles assigned on system scope. Could someone please explain why this user is able to create a domain in the cloud? Looking at the policy rule he shouldn't or maybe I'm reading it in a wrong way? Is there any "backward compatibility" casting "cloud admin" role to "system_scope:all"? Please help Thanks Piotr

2 1

[largescale-sig] Next meeting: Nov 24th, 15utc
by Thierry Carrez 24 Nov '21

24 Nov '21

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20211124T15 We will be discussing our upcoming OpenInfra Live! episode on Dec 9. Feel free to add topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[all][sphinx] "rST localisation for language "foo" not found" error
by Stephen Finucane 24 Nov '21

24 Nov '21

We've seen issues in some projects, whereby attempting to build translations as part of the 'openstack-tox-docs' zuul job [1] can result in the following warning: Warning, treated as error: ../doc/source/contributor/contributing.rst::rST localisation for language "id" not found. Based on [2], this issue appears to have been introduced by a recent version bump of docutils, which was in turn introduced by the introduction of Sphinx 4.x. We don't have a resolution for this bug yet but we'll update when we do. Hopefully this email will improve the "Google'ability" of this bug. Stephen [1] https://opendev.org/openstack/openstackdocstheme/src/commit/08461c5311aa692… [2] https://lists.freebsd.org/pipermail/freebsd-ports/2021-April/120848.html

2 2

[cinder] Bug deputy report for week of 11-24-2021
by Sofia Enriquez 24 Nov '21

24 Nov '21

This is a bug report from 11-17-2021 to 11-24-2021. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium The next 3 bugs are related to multipath config everywhere <https://review.opendev.org/c/openstack/cinder/+/818994>: - https://bugs.launchpad.net/cinder/+bug/1951982 "On encrypted volume cloning rekeying doesn't use multipathing". Assigned to Gorka Eguileor. - https://bugs.launchpad.net/cinder/+bug/1951981 "Kaminario not using multipath for creating volume". Assigned to Gorka Eguileor. - https://bugs.launchpad.net/cinder/+bug/1951977 "Cinder backup - The multipath flag is not set on a create_backup operation". Assigned to Gorka Eguileor. Low - https://bugs.launchpad.net/cinder/+bug/1951250 "[Storwize] Fix multiple lsiogrp,lsvdisk calls in Retype". Unassigned. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com> -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[kolla] Transition kolla/kolla-ansible Rocky/Stein to EOL
by Michał Nasiadka 24 Nov '21

24 Nov '21

Hello, We've kept stable/rocky and stable/stein branches of kolla and kolla-ansible around under extended maintenance for a while now. The coordinated Rocky release was 30th Aug, 2018 (Kolla on 22nd Oct, 2018) and Stein was 10th Apr, 2019 (Kolla on 17th Jul, 2019). The last time a change was backported to these branches has been a while: - kolla stable/rocky: Sep 4, 2020 stable/stein: Mar 19, 2021 - kolla-ansible stable/rocky: Sep 13, 2020 stable/stein: May 28,2021 There is no CI running for those projects periodically, last CI jobs have been run: - kolla: stable/rocky: Jul 8th 2019 stable/stein: Dec 10th 2020 - kolla-ansible stable/rocky: Sep 14th 2019 stable/stein: Dec 10th 2020 All changes to those branches have been abandoned long time (and nothing new proposed for at least a year): https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla <https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla> https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla <https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla> https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla-an… <https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla-an…> https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla-an… <https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla-an…> In addition to that - kolla-ansible CI jobs have incorrect Zuul configurations (see http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025797… <http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025797…>) - and on one of the Kolla weekly meetings it was decided to mark those branches as EOL. So, I'd like to request that these branches be moved to end-of-life. Patch proposed to add EOL tags to both kolla and kolla-ansible: https://review.opendev.org/c/openstack/releases/+/818920 <https://review.opendev.org/c/openstack/releases/+/818920> Best regards, Michal

1 0

[kolla][release][Release-job-failures] Release of openstack/kolla for ref refs/tags/10.4.0 failed
by Előd Illés 24 Nov '21

24 Nov '21

Hi Kolla team, The last release of kolla (10.4.0 [1]) had release job failures (see below the attached, forwarded mail). The two job that failed are: - kolla-publish-centos8-source-dockerhub https://zuul.opendev.org/t/openstack/build/feb51c87f74c4dadb4984733d8b3793c : FAILURE in 2h 12m 37s - kolla-publish-centos8-binary-dockerhub https://zuul.opendev.org/t/openstack/build/ebce7219ff4b44368869bc1f3172d688 : FAILURE in 1h 46m 56s (non-voting) Could you please have a look at these? Thanks in advance, Előd (elodilles @ #openstack-release) [1] https://review.opendev.org/c/openstack/releases/+/818882 -------- Forwarded Message -------- Subject: [Release-job-failures] Release of openstack/kolla for ref refs/tags/10.4.0 failed Date: Tue, 23 Nov 2021 17:22:02 +0000 From: zuul(a)openstack.org Reply-To: openstack-discuss(a)lists.openstack.org To: release-job-failures(a)lists.openstack.org Build failed. - openstack-upload-github-mirror https://zuul.opendev.org/t/openstack/build/4834324fec464b309952930a2c221336 : SUCCESS in 1m 45s - release-openstack-python https://zuul.opendev.org/t/openstack/build/e3f27c5f3f9b4b289dc73570243507ff : SUCCESS in 3m 06s - announce-release https://zuul.opendev.org/t/openstack/build/1d759f048a1145af9b253b6e5d69b4f4 : SUCCESS in 4m 23s - propose-update-constraints https://zuul.opendev.org/t/openstack/build/3a600e6a4c494a829bfcd45c91931067 : SUCCESS in 3m 42s - kolla-publish-centos8-source-dockerhub https://zuul.opendev.org/t/openstack/build/feb51c87f74c4dadb4984733d8b3793c : FAILURE in 2h 12m 37s - kolla-publish-centos8-binary-dockerhub https://zuul.opendev.org/t/openstack/build/ebce7219ff4b44368869bc1f3172d688 : FAILURE in 1h 46m 56s (non-voting) - kolla-publish-debian-source-dockerhub https://zuul.opendev.org/t/openstack/build/e076910efb394683b98106d5648c2ab5 : SUCCESS in 1h 54m 47s (non-voting) - kolla-publish-debian-source-aarch64-dockerhub https://zuul.opendev.org/t/openstack/build/e201c6f797ea4248b933df8c53d65fa8 : SUCCESS in 2h 03m 51s (non-voting) - kolla-publish-debian-binary-dockerhub https://zuul.opendev.org/t/openstack/build/d0cf2b3aa48e49d8a5aa46719b84c926 : SUCCESS in 1h 18m 03s (non-voting) - kolla-publish-ubuntu-source-dockerhub https://zuul.opendev.org/t/openstack/build/17644441b45f4964b52a1e26e8ef3f19 : SUCCESS in 2h 01m 47s - kolla-publish-ubuntu-binary-dockerhub https://zuul.opendev.org/t/openstack/build/63d1810841da4d9c96d4755177761f4d : SUCCESS in 1h 34m 09s (non-voting) _______________________________________________ Release-job-failures mailing list Release-job-failures(a)lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures

1 0

[neutron] default QOS on L3 gateway
by Damien Rannou 24 Nov '21

24 Nov '21

Hello We are currently playing with QOS on L3 agent, mostly for SNAT, but it can apply also on FIP. Everything is working properly, but I’m wondering if there is a way to define a « default » QOS that would be applied on Router creation, but also when the user is setting « no_qos » on his router. On a public cloud environnement, we cannot let the customers without any QOS limitation. Thanks ! Damien

3 3

[kolla-ansible][wallaby][magnum][Kubernetes] Cannot auto-scale workers
by wodel youchi 24 Nov '21

24 Nov '21

Hi, I have a new kolla-ansible deployment with wallaby. I have created a kubernetes cluster using calico (flannel didn't work for me). I configured an autoscale test to see if it works. - pods autoscale is working. - worker nodes autoscale is not working. This is my deployment file :*cat php-apache.yaml* apiVersion: apps/v1 kind: Deployment metadata: name: php-apache-deployment spec: selector: matchLabels: app: php-apache replicas: 2 template: metadata: labels: app: php-apache spec: containers: - name: php-apache image: k8s.gcr.io/hpa-example ports: - containerPort: 80 resources: limits: cpu: 500m requests: cpu: 200m --- apiVersion: v1 kind: Service metadata: name: php-apache-service labels: app: php-apache spec: ports: - port: 80 targetPort: 80 protocol: TCP selector: app: php-apache type: LoadBalancer This is my HPA file :*cat php-apache-hpa.yaml* apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: php-apache-hpa namespace: default labels: service: php-apache-service spec: minReplicas: 2 maxReplicas: 30 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: php-apache-deployment metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 30 # en pourcentage This is my load program : kubectl run -i --tty load-generator-1 --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://ip_load_balancer; done" Here are the output of my kub cluster before the test : [kube8@cdndeployer ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE php-apache-deployment-5b65bbc75c-95k6k 1/1 Running 0 24m php-apache-deployment-5b65bbc75c-mv5h6 1/1 Running 0 24m [kube8@cdndeployer ~]$ kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE php-apache-hpa Deployment/php-apache-deployment *0%/30%* 2 15 2 24m [kube8@cdndeployer ~]$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 13h php-apache-service LoadBalancer 10.254.3.54 *xx.xx.xx.213* 80:31763/TCP 25m When I apply the load : - pods autoscale creates new pods, then some of them get in the state of : *pending * [kube8@cdndeployer ~]$ kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE php-apache-hpa Deployment/php-apache-deployment *155%/30%* 2 15 4 27m [kube8@cdndeployer ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE load-generator-1 1/1 Running 0 97s load-generator-2 1/1 Running 0 94s php-apache-deployment-5b65bbc75c-95k6k 1/1 Running 0 28m *php-apache-deployment-5b65bbc75c-cjkwk 0/1 Pending 0 33s* *php-apache-deployment-5b65bbc75c-cn5rt 0/1 Pending 0 33s* *php-apache-deployment-5b65bbc75c-cxctx 0/1 Pending 0 48s* php-apache-deployment-5b65bbc75c-fffnc 1/1 Running 0 64s php-apache-deployment-5b65bbc75c-hbfw8 0/1 Pending 0 33s php-apache-deployment-5b65bbc75c-l8496 1/1 Running 0 48s php-apache-deployment-5b65bbc75c-mv5h6 1/1 Running 0 28m php-apache-deployment-5b65bbc75c-qddrb 1/1 Running 0 48s php-apache-deployment-5b65bbc75c-dd5r5 0/1 Pending 0 48s php-apache-deployment-5b65bbc75c-tr65j 1/1 Running 0 64s 2 - The cluster is unable to create more pods/workers and I get this error message from the pending pods kubectl describe pod php-apache-deployment-5b65bbc75c-dd5r5 Name: php-apache-deployment-5b65bbc75c-dd5r5 Namespace: default Priority: 0 Node: <none> Labels: app=php-apache pod-template-hash=5b65bbc75c Annotations: kubernetes.io/psp: magnum.privileged *Status: Pending* IP: IPs: <none> Controlled By: ReplicaSet/php-apache-deployment-5b65bbc75c Containers: php-apache: Image: k8s.gcr.io/hpa-example Port: 80/TCP Host Port: 0/TCP Limits: cpu: 500m Requests: cpu: 200m Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-4fsgh (ro) Conditions: Type Status PodScheduled False Volumes: default-token-4fsgh: Type: Secret (a volume populated by a Secret) SecretName: default-token-4fsgh Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- * Warning FailedScheduling 2m48s default-scheduler 0/4 nodes are available: 2 Insufficient cpu, 2 node(s) had taint {node-role.kubernetes.io/master <http://node-role.kubernetes.io/master>: }, that the pod didn't tolerate.* * Warning FailedScheduling 2m48s default-scheduler 0/4 nodes are available: 2 Insufficient cpu, 2 node(s) had taint {node-role.kubernetes.io/master <http://node-role.kubernetes.io/master>: }, that the pod didn't tolerate.* * Normal NotTriggerScaleUp 2m42s cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 in backoff after failed scale-u**p* I have this error message from the autoscaller pod *cluster-autoscaler*-f4bd5f674-b9692 : I1123 00:50:27.714801 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 12.709µs I1123 00:51:34.181145 1 scale_up.go:658] Scale-up: setting group default-worker size to 3 *W1123 00:51:34.381953 1 clusterstate.go:281] Disabling scale-up for node group default-worker until 2021-11-23 00:56:34.180840351 +0000 UTC m=+47174.376164120; errorClass=Other; errorCode=cloudProviderError* *E1123 00:51:34.382081 1 static_autoscaler.go:415] Failed to scale up: failed to increase node group size: could not check current nodegroup size: could not get cluster: Get https://dash.cdn.domaine.tld:9511/v1/clusters/b4a6b3eb-fcf3-416f-b740-11a08… <https://dash.cdn.cerist.dz:9511/v1/clusters/b4a6b3eb-fcf3-416f-b740-11a083d…>: dial tcp: lookup dash.cdn.domaine. <http://dash.cdn.cerist.dz>tld on 10.254.0.10:53 <http://10.254.0.10:53>: no such host* W1123 00:51:44.392523 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:51:54.410273 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:52:04.422128 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:52:14.434278 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff W1123 00:52:24.442480 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff I1123 00:52:27.715019 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache I did some tests on the DNS pod and : kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 13h default php-apache-service LoadBalancer 10.254.3.54 xx.xx.xx.213 80:31763/TCP 19m kube-system dashboard-metrics-scraper ClusterIP 10.254.19.191 <none> 8000/TCP 13h *kube-system kube-dns ClusterIP 10.254.0.10 <none> 53/UDP,53/TCP,9153/TCP 13h* kube-system kubernetes-dashboard ClusterIP 10.254.132.17 <none> 443/TCP 13h kube-system magnum-metrics-server ClusterIP 10.254.235.147 <none> 443/TCP 13h I have noticed this behaviour about the horizon url, sometimes the dns pod responds sometimes it does not !!!!! [root@k8multiclustercalico-ve5t6uuoo245-master-0 ~]# *dig @10.254.0.10 <http://10.254.0.10> dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld* ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc33 <<>> @10.254.0.10 dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 5646 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;dash.cd <http://dash.cdn.cerist.dz>n.domaine.tld. IN A ;; AUTHORITY SECTION: *cdn. <http://cdn.cerist.dz>domaine.tld. 30 IN SOA cdn. <http://cdn.cerist.dz>domaine.tld. root.cdn. <http://root.cdn.cerist.dz>domaine.tld. 2021100900 604800 86400 2419200 604800* ;; Query time: 84 msec ;; SERVER: 10.254.0.10#53(10.254.0.10) ;; WHEN: Tue Nov 23 01:08:03 UTC 2021 ;; MSG SIZE rcvd: 12 2 secondes later [root@k8multiclustercalico-ve5t6uuoo245-master-0 ~]# *dig @10.254.0.10 <http://10.254.0.10> dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld* ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc33 <<>> @10.254.0.10 dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7653 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld. IN A ;; ANSWER SECTION: *dash.cdn. <http://dash.cdn.cerist.dz>domaine.tld. 30 IN A xx.xx.xx.129* ;; Query time: 2 msec ;; SERVER: 10.254.0.10#53(10.254.0.10) ;; WHEN: Tue Nov 23 01:08:21 UTC 2021 ;; MSG SIZE rcvd: 81 In the log of the dns pod I have this kubectl logs *kube-dns-autoscaler*-75859754fd-q8z4w -n kube-system *E1122 20:56:09.944449 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource E1122 20:56:19.945294 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource E1122 20:56:29.944245 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource E1122 20:56:39.946346 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource* *E1122 20:56:49.944693 1 autoscaler_server.go:120] Update failure: the server could not find the requested resource* I don't have experience on kubernetes yet, could someone help me debug this? Regards.

1 0

[horizon] Cancelling Today's weekly meeting
by vishal manchanda 23 Nov '21

23 Nov '21

Hello Team, Since there are no agenda items [1] to discuss for today's horizon weekly meeting. Also, Today is a holiday for me or maybe for others as well So let's cancel today's weekly meeting. Next weekly meeting will be on 01 December. Thanks & Regards, Vishal Manchanda [1] https://etherpad.opendev.org/p/horizon-release-priorities

1 0

[Glance] No weekly meeting
by Abhishek Kekane 23 Nov '21

23 Nov '21

Hello Team, As this is the holiday season and ThanksGiving is on our weekly meeting day, we are not meeting this week. Next meeting will be on 02 December. Thanks & Best Regards, Abhishek Kekane

1 0

[neutron] Neutron OVN+QoS Support
by Erlon Cruz 23 Nov '21

23 Nov '21

4 10

[Neutron][OVN] networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports failed
by Tony Liu 23 Nov '21

23 Nov '21

Hi, Is this known issue? Any filed bug or fix to it? ======================================= 2021-11-23 23:04:20.943 40 ERROR futurist.periodics [-] Failed to call periodic 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports' (it runs every 600.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2021-11-23 23:04:20.943 40 ERROR futurist.periodics Traceback (most recent call last): 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction 2021-11-23 23:04:20.943 40 ERROR futurist.periodics yield self._nested_txns_map[cur_thread_id] 2021-11-23 23:04:20.943 40 ERROR futurist.periodics KeyError: 140347734591640 2021-11-23 23:04:20.943 40 ERROR futurist.periodics 2021-11-23 23:04:20.943 40 ERROR futurist.periodics During handling of the above exception, another exception occurred: 2021-11-23 23:04:20.943 40 ERROR futurist.periodics 2021-11-23 23:04:20.943 40 ERROR futurist.periodics Traceback (most recent call last): 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 290, in run 2021-11-23 23:04:20.943 40 ERROR futurist.periodics work() 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 64, in __call__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 178, in decorator 2021-11-23 23:04:20.943 40 ERROR futurist.periodics return f(*args, **kwargs) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 667, in check_for_mcast_flood_reports 2021-11-23 23:04:20.943 40 ERROR futurist.periodics for port in self._nb_idl.lsp_list().execute(check_error=True): 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute 2021-11-23 23:04:20.943 40 ERROR futurist.periodics t.add(self) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics next(self.gen) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 183, in transaction 2021-11-23 23:04:20.943 40 ERROR futurist.periodics yield t 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics next(self.gen) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction 2021-11-23 23:04:20.943 40 ERROR futurist.periodics del self._nested_txns_map[cur_thread_id] 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ 2021-11-23 23:04:20.943 40 ERROR futurist.periodics self.result = self.commit() 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2021-11-23 23:04:20.943 40 ERROR futurist.periodics raise result.ex 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run 2021-11-23 23:04:20.943 40 ERROR futurist.periodics txn.results.put(txn.do_commit()) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit 2021-11-23 23:04:20.943 40 ERROR futurist.periodics raise RuntimeError(msg) 2021-11-23 23:04:20.943 40 ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it 2021-11-23 23:04:20.943 40 ERROR futurist.periodics ======================================= Thanks! Tony

2 2

[cinder] spec opportunities and deadline
by Brian Rosmaita 23 Nov '21

23 Nov '21

To anyone working on a cinder spec for Yoga: This is a reminder that all Cinder Specs for features to be implemented in Yoga must be approved by Friday 17 December 2021 (23:59 UTC). There are two upcoming opportunities to get feedback on your spec proposal and/or ask questions about your spec from the cinder team: 1. Tomorrow's cinder weekly meeting (Wednesday 24 November, 1400 UTC) is being held in videoconference; if you'd like a discussion, please put a topic on the weekly agenda (which also has connection details): https://etherpad.opendev.org/p/cinder-yoga-meetings 2. The cinder yoga R-17 virtual midcycle meeting is being held next week on Wednesday 1 December 1400-1600 UTC; if you'd like to discuss your proposal, please add it to the planning etherpad: https://etherpad.opendev.org/p/cinder-yoga-midcycles cheers, brian

1 0

[kolla-ansible][wallaby][glance] Problem with image list after reconfigure
by Franck VEDEL 23 Nov '21

23 Nov '21

Hello everyone I have a strange problem and I haven't found the solution yet. Following a certificate update I had to do a "kolla-ansible -t multinode reconfigure ». Well, after several attempts (it is not easy to use certificates with Kolla-ansible, and from my advice, not documented enough for beginners), I have my new functional certificates. Perfect ... well almost. I am trying to create a new instance to check general operation. ERROR. Okay, I look in the logs and I see that Cinder is having problems creating volumes with an error that I never had ("TypeError: 'NoneType' object is not iterable). I dig and then I wonder if it is not the Glance images which cannot be used, while they are present (openstack image list is OK). I create an empty volume: it works. I am creating a volume from an image: Failed. However, I have my list of ten images in glance. I create a new image and create a volume with this new image: it works. I create an instance with this new image: OK. What is the problem ? The images present before the "reconfigure" are listed, visible in horizon for example, but unusable. Is there a way to fix this, or do we have to reinstall them all? Thanks in advance for your help if this problem speaks to you. Franck VEDEL Dép. Réseaux Informatiques & Télécoms IUT1 - Univ GRENOBLE Alpes 0476824462 Stages, Alternance, Emploi. http://www.rtgrenoble.fr <http://www.rtgrenoble.fr/>

4 13

Cinder NFS Encryption
by Hitesh Mathur 23 Nov '21

23 Nov '21

Hi, I am not able to find whether Cinder support NFS data-in-transit encryption or not. Can you please provide the information on this and how to use it ? -- Regards Hitesh

2 1

[magnum] [victoria] [fedora-coreos]
by Namrata Sitlani 23 Nov '21

23 Nov '21

Hello All, We run release Victoria(11.1.1) and we deployed Kubernetes version 1.18.20 on Magnum with Fedora CoreOS version 33 and recently we ran into issues with Cinder CSI plugin. Multiple master nodes break the CSI plugin, but if I use a single master everything works fine. In my understanding it could be related to Fedora CoreOS version. It is not clear which versions are supported. Can somebody give me information on which version is supported? Thanks, Namrata Sitlani

1 0

[openstack-helm] No Meeting Tomorrow
by Gage Hugo 22 Nov '21

22 Nov '21

Hey team, Since there are no agenda items [0] for the IRC meeting tomorrow, the meeting is cancelled. Our next meeting will be November 30th. [0] https://etherpad.opendev.org/p/openstack-helm-weekly-meeting

1 0

[all][refstack][neutron][kolla][ironic][heat][trove][senlin][barbican][manila] Fixing Zuul Config Errors
by Clark Boylan 22 Nov '21

22 Nov '21

Hello, The OpenStack tenant in Zuul currently has 134 config errors. You can find these errors at https://zuul.opendev.org/t/openstack/config-errors or by clicking the blue bell icon in the top right of https://zuul.opendev.org/t/openstack/status. The vast majority of these errors appear related to project renames that have been requested of OpenDev or project retirements. Can you please look into fixing these as they can be an attractive nuisance when debugging Zuul problems (they also indicate that a number of your jobs are probably not working). Project renames creating issues: * openstack/python-tempestconf -> osf/python-tempestconf -> openinfra/python-tempestconf * openstack/refstack -> osf/refstack -> openinfra/refstack * x/tap-as-a-service -> openstack/tap-as-a-service * openstack/networking-l2gw -> x/networking-l2gw Project retirements creating issues: * openstack/neutron-lbaas * recordsansible/ara Projects whose configs have errors: * openinfra/python-tempestconf * openstack/heat * openstack/ironic * openstack/kolla-ansible * openstack/kuryr-kubernetes * openstack/murano-apps * openstack/networking-midonet * openstack/networking-odl * openstack/neutron * openstack/neutron-fwaas * openstack/python-troveclient * openstack/senlin * openstack/tap-as-a-service * openstack/zaqar * x/vmware-nsx * openinfra/openstackid * openstack/barbican * openstack/cookbook-openstack-application-catalog * openstack/heat-dashboard * openstack/manila-ui * openstack/python-manilaclient Let us know if we can help decipher any errors, Clark

6 8

[requirements][cinder][manila] pyparsing update needs handling
by Matthew Thode 22 Nov '21

22 Nov '21

https://review.opendev.org/818614 For cinder it looks like operatorPrecedence is gone now. For manila it looks like the same thing. -- Matthew Thode

2 1

[manila] First Yoga cycle bug squash - Nov 29th - 3rd Dec
by Carlos Silva 22 Nov '21

22 Nov '21

Greetings Zorillas and interested stackers! As mentioned in the previous weekly meetings, we will soon be meeting for the first bugsquash of the Yoga release! The event will be held from Nov 29th to 3rd December, 2021, providing an extended contribution window. We will start the event with a call on the first day (Nov 29th). There will be three calls (one of them using our Manila upstream meeting time slot). All of the three calls of the week will be held in a Jitsi room [1]. Nov 29th - 1500 to 1540 UTC - Event opening and bug assignments Dec 2nd - 1500 to 1600 UTC - Collab review session (no regular Manila meeting on this day) Dec 3rd - 1500 to 1540 UTC - Status update and event wrap up A list of selected bugs will be shared here [2]. Please feel free to add any additional bugs you would like to address during the event. [1] https://meetpad.opendev.org/ManilaYogaM1Bugsquash [2] https://ethercalc.openstack.org/wvb2oa23rxbb Thank you in advance! Hope to see you there :) carloss

1 0

[manila] No IRC meeting on 25th Nov
by Goutham Pacha Ravi 22 Nov '21

22 Nov '21

Hello Zorillas, In lieu of holidays this week, we'll be skipping the weekly meeting earlier scheduled at 1500 UTC on 25th Nov 2021. Please reach out here or on OFTC's #openstack-manila should you have any matters that need to be addressed. Thanks, Goutham

1 0

[cinder] reminder: this week's meeting in video+IRC
by Brian Rosmaita 22 Nov '21

22 Nov '21

Quick reminder that this week's Cinder team meeting on Wednesday 24 November, being the final meeting of the month, will be held in both videoconference and IRC at the regularly scheduled time of 1400 UTC. These are the video meeting rules we've agreed to: * Everyone will keep IRC open during the meeting. * We'll take notes in IRC to leave a record similar to what we have for our regular IRC meetings. * Some people are more comfortable communicating in written English. So at any point, any attendee may request that the discussion of the current topic be conducted entirely in IRC. * The meeting will be recorded. connection info: https://bluejeans.com/3228528973 meeting agenda: https://etherpad.opendev.org/p/cinder-yoga-meetings cheers, brian

1 0

Migration from Midonet to OVN
by Jake Yip 22 Nov '21

22 Nov '21

Hi all, We are planning a migration from Midonet to OVN. The general idea is to: - pause neutron - do a `neutron-ovn-db-sync-util` - change networks/ports to geneve/ovs - hard reboot the instances We are wondering if anyone has done a migration like this before, and will like to share their experiences. Any input will be greatly appreciated. Regards, Jake -- Jake Yip DevOps Engineer, ARDC Nectar Research Cloud

2 1

[kolla-ansible][wallaby][magnum] calico works, flannel does not work
by wodel youchi 22 Nov '21

22 Nov '21

Hi, I don't have any experience in kubernetes yet, my goal in the present is to test that the auto-sacele of a kubernetes cluster is working properly using magnum. I followed some tutorials to get the test done. I tried to configure the test but I had problems with magnum-metrics-server not being able to provide metrics, after digging and reading a lot of things on the web, I found out that kubernetes master couldn't connect to the metrics server (curl -k https://ip_metrics_server didn't work) So I created another template but this time using calico as a network provider, and this time it worked. How can I find why flannel does not work? where should I look? Regards.

2 1

[neutron] Bug deputy report (Nov 15 - 21)
by Hongbin Lu 21 Nov '21

21 Nov '21

Hi, I am the bug deputy this week. Please find the report below: High * https://bugs.launchpad.net/neutron/+bug/1951010 Restarting Neutron floods Nova with segment aggregates calls * https://bugs.launchpad.net/neutron/+bug/1951225 [OVN] Agent can't be found in functional test sometimes Medium * https://bugs.launchpad.net/neutron/+bug/1951149 [OVN] If "chassis" register is deleted, "chassis_private" can have 0 "chassis" associated * https://bugs.launchpad.net/neutron/+bug/1951559 [OVN] Router ports gateway_mtu option should not always be set * https://bugs.launchpad.net/neutron/+bug/1951564 snat random-fully supported with iptables 1.6.0 Low * https://bugs.launchpad.net/neutron/+bug/1951272 [OVN] OVS to OVN migration should be stopped if OVS agent firewall is "iptables_hybrid" * https://bugs.launchpad.net/neutron/+bug/1951429 Neutron API responses should not contain tracebacks * https://bugs.launchpad.net/neutron/+bug/1951569 Undecided (might need further traiging from domain experts) * https://bugs.launchpad.net/neutron/+bug/1951074 [OVN] default setting leak nameserver config from the host to instances * https://bugs.launchpad.net/neutron/+bug/1951493 OVS DPDK restart results in that tap interfaces in network namespaces can't be opened * https://bugs.launchpad.net/neutron/+bug/1951517 Segmentation ID should be lower or equal to 4095 Best regards, Hongbin

1 0

[all][tc] What's happening in Technical Committee: summary 19th Nov, 21: Reading: 10 min
by Ghanshyam Mann 19 Nov '21

19 Nov '21

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * We cancelled this week's meeting due to Open Infra Keynotes at the same time. * Next week's meeting is on IRC on 25th Nov, Thursday 15:00 UTC, feel free the topic on the agenda[1] by Nov 24th. 2. What we completed this week: ========================= * Updated Yoga testing runtime [2] * Retire puppet-senlin[3] * Removed TC office hours in favour of weekly meetings [4] 3. Activities In progress: ================== TC Tracker for Yoga cycle ------------------------------ * This etherpad includes the Yoga cycle targets/working items for TC[5]. Open Reviews ----------------- * 8 open reviews for ongoing activities[6]. Fixing Zuul config error -------------------------- Clark sent a email tagging projects need action for zuul config error in their projects[7]. Please fix those config error which is not causing failure now but will fail on changes in those repo or at least is not testing what we think they are. Few projects are fixing those but still we need to clean it up completly. RBAC discussion: continuing from PTG ---------------------------------------------- We had another meeting on Wed and figured out the open question and schedule. Meeting notes are in this etherpad[8]. Final design and schedule is up in this goal rework patch[9]. Please review that and provide feedback from your project perspective. Community-wide goal updates ------------------------------------ * RBAC goal is pretty much in good shape now, feel free to review it[9] * There is one more goal proposal for 'FIPS compatibility and compliance' which needs feedback from community as well from TC[10]. * TC is working to prepare the final goal(s) soon, please wait for some more time. Adjutant need maintainers and PTLs ------------------------------------------- We received response from Braden, Albert on ML, so let's wait on the final call to help here[11]. New project 'Skyline' proposal ------------------------------------ * We discussed it in TC PTG, we are ok to have Skyline as official project but we have few open queries on gerrit and still waiting from skyline team to respond[12]. TC tags analysis ------------------- * TC agreed to remove the framework and it is communicated in ML[13]. Project updates ------------------- * Rename ‘Extended Maintenance’ SIG to the ‘Stable Maintenance’[14] * Retire training-labs repo[15] * Add ProxySQL repository for OpenStack-Ansible[16] * Retire js-openstack-lib [17] 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[18]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [19] 3. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance/+/815851 [3] https://review.opendev.org/c/openstack/governance/+/817329 [4] https://review.opendev.org/c/openstack/governance/+/817493 [5] https://etherpad.opendev.org/p/tc-yoga-tracker [6] https://review.opendev.org/q/projects:openstack/governance+status:open [7] http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025758… [8] https://etherpad.opendev.org/p/policy-popup-yoga-ptg [9] https://review.opendev.org/c/openstack/governance/+/815158 [10] https://review.opendev.org/c/openstack/governance/+/816587 [11] http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025786… [12] https://review.opendev.org/c/openstack/governance/+/814037 [13] http://lists.openstack.org/pipermail/openstack-discuss/2021-October/025571.… [14] https://review.opendev.org/c/openstack/governance-sigs/+/817499 [15] https://review.opendev.org/c/openstack/governance/+/817511 [16] https://review.opendev.org/c/openstack/governance/+/817245 [17] https://review.opendev.org/c/openstack/governance/+/807163 [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [19] http://eavesdrop.openstack.org/#Technical_Committee_Meeting-gmann

1 0

[keystone] No weekly meeting next week
by Douglas Mendizabal 19 Nov '21

19 Nov '21

Hi Keystone friends, I'll be out on PTO next week, so I'm canceling the Keystone weekly meeting for November 23. Meetings will resume the following week on November 30. Thanks, - Douglas Mendizábal (redrobot)

1 0

[requirements][keystone] pysaml2 and oslo.policy not able to be updated due to keystone test failures
by Matthew Thode 19 Nov '21

19 Nov '21

more or less as the title states the following reviews show the failures pysaml2: https://review.opendev.org/818612 oslo.policy: https://review.opendev.org/815820 -- Matthew Thode

2 1

[sdk][tc] A new home for Gophercloud
by Emilien Macchi 19 Nov '21

19 Nov '21

Hi everyone, The Gophercloud project is a stable OpenStack SDK in Golang, which has been widely used by many communities now, including Terraform and all the Kubernetes on OpenStack projects. The project was initiated by Rackspace in 2013 and has already successfully managed their departure as a principal contributor. Fast forward to 2021, Joe Topjian (major maintainer) who used to be an active contributor in Puppet modules and also a voice in the OpenStack operators space, has reached out to me so we can discuss a transition for maintenance: https://github.com/gophercloud/gophercloud/issues/2246 We have discussed this internally and here are our notes: https://github.com/gophercloud/gophercloud/issues/2246#issuecomment-9575894… . To make it short, we are figuring out whether it would make sense to find a new home for the project and if yes, where. The main reason we're reaching out to the opendev community first is because we think this is the most logical place to host the project, alongside OpenStack: Some ideas: - The project would potentially have more visibility in the community, it’s a SDK therefore strongly relying on OpenStack APIs stability. - Use (some) opendev tools, mainly Zuul & nodepool resources - integrate with other projects. - Governance: - Gerrit / not Gerrit: we don’t think we would move to Gerrit yet, as the existing contributors are probably more used to the Github workflow, and we clearly don’t want to lose anyone in the process). - IRC: we could have an IRC channel, potentially Slack as well. Many things to figure out and before we answer these questions, we would like to poll the community: what do you think? Have you contributed to the project? What's your feeling about the ideas here? We welcome any feedback and will take it in consideration in our discussions. Thanks everyone, -- Emilien Macchi, on behalf of the Gophercloud contributors

8 13

[nova][all] Adding /healthcheck support in Nova, and better healthcheck in every projects
by Thomas Goirand 19 Nov '21

19 Nov '21

Hi, About a year and a half ago, I attempted to add /healthcheck support by default in all projects. For Nova, this resulted in this patch: https://review.opendev.org/c/openstack/nova/+/724684 For other projects, it's been merged almost everywhere (I'd have to survey all project to see if that's the case, or if I still have debian specific patches somewhere). Though for Nova, this sparked a discussion where it's been said that the current implementation of /healthcheck wasn't good enough. This resulted in threads about how to better do it. Unfortunately, this blocked my patch from being merged in Nova. It is my point of view to recognize a failure here. The /healthcheck URL was added in oslo.middleware so one can use it with something like haproxy to verify that the API is up, and responds. It was never designed to check, for example, if nova-api has a valid connectivity to MySQL and RabbitMQ. Yes, this is welcome, but in the mean time, operators must tweak the default file to have a valid, useable /etc/nova/api-paste.ini. So I am hereby asking the nova team: Can we please move forward and agree that 1.5 years waiting for such a minor patch is too long, and that such patch should be approved, prior to having a better healtcheck mechanism? I don't think it's a good idea to ask Nova users to wait potentially more development cycles to have a good-by-default api-paste.ini file. At the same time, I am wondering: is anyone even working on a better healthcheck system? I haven't heard that anyone is working on this. Though it would be more than welcome. Currently, to check that a daemon is alive and well, operators are stuck with: - checking with ss if the daemon is correctly connected to a given port - check the logs for rabbitmq and mysql errors (with something like filebeat + elastic search and alarming) Clearly, this doesn't scale. When running many large OpenStack clusters, it is not trivial to have a monitoring system that works and scales. The effort to deploy such a monitoring system is also not trivial at all. So what's been discussed at the time for improving the monitoring would be very much welcome, though not only for the API service: something to check the health of other daemons would be very much welcome. I'd very much would like to participate in a Yoga effort to improve the current situation, and contribute the best I can, though I'm not sure I'd be the best person to drive this... Is there anyone else willing to work on this? Hoping this message is helpful, Cheers, Thomas Goirand (zigo)

6 9

[neutron] Drivers meeting - Friday 18.11.2021 - cancelled
by Lajos Katona 18 Nov '21

18 Nov '21

Hi Neutron Drivers! Due to the lack of agenda, let's cancel tomorrow's drivers meeting. See You on the meeting next week. Lajos Katona (lajoskatona)

1 0

[all] tempest-integrated-compute-centos-8-stream job is failing
by Balazs Gibizer 18 Nov '21

18 Nov '21

Hi, The centos 8 stream job is failing in 100% of the cases with mirror issues [2]. You probably need to hold you recheck until it is resolved. cheers, gibi [1] https://zuul.opendev.org/t/openstack/builds?job_name=tempest-integrated-com… [2] https://zuul.opendev.org/t/openstack/build/67dc088859d1435c9a5097e7469e5c12…

3 5

[all][sdk][neutronclient][neutron] List of projects that use python-neutronclient as "client"
by Lajos Katona 18 Nov '21

18 Nov '21

Hi, During the Yoga PTG we discussed the possibility of deprecating python-neutronclient (see [0]). The CLI part of python-neutronclient is already deprecated (see [1]), but we have python bindings for Neutron both in python-neutronclient and in openstacksdk. As python-neutronclient's binding code is widely used as "client" code (i.e.: Heat, Horizon, Nova), but python-openstackclient uses openstacksdk's binding code for "client" this means duplicated work and maintenance for all the bindings. If we have a new API feature the binding code must go both to python-neutronclient to make it available in Heat, and to openstacksdk to have openstackclient support for it. The best would be to have the binding code in openstacksdk and deprecate python-neutronclient, but before we plan anything we would like to have a list of projects that depend on python-neutronclient. We identified a few but for sure with python-neutronclient's long history there are a lot more: * Heat * Horizon * Nova * various Networking projects * ...... Please share Your thoughts about this topic and the projects which use python-neutronclient, it would be really helpful to see how we can move forward. [0]: https://etherpad.opendev.org/p/neutron-yoga-ptg#L372 [1]: https://review.opendev.org/c/openstack/python-neutronclient/+/795475 Regards Lajos Katona (lajoskatona)

1 0

[sdk]: Check instance error message
by dmeng 17 Nov '21

17 Nov '21

Hello there, Hope everything is going well. I'm wondering if there is any method that could check the error message of an instance whose status is "ERROR"? Like from openstack cli, "openstack server show server_name", if the server is in "ERROR" status, this will return a field "fault" with a message shows the error. I tried the compute service get_server and find_server, but neither of them show the error messages of an instance. Thanks and have a great day! Catherine

3 3

[cinder] yoga R-17 virtual midcycle on 1 december
by Brian Rosmaita 17 Nov '21

17 Nov '21

As decided at today's weekly meeting, the Cinder Yoga R-17 virtual midcycle will be held: DATE: Wednesday 1 December 2021 TIME: 1400-1600 UTC LOCATION: https://bluejeans.com/3228528973 The meeting will be recorded. Please add topics to the midcycle etherpad: https://etherpad.opendev.org/p/cinder-yoga-midcycles cheers, brian

1 0

[tacker] Skip next IRC meeting
by Yasufumi Ogawa 17 Nov '21

17 Nov '21

Hi team, Due to my absence from work, I would like to skip the next IRC meeting on November 23. Thanks, Yasufumi

1 0

[barbican] No weekly meeting next week
by Douglas Mendizabal 17 Nov '21

17 Nov '21

Hi Barbicaneers, I'll be out on PTO next week, so I'm canceling the Barbican weekly meeting for November 23. Meeting will resume the following week on November 30. Thanks, - Douglas Mendizábal (redrobot)

1 0

[all] git-review broken by git version 2.34.0
by Pierre Riteau 17 Nov '21

17 Nov '21

Hello, I recently upgraded to the latest git release 2.34.0 and noticed it breaks git-review (output below is with -v): Errors running git rebase -p -i remotes/gerrit/stable/xena fatal: --preserve-merges was replaced by --rebase-merges I submitted a patch [1], but since it would break compatibility with git versions older than 2.18 that don't support the --rebase-merges (-r) option, it may need to be refined before being merged. Cheers, Pierre [1] https://review.opendev.org/c/opendev/git-review/+/818219

2 2

[cinder] Bug deputy report for week of 11-17-2021
by Sofia Enriquez 17 Nov '21

17 Nov '21

No meeting today :( This is a bug report from 11-10-2021 to 11-17-2021. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- High - https://bugs.launchpad.net/cinder/+bug/1950474 "Xena accept transfer policy breaks volume transfer workflow". Unassigned. Medium - https://bugs.launchpad.net/cinder/+bug/1950291 "tempest-integrated-compute-centos-8-stream fails with version conflict in boto3". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1951163 "Unable to import/manage an encrypted volume". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1951046 "DS8000 driver terminates volume connection when there still has volume attached to instances - Ussuri". Unassigned. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com> -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[nova] weird python stacktrace in nova-compute
by Arnaud Morin 17 Nov '21

17 Nov '21

Hey all, We have some python stacktrace in our nova-compute journalctl logs, which looks like that: $ journalctl -u nova-compute ... Nov 01 09:04:03 host1234 python3[161354]: Exception in thread tpool_thread_6: Nov 01 09:04:03 host1234 python3[161354]: Traceback (most recent call last): Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/threading.py", line 916, in _bootstrap_inner Nov 01 09:04:03 host1234 python3[161354]: self.run() Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/threading.py", line 864, in run Nov 01 09:04:03 host1234 python3[161354]: self._target(*self._args, **self._kwargs) Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/site-packages/eventlet/tpool.py", line 96, in tworker Nov 01 09:04:03 host1234 python3[161354]: _wsock.sendall(_bytetosend) Nov 01 09:04:03 host1234 python3[161354]: TimeoutError: [Errno 110] Connection timed out Nov 01 09:04:03 host1234 python3[161354]: Traceback (most recent call last): Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 109, in wait Nov 01 09:04:03 host1234 python3[161354]: listener.cb(fileno) Nov 01 09:04:03 host1234 python3[161354]: File "/opt/openstack/nova/lib/python3.6/site-packages/eventlet/tpool.py", line 58, in tpool_trampoline Nov 01 09:04:03 host1234 python3[161354]: assert _c Nov 01 09:04:03 host1234 python3[161354]: AssertionError Nov 01 09:04:03 host1234 python3[161354]: Removing descriptor: 16 ... After this, nova-compute is "stuck" and does nothing more, but still continue to answer on RPC (it is still up in nova services) We am not experts of python threading / eventlet stuff, and we have no idea how to debug this. Our current solution is to restart nova-compute, but it's more a dirty workaround than a real fix. Does it ring a bell to someone in the community? Cheers, Arnaud

1 0

[edge][neutron][all] Edge related documentation
by Lajos Katona 16 Nov '21

16 Nov '21

Hi, During Yoga PTG the Neutron team had a very useful discussion together with some Designate folks, see the etherpad [1]. >From Neutron team's perspective the task is to document how to set up an edge site with AZ, DNS.... etc. This is a cross project effort (even if just do it for Neutron-Designate today), is there a place for such documentation? [1]: https://etherpad.opendev.org/p/octavia-designate-neutron-ptg#L16

1 0

[cinder][rbd] festival of RBD driver reviews 11 november 1500 utc
by Brian Rosmaita 16 Nov '21

16 Nov '21

By popular demand (really!), I'm scheduling a RBD driver review festival for next week. It's a community driver, and we've got a backlog of patches: https://review.opendev.org/q/project:openstack/cinder+status:open+file:cind… If your patch is currently in merge conflict, it would be helpful if you could get conflicts resolved before the festival. Also, if you have questions about comments that have been left on your patch, this would be a good time to get them answered. who: Everyone! what: The Cinder Festival of RBD Driver Reviews when: Thursday 11 November 2021 from 1500-1600 UTC where: https://meet.google.com/fsb-qkfc-qun etherpad: https://etherpad.opendev.org/p/cinder-festival-of-driver-reviews (Note that we're trying google meet for this session.)

3 3