November 2021 - openstack-discuss

[neutron] Averaga number of rechecks in Neutron comparing to other projects
by Slawek Kaplonski 25 Nov '21

25 Nov '21

Hi neutrinos, As we discussed during the last PTG, I spent some time today to get average number of rechecks which we need to do in the last PS of the change before it's merged. In theory that number should be close to 0 as patch should be merged with first CI run when it's approved by reviewers :) All data are only from the master branch. I didn't check the same for stable branches. File with graph and raw data in csv format are in the attachments. Basically my conclusion is that Neutron's CI is really bad in that. We have to recheck many, many times before patches will be merged. We really need to think about how to improve that in Neutron. -- Slawek Kaplonski Principal Software Engineer Red Hat

2 2

[Triple 0] Undercloud deployment Getting failed
by Lokendra Rathour 25 Nov '21

25 Nov '21

Hi Team getting strange error when installing triple O Train on Centos 8.4 '--volume', '/var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--rm', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/container-puppet-zaqar.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', 'quay.io/tripleotraincentos8/centos-binary-zaqar-wsgi:current-tripleo'] run failed after Error: container_linux.go:370: starting container process caused: error adding seccomp filter rule for syscall bdflush: permission denied: OCI permission denied", " attempt(s): 3", "2021-11-23 11:00:38,384 WARNING: 58791 -- Retrying running container: zaqar", "2021-11-23 11:00:38,384 ERROR: 58791 -- Failed running container for zaqar", "2021-11-23 11:00:38,385 INFO: 58791 -- Finished processing puppet configs for zaqar", "2021-11-23 11:00:38,385 ERROR: 58782 -- ERROR configuring crond", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring glance_api", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring heat_api", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring heat", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring ironic_api", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring ironic", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring ironic_inspector", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring neutron", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring iscsid", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring keystone", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring memcached", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring mistral", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring mysql", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring nova", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring rabbitmq", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring placement", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring swift", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring swift_ringbuilder", "2021-11-23 11:00:38,386 ERROR: 58782 -- ERROR configuring zaqar" ], "stderr_lines": [], "_ansible_no_log": false, "attempts": 15 } ] ] Not cleaning working directory /home/stack/tripleo-heat-installer-templates Not cleaning ansible directory /home/stack/undercloud-ansible-mie5k51_ Install artifact is located at /home/stack/undercloud-install-20211123110040.tar.bzip2 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Deployment Failed! This issue is recurring multiple times please advise. -- ~ Lokendra

2 2

[all][tc] Technical Committee next weekly meeting on Nov 25th at 1500 UTC
by Ghanshyam Mann 25 Nov '21

25 Nov '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Nov 25th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Nov 24th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[ops][nova][victoria] QEMU/KVM libvirt Authentication Failed
by DHilsbos＠performair.com 25 Nov '21

25 Nov '21

All; I feel like I've dealt with this issue before, but I can't find any records of it. I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again. I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1]. When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com 1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#…

3 6

[all][tc] Yoga testing runtime update. Action needed if py3.9 job failing for your project
by Ghanshyam Mann 25 Nov '21

25 Nov '21

Hello Everyone. As discussed in TC PTG, we have updated the testing runtime for the Yoga cycle [1]. Changes are: * Add Debian 11 as the tested distro * Change centos stream 8 -> centos stream 9 * Bump lowest python version to test to 3.8 and highest to python 3.9 ** This removes python 3.6 from testing. I pushed the job template update[2] which will make py3.9 unit test job voting (which is non-voting currently). I do not see any projects failing consistently on py3.9[3] but still, I will keep it as -W until early next week (23rd Nov). If any project needs time to fix the failing py3.9 job, please do it before 23rd Nov. [1] https://governance.openstack.org/tc/reference/runtimes/yoga.html [2] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/818609 [3] https://zuul.openstack.org/builds?job_name=openstack-tox-py39&branch=master… -gmann

1 1

[keystone][policy][ussuri] why I can create a domain
by Piotr Misiak 24 Nov '21

24 Nov '21

Hi, Maybe a stupid question but I'm really confused. In my Ussuri cloud Keystone has a following policy for create_domain action (this is a default policy from Keystone code): "identity:create_domain": "role:admin and system_scope:all" I have a user which has "admin" role assigned in project "admin" in domain "default" - AKA cloud admin. The user does not have any roles assigned on system scope. Could someone please explain why this user is able to create a domain in the cloud? Looking at the policy rule he shouldn't or maybe I'm reading it in a wrong way? Is there any "backward compatibility" casting "cloud admin" role to "system_scope:all"? Please help Thanks Piotr

2 1

[largescale-sig] Next meeting: Nov 24th, 15utc
by Thierry Carrez 24 Nov '21

24 Nov '21

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20211124T15 We will be discussing our upcoming OpenInfra Live! episode on Dec 9. Feel free to add topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[all][sphinx] "rST localisation for language "foo" not found" error
by Stephen Finucane 24 Nov '21

24 Nov '21

We've seen issues in some projects, whereby attempting to build translations as part of the 'openstack-tox-docs' zuul job [1] can result in the following warning: Warning, treated as error: ../doc/source/contributor/contributing.rst::rST localisation for language "id" not found. Based on [2], this issue appears to have been introduced by a recent version bump of docutils, which was in turn introduced by the introduction of Sphinx 4.x. We don't have a resolution for this bug yet but we'll update when we do. Hopefully this email will improve the "Google'ability" of this bug. Stephen [1] https://opendev.org/openstack/openstackdocstheme/src/commit/08461c5311aa692… [2] https://lists.freebsd.org/pipermail/freebsd-ports/2021-April/120848.html

2 2

[cinder] Bug deputy report for week of 11-24-2021
by Sofia Enriquez 24 Nov '21

24 Nov '21

This is a bug report from 11-17-2021 to 11-24-2021. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium The next 3 bugs are related to multipath config everywhere <https://review.opendev.org/c/openstack/cinder/+/818994>: - https://bugs.launchpad.net/cinder/+bug/1951982 "On encrypted volume cloning rekeying doesn't use multipathing". Assigned to Gorka Eguileor. - https://bugs.launchpad.net/cinder/+bug/1951981 "Kaminario not using multipath for creating volume". Assigned to Gorka Eguileor. - https://bugs.launchpad.net/cinder/+bug/1951977 "Cinder backup - The multipath flag is not set on a create_backup operation". Assigned to Gorka Eguileor. Low - https://bugs.launchpad.net/cinder/+bug/1951250 "[Storwize] Fix multiple lsiogrp,lsvdisk calls in Retype". Unassigned. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com> -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[kolla] Transition kolla/kolla-ansible Rocky/Stein to EOL
by Michał Nasiadka 24 Nov '21

24 Nov '21

Hello, We've kept stable/rocky and stable/stein branches of kolla and kolla-ansible around under extended maintenance for a while now. The coordinated Rocky release was 30th Aug, 2018 (Kolla on 22nd Oct, 2018) and Stein was 10th Apr, 2019 (Kolla on 17th Jul, 2019). The last time a change was backported to these branches has been a while: - kolla stable/rocky: Sep 4, 2020 stable/stein: Mar 19, 2021 - kolla-ansible stable/rocky: Sep 13, 2020 stable/stein: May 28,2021 There is no CI running for those projects periodically, last CI jobs have been run: - kolla: stable/rocky: Jul 8th 2019 stable/stein: Dec 10th 2020 - kolla-ansible stable/rocky: Sep 14th 2019 stable/stein: Dec 10th 2020 All changes to those branches have been abandoned long time (and nothing new proposed for at least a year): https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla <https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla> https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla <https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla> https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla-an… <https://review.opendev.org/q/branch:stable/rocky+project:openstack/kolla-an…> https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla-an… <https://review.opendev.org/q/branch:stable/stein+project:openstack/kolla-an…> In addition to that - kolla-ansible CI jobs have incorrect Zuul configurations (see http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025797… <http://lists.openstack.org/pipermail/openstack-discuss/2021-November/025797…>) - and on one of the Kolla weekly meetings it was decided to mark those branches as EOL. So, I'd like to request that these branches be moved to end-of-life. Patch proposed to add EOL tags to both kolla and kolla-ansible: https://review.opendev.org/c/openstack/releases/+/818920 <https://review.opendev.org/c/openstack/releases/+/818920> Best regards, Michal

1 0