- openstack-discuss - lists.openstack.org

[nova][placement] Specs freeze will be on Jan 12th
by Sylvain Bauza 03 Jan '23

03 Jan '23

Hey, As we didn't have a quorum for today's meeting, I prefer to move our deadline for freezing the Antelope accepted specs by one week, instead of this Thursday. This means we will continue to review open specs until January the 12th. That being said, as a reminder, our Feature Freeze is on February 16th [1] which means we will only have *five* weeks for reviewing all the implementations, so please start to work on your feature branch and don't wait for your spec to be accepted or we wouldn't have time for reviewing all the open features. Thanks, -Sylvain [1] https://releases.openstack.org/antelope/schedule.html#a-ff

1 0

Kolla-ansible Yoga Magnum csi-cinder-controllerplugin-0 CrashLoopBackOff
by Oliver Weinmann 03 Jan '23

03 Jan '23

Dear all, I have a strange issue with Magnum on Yoga deployed by kolla-ansible. I noticed this on a prod cluster and so I deployed a fresh cluster from scratch and here I'm facing the very same issue. When deploying a a new K8s cluster the csi-cinder-controllerplugin-0 pod is in CrashLoopBackOff state. kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-56448757b9-62lxw 1/1 Running 0 11d kube-system coredns-56448757b9-6mvtr 1/1 Running 0 11d kube-system csi-cinder-controllerplugin-0 4/5 CrashLoopBackOff 3339 (92s ago) 11d kube-system csi-cinder-nodeplugin-88ttn 2/2 Running 0 11d kube-system csi-cinder-nodeplugin-tnxpt 2/2 Running 0 11d kube-system dashboard-metrics-scraper-67f57ff746-2vhgb 1/1 Running 0 11d kube-system k8s-keystone-auth-5djxh 1/1 Running 0 11d kube-system kube-dns-autoscaler-6d5b5dc777-mm7qs 1/1 Running 0 11d kube-system kube-flannel-ds-795hj 1/1 Running 0 11d kube-system kube-flannel-ds-p76rf 1/1 Running 0 11d kube-system kubernetes-dashboard-7b88d986b4-5bhnf 1/1 Running 0 11d kube-system magnum-metrics-server-6c4c77844b-jbpml 1/1 Running 0 11d kube-system npd-sqsbr 1/1 Running 0 11d kube-system openstack-cloud-controller-manager-c5prb 1/1 Running 0 11d I searched the mailing list and noticed that I reported the very same issue last year: https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028517.html but then I was either able to fix it or not able to reproduce it. I searched the web and found other users having the same issue: https://storyboard.openstack.org/#!/story/2010023 https://github.com/kubernetes/cloud-provider-openstack/issues/1845 They suggest to change the csi_snapshotter_tag to v4.0.0. I haven't tried it yet, but I have the feeling that something in my deployment is not correct since it wants to use v1.2.2 which is a very old version. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulling 26m (x3382 over 12d) kubelet Pulling image "quay.io/k8scsi/csi-snapshotter:v1.2.2" Warning BackOff 81s (x79755 over 12d) kubelet Back-off restarting failed container kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-admin-test-old-ygahdfhciitx-master-0 Ready master 12d v1.23.3 10.0.0.132 172.28.4.124 Fedora CoreOS 35.20220410.3.1 5.16.18-200.fc35.x86_64 docker://20.10.12 k8s-admin-test-old-ygahdfhciitx-node-0 Ready <none> 12d v1.23.3 10.0.0.160 172.28.4.128 Fedora CoreOS 35.20220410.3.1 5.16.18-200.fc35.x86_64 docker://20.10.12 I deployed on Rocky Linux 8.7 using latest Kolla-ansible for Yoga: [vagrant@seed ~]$ kolla-ansible --version 14.7.1 [vagrant@seed ~]$ cat /etc/os-release NAME="Rocky Linux" VERSION="8.7 (Green Obsidian)" ID="rocky" ID_LIKE="rhel centos fedora" VERSION_ID="8.7" PLATFORM_ID="platform:el8" PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)" ANSI_COLOR="0;32" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:rocky:rocky:8:GA" HOME_URL="https://rockylinux.org/" BUG_REPORT_URL="https://bugs.rockylinux.org/" ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8" ROCKY_SUPPORT_PRODUCT_VERSION="8.7" REDHAT_SUPPORT_PRODUCT="Rocky Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.7" Can you please help or clarify? Best Regards, Oliver

1 0

[cloudkitty][horizon] change service name list in GUI
by Satish Patel 03 Jan '23

03 Jan '23

Folks, I am trying to configure rating for network data bytes in/out but what I have seen is that the ceilometer uses a different name than what cloudkitty GUI drop down-menu has. Ceilometer using "network.outgoing.bytes" and "network.incoming.bytes" Cloudkitty GUI drop-down list has "network.outgoing.bytes.rate" & "network.incoming.bytes.rate" I have tried to change it in cloudkitty metrics.yml file to match it with the ceilometer but the rating is not calculating until I change that in cloudkitty. Please advise how to change it in cloudkitty.

2 1

[TripleO] Tripleo-ci-centos-9-content-provider jobs are failing in check line
by Amol Kahat 03 Jan '23

03 Jan '23

Hello All, We are investigating centos-9-content-provider check failure, related to build-test-packages. Jobs link[1] Details are in the related launchpad bug[2]. [1] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-co… [2] https://bugs.launchpad.net/tripleo/+bug/2000897/ -- *Amol Kahat* Software Engineer *Red Hat India Pvt. Ltd. Pune, India.* akahat(a)redhat.com B764 E6F8 F4C1 A1AF 816C 6840 FDD3 BA6C 832D 7715

1 0

[neutron] bug deputy report for week of 2022-12-26
by Bence Romsics 03 Jan '23

03 Jan '23

Hi, During the winter holidays we had hardly any new bugs reported: Medium: * https://bugs.launchpad.net/neutron/+bug/2000634 [OVN] Maintenance task for availability zones changes failing proposed fix: https://review.opendev.org/c/openstack/neutron/+/868746 Incomplete: * https://bugs.launchpad.net/neutron/+bug/2000495 See error "HashRingIsEmpty" when create instance waiting for more information Cheers, Bence

1 0

[ansible][ACO] - Contribution questions
by Gaël THEROND 31 Dec '22

31 Dec '22

Hi ansible collections openstack team! I finally had time to list all my issues met with the project, created few bug reports and even contributed to a patch today (minor, mainly copy/paste) however I’ve few questions regarding the CI process! Overall, what’s the rule with the CI code testing? I’ve read the contributing guide and had an eye on previous patches to see how it’s used but I’m having a hard time to find a real unified method. For instance, it seems that some module miss CI tasks (such as compute_service_info) or did I missed something? Thanks a lot for all the good job!

4 7

[release] Release countdown for week R-11, Jan 2 - 6
by Thierry Carrez 30 Dec '22

30 Dec '22

Development Focus ----------------- The Antelope-2 milestone is next week, on January 5, 2023! Antelope-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/antelope/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library that has not been otherwise released since milestone 1. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. Next week is also the deadline to freeze the contents of the final release. All new 'Antelope' deliverables need to have a deliverable file in https://opendev.org/openstack/releases/src/branch/master/deliverables and need to have done a release by milestone-2. Changes proposing those deliverables for inclusion in Antelope have been posted, please update them with an actual release request before the milestone-2 deadline if you plan on including that deliverable in Antelope, or -1 if you need one more cycle to be ready. Upcoming Deadlines & Dates -------------------------- Antelope-2 Milestone: January 5th, 2023 OpenInfra Summit Vancouver CFP deadline: January 10th, 2023 Final 2023.1 Antelope release: March 22nd, 2023 -- Thierry Carrez (ttx)

2 1

Re: [ops][nova] RBD IOPS bottleneck on client-side
by Ümit Seren 30 Dec '22

30 Dec '22

We had a similar performance issue with networking (via openswitch) instead of I/O. Our hypervisor and VM configuration were like yours (VCPU pinning + isolcpus). We saw a 50% drop in virtualized networking throughput (measure via iperf). This was because the vhost_net kthreads which are responsible for the virtualized networking were pinned to 2 cores per socket and this quickly became the bottleneck. This was with OpenStack Queens and RHEL 7.6. We ended up keeping the VCPU pinning but removing the isolcpus kernel setting. This fixed the performance regression. Unfortunately, we didn’t further investigate this, so I don’t know why a newer kernel and/or newer Openstack release improves it. Hope this still helps Best Ümit On 27.12.22, 13:33, "Can Özyurt" <acozyurt(a)gmail.com> wrote: Hi everyone, I hope you are all doing well. We are trying to pinpoint an IOPS problem with RBD and decided to ask you for your take on it. 1 control plane 1 compute node 5 storage with 8 SSD disks each Openstack Stein/Ceph Mimic deployed with kolla-ansible on ubuntu-1804 (kernel 5.4) isolcpus 4-127 on compute vcpu_pin_set 4-127 in nova.conf image_metadatas: hw_scsi_model: virtio-scsi hw_disk_bus: scsi flavor_metadatas: hw:cpu_policy: dedicated What we have tested: fio --directory=. --ioengine=libaio --direct=1 --name=benchmark_random_read_write --filename=test_rand --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=50 --time_based --runtime=300s --numjobs=16 1. First we run the fio test above on a guest VM, we see average 5K/5K read/write IOPS consistently. What we realize is that during the test, one single core on compute host is used at max, which is the first of the pinned cpus of the guest. 'top -Hp $qemupid' shows that some threads (notably tp_librbd) share the very same core throughout the test. (also emulatorpin set = vcpupin set as expected) 2. We remove isolcpus and every other configuration stays the same. Now fio tests now show 11K/11K read/write IOPS. No bottlenecked single cpu on the host, observed threads seem to visit all emulatorpins. 3. We bring isolcpus back and redeploy the cluster with Train/Nautilus on ubuntu-1804. Observations are identical to #1. 4. We tried replacing vcpu_pin_set to cpu_shared_set and cpu_dedicated_set to be able to pin emulator cpuset to 0-4 to no avail. Multiple guests on a host can easily deplete resources and IOPS drops. 5. Isolcpus are still in place and we deploy Ussuri with kolla-ansible and Train (to limit the moving parts) with ceph-ansible both on ubuntu-1804. Now we see 7K/7K read/write IOPS. 6. We destroy only the compute node and boot it with ubuntu-2004 with isolcpus set. Add it back to the existing cluster and fio shows slightly above 10K/10K read/write IOPS. What we think happens: 1. Since isolcpus disables scheduling between given cpus, qemu process and its threads are stuck at the same cpu which created the bottleneck. They should be runnable on any given emulatorpin cpus. 2. Ussuri is more performant despite isolcpus, with the improvements made over time. 3. Ubuntu-2004 is more performant despite isolcpus, with the improvements made over time in the kernel. Now the questions are: 1. What else are we missing here? 2. Are any of those assumptions false? 3. If all true, what can we do to solve this issue given that we cannot upgrade openstack nor ceph on production overnight? 4. Has anyone dealt with this issue before? We welcome any opinion and suggestions at this point as we need to make sure that we are on the right path regarding the problem and upgrade is not the only solution. Thanks in advance.

3 2

Glance api deployed only on a single controller on multi-controller deployment [kolla]
by A Monster 30 Dec '22

30 Dec '22

I've deployed openstack xena using kolla ansible on a centos 8 stream cluster, using two controller nodes, however I found out after the deployment that glance api is not available in one node, I tried redeploying but I got the same behavior, although the deployment finished without displaying any error. Thank you. Regards

4 4

[all][requirements][qa]: requirement-checks job broken for <=stable/yoga because it run on ubuntu-jammy on stable branch
by Ghanshyam Mann 29 Dec '22

29 Dec '22

Hello Everyone, While base jobs in opendev moved to run on ubuntu-jammy, requirement-checks job start running on ubuntu-jammy which is ok for the master but not for stable branches. As result, requirement-checks on stable branch running on ubuntu-jammy and started failing for stable/yoga and less: Error: No package matching 'python-dev' is available - https://zuul.opendev.org/t/openstack/build/811ca9311b894b97bfae4438070e5e22 Fixing it by setting the nodeset explicitly in that job so that it does not get changed in future when we move to ubuntu new version for testing. - https://review.opendev.org/q/Ie676079c0b798335ab2a82c03fe4715e0eee4d57 If you see the requirement-checks job failing (if there is a requirement change in the stable branch backport), please hold the recheck until the above fixes are merged. As tox4 fails the requirement gate, pushing the above fixes on top of tox4 fixes - https://review.opendev.org/q/I7bc7c98954395765f16cc943e5c826983db5dba0 -gmann

1 0