December 2022 - openstack-discuss

[release] Release countdown for week R-11, Jan 2 - 6
by Thierry Carrez 30 Dec '22

30 Dec '22

Development Focus ----------------- The Antelope-2 milestone is next week, on January 5, 2023! Antelope-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/antelope/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library that has not been otherwise released since milestone 1. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. Next week is also the deadline to freeze the contents of the final release. All new 'Antelope' deliverables need to have a deliverable file in https://opendev.org/openstack/releases/src/branch/master/deliverables and need to have done a release by milestone-2. Changes proposing those deliverables for inclusion in Antelope have been posted, please update them with an actual release request before the milestone-2 deadline if you plan on including that deliverable in Antelope, or -1 if you need one more cycle to be ready. Upcoming Deadlines & Dates -------------------------- Antelope-2 Milestone: January 5th, 2023 OpenInfra Summit Vancouver CFP deadline: January 10th, 2023 Final 2023.1 Antelope release: March 22nd, 2023 -- Thierry Carrez (ttx)

2 1

Re: [ops][nova] RBD IOPS bottleneck on client-side
by Ümit Seren 30 Dec '22

30 Dec '22

We had a similar performance issue with networking (via openswitch) instead of I/O. Our hypervisor and VM configuration were like yours (VCPU pinning + isolcpus). We saw a 50% drop in virtualized networking throughput (measure via iperf). This was because the vhost_net kthreads which are responsible for the virtualized networking were pinned to 2 cores per socket and this quickly became the bottleneck. This was with OpenStack Queens and RHEL 7.6. We ended up keeping the VCPU pinning but removing the isolcpus kernel setting. This fixed the performance regression. Unfortunately, we didn’t further investigate this, so I don’t know why a newer kernel and/or newer Openstack release improves it. Hope this still helps Best Ümit On 27.12.22, 13:33, "Can Özyurt" <acozyurt(a)gmail.com> wrote: Hi everyone, I hope you are all doing well. We are trying to pinpoint an IOPS problem with RBD and decided to ask you for your take on it. 1 control plane 1 compute node 5 storage with 8 SSD disks each Openstack Stein/Ceph Mimic deployed with kolla-ansible on ubuntu-1804 (kernel 5.4) isolcpus 4-127 on compute vcpu_pin_set 4-127 in nova.conf image_metadatas: hw_scsi_model: virtio-scsi hw_disk_bus: scsi flavor_metadatas: hw:cpu_policy: dedicated What we have tested: fio --directory=. --ioengine=libaio --direct=1 --name=benchmark_random_read_write --filename=test_rand --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=50 --time_based --runtime=300s --numjobs=16 1. First we run the fio test above on a guest VM, we see average 5K/5K read/write IOPS consistently. What we realize is that during the test, one single core on compute host is used at max, which is the first of the pinned cpus of the guest. 'top -Hp $qemupid' shows that some threads (notably tp_librbd) share the very same core throughout the test. (also emulatorpin set = vcpupin set as expected) 2. We remove isolcpus and every other configuration stays the same. Now fio tests now show 11K/11K read/write IOPS. No bottlenecked single cpu on the host, observed threads seem to visit all emulatorpins. 3. We bring isolcpus back and redeploy the cluster with Train/Nautilus on ubuntu-1804. Observations are identical to #1. 4. We tried replacing vcpu_pin_set to cpu_shared_set and cpu_dedicated_set to be able to pin emulator cpuset to 0-4 to no avail. Multiple guests on a host can easily deplete resources and IOPS drops. 5. Isolcpus are still in place and we deploy Ussuri with kolla-ansible and Train (to limit the moving parts) with ceph-ansible both on ubuntu-1804. Now we see 7K/7K read/write IOPS. 6. We destroy only the compute node and boot it with ubuntu-2004 with isolcpus set. Add it back to the existing cluster and fio shows slightly above 10K/10K read/write IOPS. What we think happens: 1. Since isolcpus disables scheduling between given cpus, qemu process and its threads are stuck at the same cpu which created the bottleneck. They should be runnable on any given emulatorpin cpus. 2. Ussuri is more performant despite isolcpus, with the improvements made over time. 3. Ubuntu-2004 is more performant despite isolcpus, with the improvements made over time in the kernel. Now the questions are: 1. What else are we missing here? 2. Are any of those assumptions false? 3. If all true, what can we do to solve this issue given that we cannot upgrade openstack nor ceph on production overnight? 4. Has anyone dealt with this issue before? We welcome any opinion and suggestions at this point as we need to make sure that we are on the right path regarding the problem and upgrade is not the only solution. Thanks in advance.

3 2

Glance api deployed only on a single controller on multi-controller deployment [kolla]
by A Monster 30 Dec '22

30 Dec '22

I've deployed openstack xena using kolla ansible on a centos 8 stream cluster, using two controller nodes, however I found out after the deployment that glance api is not available in one node, I tried redeploying but I got the same behavior, although the deployment finished without displaying any error. Thank you. Regards

4 4

[all][requirements][qa]: requirement-checks job broken for <=stable/yoga because it run on ubuntu-jammy on stable branch
by Ghanshyam Mann 29 Dec '22

29 Dec '22

Hello Everyone, While base jobs in opendev moved to run on ubuntu-jammy, requirement-checks job start running on ubuntu-jammy which is ok for the master but not for stable branches. As result, requirement-checks on stable branch running on ubuntu-jammy and started failing for stable/yoga and less: Error: No package matching 'python-dev' is available - https://zuul.opendev.org/t/openstack/build/811ca9311b894b97bfae4438070e5e22 Fixing it by setting the nodeset explicitly in that job so that it does not get changed in future when we move to ubuntu new version for testing. - https://review.opendev.org/q/Ie676079c0b798335ab2a82c03fe4715e0eee4d57 If you see the requirement-checks job failing (if there is a requirement change in the stable branch backport), please hold the recheck until the above fixes are merged. As tox4 fails the requirement gate, pushing the above fixes on top of tox4 fixes - https://review.opendev.org/q/I7bc7c98954395765f16cc943e5c826983db5dba0 -gmann

1 0

kolla openstack the freezer end point is not reachable
by K Santhosh 29 Dec '22

29 Dec '22

Hai , In kolla openstack the freezer end point is not reachable and the freezer api couldn't reach in web browser if any document is there make it to me

1 0

Container trying to access mounted device file
by vincent lee 29 Dec '22

29 Dec '22

Hi all, I am trying to access a mounted device file from my node to my newly created container. I am currently using Kolla-ansible for the deployment, and it is in the yoga version. For example, when I mount a device to my node, such as a radio receiver, the device file (/dev/mounted device name) is accessible on my node directly. However, I am looking for an option to enable the newly created container to access that device file. Best, Vincent

1 0

[Glance] Extending Specs Freeze Deadline by 2 weeks
by Pranali Deore 28 Dec '22

28 Dec '22

Hello All, As discussed in last glance meeting [1] we are extending the glance specs freeze deadline by 2 weeks due to unavailability of core members and less review bandwidth because of year end holidays. Current Spec Freeze date (for Glance) : 5th Jan 2023 New Spec Freeze date (for Glance) : 19th Jan 2023 Please make sure your specs are updated before the new deadline. [1]: https://meetings.opendev.org/meetings/glance/2022/glance.2022-12-22-14.00.l… Thanks, Pranali deore

1 0

Re: openstack-discuss Digest, Vol 50, Issue 63
by Kerem Çeliker 28 Dec '22

28 Dec '22

Hello Can, To fix an RBD (Rados Block Device) IOPS bottleneck on the client side in OpenStack, you can try the following: Monitor the CPU and memory usage on the client machine to ensure that it has sufficient resources. You can use tools like top or htop to view real-time resource usage. Check the network bandwidth between the client and the storage system to ensure that it is not a bottleneck. You can use tools like iperf or tcpdump to measure network performance. Review the configuration of the storage system to ensure that it is optimized for the workload. This may include adjusting the number and type of disks used, as well as the RAID level and chunk size. Consider using a storage system with a higher IOPS rating to see if it can improve performance. This may involve upgrading to faster disks or using a storage solution with more disks or SSDs. Try using a different client machine with more resources (e.g., a machine with a faster CPU and more memory) to see if it can issue more I/O requests. Consider using a different network connection between the client and the storage system, such as a faster network card or a direct connection rather than a network switch. If you are using Ceph as the underlying storage system, you can try adjusting the Ceph configuration to improve performance. This may include adjusting the number of placement groups, the object size, or the number of OSDs (object storage devices). It's also worth noting that an IOPS bottleneck can also occur on the server side (i.e., within the storage system itself). In this case, you may need to adjust the configuration of the storage system or add more resources (e.g., disks or SSDs) to improve performance. BR, Kerem Çeliker keremceliker.medium.com IBM | Red Hat Champion Sent from my iPhone > On 28 Dec 2022, at 08:13, openstack-discuss-request(a)lists.openstack.org wrote: > > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > or, via email, send a message with subject or body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. [ops][nova] RBD IOPS bottleneck on client-side (Can ?zyurt) > 2. [keystone][Meeting] Reminder Keystone meeting is cancelled > today (Dave Wilde) > 3. Nova libvirt/kvm sound device (Zakhar Kirpichenko) > 4. Re: [Tacker][SRBAC] Update regarding implementation of > project personas in Tacker (Yasufumi Ogawa) > 5. Re: [Tacker][SRBAC] Update regarding implementation of > project personas in Tacker (manpreet kaur) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 27 Dec 2022 15:33:56 +0300 > From: Can ?zyurt <acozyurt(a)gmail.com> > To: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> > Subject: [ops][nova] RBD IOPS bottleneck on client-side > Message-ID: > <CAMf4N71awOfNyBk4FfpM6UCjH4AZWz+QuJwUnOv+itvqvPbTjw(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Hi everyone, > > I hope you are all doing well. We are trying to pinpoint an IOPS > problem with RBD and decided to ask you for your take on it. > > 1 control plane > 1 compute node > 5 storage with 8 SSD disks each > Openstack Stein/Ceph Mimic deployed with kolla-ansible on ubuntu-1804 > (kernel 5.4) > isolcpus 4-127 on compute > vcpu_pin_set 4-127 in nova.conf > > image_metadatas: > hw_scsi_model: virtio-scsi > hw_disk_bus: scsi > flavor_metadatas: > hw:cpu_policy: dedicated > > What we have tested: > fio --directory=. --ioengine=libaio --direct=1 > --name=benchmark_random_read_write --filename=test_rand --bs=4k > --iodepth=32 --size=1G --readwrite=randrw --rwmixread=50 --time_based > --runtime=300s --numjobs=16 > > 1. First we run the fio test above on a guest VM, we see average 5K/5K > read/write IOPS consistently. What we realize is that during the test, > one single core on compute host is used at max, which is the first of > the pinned cpus of the guest. 'top -Hp $qemupid' shows that some > threads (notably tp_librbd) share the very same core throughout the > test. (also emulatorpin set = vcpupin set as expected) > 2. We remove isolcpus and every other configuration stays the same. > Now fio tests now show 11K/11K read/write IOPS. No bottlenecked single > cpu on the host, observed threads seem to visit all emulatorpins. > 3. We bring isolcpus back and redeploy the cluster with Train/Nautilus > on ubuntu-1804. Observations are identical to #1. > 4. We tried replacing vcpu_pin_set to cpu_shared_set and > cpu_dedicated_set to be able to pin emulator cpuset to 0-4 to no > avail. Multiple guests on a host can easily deplete resources and IOPS > drops. > 5. Isolcpus are still in place and we deploy Ussuri with kolla-ansible > and Train (to limit the moving parts) with ceph-ansible both on > ubuntu-1804. Now we see 7K/7K read/write IOPS. > 6. We destroy only the compute node and boot it with ubuntu-2004 with > isolcpus set. Add it back to the existing cluster and fio shows > slightly above 10K/10K read/write IOPS. > > > What we think happens: > > 1. Since isolcpus disables scheduling between given cpus, qemu process > and its threads are stuck at the same cpu which created the > bottleneck. They should be runnable on any given emulatorpin cpus. > 2. Ussuri is more performant despite isolcpus, with the improvements > made over time. > 3. Ubuntu-2004 is more performant despite isolcpus, with the > improvements made over time in the kernel. > > Now the questions are: > > 1. What else are we missing here? > 2. Are any of those assumptions false? > 3. If all true, what can we do to solve this issue given that we > cannot upgrade openstack nor ceph on production overnight? > 4. Has anyone dealt with this issue before? > > We welcome any opinion and suggestions at this point as we need to > make sure that we are on the right path regarding the problem and > upgrade is not the only solution. Thanks in advance. > > > > ------------------------------ > > Message: 2 > Date: Tue, 27 Dec 2022 07:00:30 -0800 > From: Dave Wilde <dwilde(a)redhat.com> > To: openstack-discuss(a)lists.openstack.org > Subject: [keystone][Meeting] Reminder Keystone meeting is cancelled > today > Message-ID: > <CAJEkJ+qYCHZayR-t9w4e=ZB846CZSrggcqdYKE85fWkj2XAo5w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Just a quick reminder that there won?t be the keystone weekly meeting > today. We?ll resume our regulararly scheduled programming on 03-Jan-2023. > Please update the agenda if you have anything you?d like to discuess. The > reviewathon is also cancelled this week, to be resumed on 06-Jan-2023. > > /Dave >

1 0

[Tacker][SRBAC] Update regarding implementation of project personas in Tacker
by manpreet kaur 27 Dec '22

27 Dec '22

Hi Ogawa san and Tacker team, This mailer is regarding the SRBAC implementation happening in Tacker. In the Tacker release 2023.1 virtual PTG [1], it was decided by the Tacker community to partially implement the project personas (project-reader role) in the current release. And in upcoming releases, we will implement the remaining project-member role. To address the above requirement, I have prepared a specification [2] and pushed the same in Gerrit for community review. Ghanshyam san reviewed the specification and shared TC's opinion and suggestion to implement both the project-reader and project-member roles. The complete persona implementation will depreciate the 'owner' rule, and help in restricting any other role to accessing project-based resources. Additionally, intact legacy admin (current admin), works in the same way so that we do not break things and introduce the project personas which should be additional things to be available for operators to adopt. Current Status: Incorporated the new requirement and uploaded a new patch set to address the review comment. Note: The Tacker spec freeze date is 28th Dec 2022, there might be some delay in merging the specification in shared timelines. [1] https://etherpad.opendev.org/p/tacker-antelope-ptg#L186 [2] https://review.opendev.org/c/openstack/tacker-specs/+/866956 Thanks & Regards, Manpreet Kaur

3 3

Re: openstack-discuss Digest, Vol 50, Issue 62
by Kerem Çeliker 27 Dec '22

27 Dec '22

Hello Vincent, To allow a user from a container to access files on OpenStack, you can use the --volume or -v option when creating the container to mount a host file or directory as a volume inside the container. For example: Copy code docker run -it --volume /path/on/host:/path/in/container image_name This will mount the host file or directory located at /path/on/host as a volume inside the container at /path/in/container. The user inside the container will be able to access the files in the volume at /path/in/container. You can also use the --mount option to mount a volume. This option provides more control over the volume, such as specifying the volume driver and options. For example: Copy code docker run -it --mount type=bind,source=/path/on/host,target=/path/in/container image_name This will also mount the host file or directory located at /path/on/host as a volume inside the container at /path/in/container, and the user inside the container will be able to access the files in the volume at /path/in/container. Note that the /path/on/host must be an absolute path and must exist on the host system. The /path/in/container can be any path inside the container and does not need to exist prior to the mount. It's also important to note that the user inside the container must have the appropriate permissions to access the mounted files. You can use the --user option to specify the user that the container should run as, or you can set the ownership and permissions of the mounted files on the host system to allow the user inside the container to access them. Hope it works for you.. — Best, Regards, Kerem Çeliker IBM | Red Hat Champion keremceliker.medium.com Sent from my iPhone > On 27 Dec 2022, at 15:07, openstack-discuss-request(a)lists.openstack.org wrote: > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > or, via email, send a message with subject or body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. Container access mounted device (vincent lee) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 27 Dec 2022 19:01:21 +0800 > From: vincent lee <vincentlee676(a)gmail.com> > To: openstack-discuss(a)lists.openstack.org > Subject: Container access mounted device > Message-ID: > > > Hi, I have deployed a working OpenStack and would like to know if there are > any possible options I can enable so that when creating a new container, > files can be mounted to it, and a user from the container can access it. > For instance, having a mounted device from the compute host and allowing > the /dev/(mounted device name) file to be accessible in the container. > > Best regards, > Vincent >

1 0