Hello Can, To fix an RBD (Rados Block Device) IOPS bottleneck on the client side in OpenStack, you can try the following: Monitor the CPU and memory usage on the client machine to ensure that it has sufficient resources. You can use tools like top or htop to view real-time resource usage. Check the network bandwidth between the client and the storage system to ensure that it is not a bottleneck. You can use tools like iperf or tcpdump to measure network performance. Review the configuration of the storage system to ensure that it is optimized for the workload. This may include adjusting the number and type of disks used, as well as the RAID level and chunk size. Consider using a storage system with a higher IOPS rating to see if it can improve performance. This may involve upgrading to faster disks or using a storage solution with more disks or SSDs. Try using a different client machine with more resources (e.g., a machine with a faster CPU and more memory) to see if it can issue more I/O requests. Consider using a different network connection between the client and the storage system, such as a faster network card or a direct connection rather than a network switch. If you are using Ceph as the underlying storage system, you can try adjusting the Ceph configuration to improve performance. This may include adjusting the number of placement groups, the object size, or the number of OSDs (object storage devices). It's also worth noting that an IOPS bottleneck can also occur on the server side (i.e., within the storage system itself). In this case, you may need to adjust the configuration of the storage system or add more resources (e.g., disks or SSDs) to improve performance. BR, Kerem Çeliker keremceliker.medium.com IBM | Red Hat Champion Sent from my iPhone
On 28 Dec 2022, at 08:13, openstack-discuss-request@lists.openstack.org wrote:
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via the World Wide Web, visit https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
or, via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. [ops][nova] RBD IOPS bottleneck on client-side (Can ?zyurt) 2. [keystone][Meeting] Reminder Keystone meeting is cancelled today (Dave Wilde) 3. Nova libvirt/kvm sound device (Zakhar Kirpichenko) 4. Re: [Tacker][SRBAC] Update regarding implementation of project personas in Tacker (Yasufumi Ogawa) 5. Re: [Tacker][SRBAC] Update regarding implementation of project personas in Tacker (manpreet kaur)
----------------------------------------------------------------------
Message: 1 Date: Tue, 27 Dec 2022 15:33:56 +0300 From: Can ?zyurt <acozyurt@gmail.com> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [ops][nova] RBD IOPS bottleneck on client-side Message-ID: <CAMf4N71awOfNyBk4FfpM6UCjH4AZWz+QuJwUnOv+itvqvPbTjw@mail.gmail.com> Content-Type: text/plain; charset="UTF-8"
Hi everyone,
I hope you are all doing well. We are trying to pinpoint an IOPS problem with RBD and decided to ask you for your take on it.
1 control plane 1 compute node 5 storage with 8 SSD disks each Openstack Stein/Ceph Mimic deployed with kolla-ansible on ubuntu-1804 (kernel 5.4) isolcpus 4-127 on compute vcpu_pin_set 4-127 in nova.conf
image_metadatas: hw_scsi_model: virtio-scsi hw_disk_bus: scsi flavor_metadatas: hw:cpu_policy: dedicated
What we have tested: fio --directory=. --ioengine=libaio --direct=1 --name=benchmark_random_read_write --filename=test_rand --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=50 --time_based --runtime=300s --numjobs=16
1. First we run the fio test above on a guest VM, we see average 5K/5K read/write IOPS consistently. What we realize is that during the test, one single core on compute host is used at max, which is the first of the pinned cpus of the guest. 'top -Hp $qemupid' shows that some threads (notably tp_librbd) share the very same core throughout the test. (also emulatorpin set = vcpupin set as expected) 2. We remove isolcpus and every other configuration stays the same. Now fio tests now show 11K/11K read/write IOPS. No bottlenecked single cpu on the host, observed threads seem to visit all emulatorpins. 3. We bring isolcpus back and redeploy the cluster with Train/Nautilus on ubuntu-1804. Observations are identical to #1. 4. We tried replacing vcpu_pin_set to cpu_shared_set and cpu_dedicated_set to be able to pin emulator cpuset to 0-4 to no avail. Multiple guests on a host can easily deplete resources and IOPS drops. 5. Isolcpus are still in place and we deploy Ussuri with kolla-ansible and Train (to limit the moving parts) with ceph-ansible both on ubuntu-1804. Now we see 7K/7K read/write IOPS. 6. We destroy only the compute node and boot it with ubuntu-2004 with isolcpus set. Add it back to the existing cluster and fio shows slightly above 10K/10K read/write IOPS.
What we think happens:
1. Since isolcpus disables scheduling between given cpus, qemu process and its threads are stuck at the same cpu which created the bottleneck. They should be runnable on any given emulatorpin cpus. 2. Ussuri is more performant despite isolcpus, with the improvements made over time. 3. Ubuntu-2004 is more performant despite isolcpus, with the improvements made over time in the kernel.
Now the questions are:
1. What else are we missing here? 2. Are any of those assumptions false? 3. If all true, what can we do to solve this issue given that we cannot upgrade openstack nor ceph on production overnight? 4. Has anyone dealt with this issue before?
We welcome any opinion and suggestions at this point as we need to make sure that we are on the right path regarding the problem and upgrade is not the only solution. Thanks in advance.
------------------------------
Message: 2 Date: Tue, 27 Dec 2022 07:00:30 -0800 From: Dave Wilde <dwilde@redhat.com> To: openstack-discuss@lists.openstack.org Subject: [keystone][Meeting] Reminder Keystone meeting is cancelled today Message-ID: <CAJEkJ+qYCHZayR-t9w4e=ZB846CZSrggcqdYKE85fWkj2XAo5w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Just a quick reminder that there won?t be the keystone weekly meeting today. We?ll resume our regulararly scheduled programming on 03-Jan-2023. Please update the agenda if you have anything you?d like to discuess. The reviewathon is also cancelled this week, to be resumed on 06-Jan-2023.
/Dave