<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="en-AT" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">We had a similar performance issue with networking (via openswitch) instead of I/O.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Our hypervisor and VM configuration were like yours (VCPU pinning + isolcpus). We saw a 50% drop in virtualized networking throughput (measure via iperf).
<br>
This was because the vhost_net kthreads which are responsible for the virtualized networking were pinned to 2 cores per socket and this quickly became the bottleneck. This was with OpenStack Queens and RHEL 7.6.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">We ended up keeping the VCPU pinning but removing the isolcpus kernel setting. This fixed the performance regression.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Unfortunately, we didn’t further investigate this, so I don’t know why a newer kernel and/or newer Openstack release improves it.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hope this still helps<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Best<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Ümit<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">On 27.12.22, 13:33, "Can Özyurt" <acozyurt@gmail.com> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">Hi everyone,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I hope you are all doing well. We are trying to pinpoint an IOPS<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">problem with RBD and decided to ask you for your take on it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1 control plane<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">1 compute node<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">5 storage with 8 SSD disks each<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Openstack Stein/Ceph Mimic deployed with kolla-ansible on ubuntu-1804<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">(kernel 5.4)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">isolcpus 4-127 on compute<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">vcpu_pin_set 4-127 in nova.conf<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">image_metadatas:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> hw_scsi_model: virtio-scsi<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> hw_disk_bus: scsi<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">flavor_metadatas:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> hw:cpu_policy: dedicated<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What we have tested:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">fio --directory=. --ioengine=libaio --direct=1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">--name=benchmark_random_read_write --filename=test_rand --bs=4k<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">--iodepth=32 --size=1G --readwrite=randrw --rwmixread=50 --time_based<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">--runtime=300s --numjobs=16<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1. First we run the fio test above on a guest VM, we see average 5K/5K<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">read/write IOPS consistently. What we realize is that during the test,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">one single core on compute host is used at max, which is the first of<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">the pinned cpus of the guest. 'top -Hp $qemupid' shows that some<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">threads (notably tp_librbd) share the very same core throughout the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">test. (also emulatorpin set = vcpupin set as expected)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2. We remove isolcpus and every other configuration stays the same.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Now fio tests now show 11K/11K read/write IOPS. No bottlenecked single<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">cpu on the host, observed threads seem to visit all emulatorpins.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">3. We bring isolcpus back and redeploy the cluster with Train/Nautilus<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">on ubuntu-1804. Observations are identical to #1.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">4. We tried replacing vcpu_pin_set to cpu_shared_set and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">cpu_dedicated_set to be able to pin emulator cpuset to 0-4 to no<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">avail. Multiple guests on a host can easily deplete resources and IOPS<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">drops.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">5. Isolcpus are still in place and we deploy Ussuri with kolla-ansible<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and Train (to limit the moving parts) with ceph-ansible both on<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">ubuntu-1804. Now we see 7K/7K read/write IOPS.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">6. We destroy only the compute node and boot it with ubuntu-2004 with<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">isolcpus set. Add it back to the existing cluster and fio shows<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">slightly above 10K/10K read/write IOPS.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What we think happens:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1. Since isolcpus disables scheduling between given cpus, qemu process<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and its threads are stuck at the same cpu which created the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">bottleneck. They should be runnable on any given emulatorpin cpus.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2. Ussuri is more performant despite isolcpus, with the improvements<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">made over time.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">3. Ubuntu-2004 is more performant despite isolcpus, with the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">improvements made over time in the kernel.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Now the questions are:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1. What else are we missing here?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2. Are any of those assumptions false?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">3. If all true, what can we do to solve this issue given that we<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">cannot upgrade openstack nor ceph on production overnight?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">4. Has anyone dealt with this issue before?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We welcome any opinion and suggestions at this point as we need to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">make sure that we are on the right path regarding the problem and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">upgrade is not the only solution. Thanks in advance.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>