<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>I would suggest to:</p>
<p>- make sure that "none" I/O scheduler is used inside VM (e.g.
/sys/block/sda/queue/scheduler). I assume quite recent kernel,
otherwise "noop".<br>
</p>
<p>- make sure that host has CPU C-States above C1 disabled (check
values of all /sys/devices/system/cpu/*/cpuidle/state*/disable for
while [..]/name is different than "POLL", C1, C1E) or use some
tool that disables that.<br>
</p>
<p>- Use raw images instead of qcow2: in [libvirt] section of
nova.conf set force_raw_images=True and images_type=flat and
recreate the instance<br>
</p>
<p>Is the difference so big also when you lower I/O depth (for
example to 1) or increase block size (for example to 64k) ?<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 09/08/2023 10:02, Jan Wasilewski
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAN4DDNgkk7qVUN3t8bqr4_pBudJ3MOJxJauc_bcWLV0C1qj5yw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hi,</div>
<p>I am reaching out to inquire about the performance of our
local storage setup. Currently, I am conducting tests using
NVMe disks; however, the results appear to be underwhelming.</p>
<p>In terms of my setup, I have recently incorporated two NVMe
disks into my compute node. These disks have been configured
as RAID1 under md127 and subsequently mounted at
/var/lib/nova/instances [1]. During benchmarking using the fio
tool within this directory, I am achieving approximately
160,000 IOPS [2]. This figure serves as a satisfactory
baseline and reference point for upcoming VM tests.</p>
<p>As the next phase, I have established a flavor that employs a
root disk for my virtual machine [3]. Regrettably, the
resulting performance yields around 18,000 IOPS, which is
nearly ten times poorer than the compute node results [4].
While I expected some degradation, a tenfold decrease seems
excessive. Realistically, I anticipated no more than a twofold
reduction compared to the compute node's performance. Hence, I
am led to ask: what should be configured to enhance
performance?</p>
<p>I have already experimented with the settings recommended on
the Ceph page for image properties [5]; however, these changes
did not yield the desired improvements. In addition, I
attempted to modify the CPU architecture within the nova.conf
file, switching to Cascade Lake architecture, yet this
endeavor also proved ineffective. For your convenience, I have
included a link to my current dumpxml results [6].</p>
<p>Your insights and guidance would be greatly appreciated. I am
confident that there is a solution to this performance
disparity that I may have overlooked. Thank you in advance for
your help.</p>
<div>/Jan Wasilewski<br>
</div>
<div><br>
</div>
<div><i>References:</i></div>
<div><i>[1] nvme allocation and raid configuration: <a
href="https://paste.openstack.org/show/bMMgGqu5I6LWuoQWV7TV/"
moz-do-not-send="true" class="moz-txt-link-freetext">https://paste.openstack.org/show/bMMgGqu5I6LWuoQWV7TV/</a></i></div>
<div><i>[2] fio performance inside compute node: <a
href="https://paste.openstack.org/show/bcMi4zG7QZwuJZX8nyct/"
moz-do-not-send="true" class="moz-txt-link-freetext">https://paste.openstack.org/show/bcMi4zG7QZwuJZX8nyct/</a></i></div>
<div><i>[3] Flavor configuration: <a
href="https://paste.openstack.org/show/b7o9hCKilmJI3qyXsP5u/"
moz-do-not-send="true" class="moz-txt-link-freetext">https://paste.openstack.org/show/b7o9hCKilmJI3qyXsP5u/</a></i></div>
<div><i>[4] fio performance inside VM: <a
href="https://paste.openstack.org/show/bUjqxfU4nEtSFqTlU8oH/"
moz-do-not-send="true" class="moz-txt-link-freetext">https://paste.openstack.org/show/bUjqxfU4nEtSFqTlU8oH/</a></i></div>
<div><i>[5] image properties: <a
href="https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties"
moz-do-not-send="true" class="moz-txt-link-freetext">https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties</a></i></div>
<div><i>[6] dumpxml of vm: <a
href="https://paste.openstack.org/show/bRECcaSMqa8TlrPp0xrT/"
moz-do-not-send="true" class="moz-txt-link-freetext">https://paste.openstack.org/show/bRECcaSMqa8TlrPp0xrT/</a></i></div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Damian Pietras</pre>
</body>
</html>