Open Stack

Hi,

I am reaching out to inquire about the performance of our local storage
setup. Currently, I am conducting tests using NVMe disks; however, the
results appear to be underwhelming.

In terms of my setup, I have recently incorporated two NVMe disks into my
compute node. These disks have been configured as RAID1 under md127 and
subsequently mounted at /var/lib/nova/instances [1]. During benchmarking
using the fio tool within this directory, I am achieving approximately
160,000 IOPS [2]. This figure serves as a satisfactory baseline and
reference point for upcoming VM tests.

As the next phase, I have established a flavor that employs a root disk for
my virtual machine [3]. Regrettably, the resulting performance yields
around 18,000 IOPS, which is nearly ten times poorer than the compute node
results [4]. While I expected some degradation, a tenfold decrease seems
excessive. Realistically, I anticipated no more than a twofold reduction
compared to the compute node's performance. Hence, I am led to ask: what
should be configured to enhance performance?

I have already experimented with the settings recommended on the Ceph page
for image properties [5]; however, these changes did not yield the desired
improvements. In addition, I attempted to modify the CPU architecture
within the nova.conf file, switching to Cascade Lake architecture, yet this
endeavor also proved ineffective. For your convenience, I have included a
link to my current dumpxml results [6].

Your insights and guidance would be greatly appreciated. I am confident
that there is a solution to this performance disparity that I may have
overlooked. Thank you in advance for your help.
/Jan Wasilewski

*References:*
*[1] nvme allocation and raid configuration:
https://paste.openstack.org/show/bMMgGqu5I6LWuoQWV7TV/
<https://paste.openstack.org/show/bMMgGqu5I6LWuoQWV7TV/>*
*[2] fio performance inside compute node:
https://paste.openstack.org/show/bcMi4zG7QZwuJZX8nyct/
<https://paste.openstack.org/show/bcMi4zG7QZwuJZX8nyct/>*
*[3] Flavor configuration:
https://paste.openstack.org/show/b7o9hCKilmJI3qyXsP5u/
<https://paste.openstack.org/show/b7o9hCKilmJI3qyXsP5u/>*
*[4] fio performance inside VM:
https://paste.openstack.org/show/bUjqxfU4nEtSFqTlU8oH/
<https://paste.openstack.org/show/bUjqxfU4nEtSFqTlU8oH/>*
*[5] image properties:
https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties
<https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties>*
*[6] dumpxml of vm: https://paste.openstack.org/show/bRECcaSMqa8TlrPp0xrT/
<https://paste.openstack.org/show/bRECcaSMqa8TlrPp0xrT/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230809/e687a7e1/attachment-0001.htm>