<div dir="ltr"><div>Hi,</div><div><br></div><span></span><span class="gmail-flex-grow gmail-flex-shrink-0"></span><div class="gmail-flex gmail-flex-grow gmail-flex-col gmail-gap-3"><div class="gmail-min-h-[20px] gmail-flex gmail-flex-col gmail-items-start gmail-gap-3 gmail-overflow-x-auto gmail-whitespace-pre-wrap gmail-break-words"><div class="gmail-markdown gmail-prose gmail-w-full gmail-break-words gmail-dark:prose-invert gmail-dark">Thank you once again for your valuable suggestions. I conducted another round of tests with C-states disabled. I checked the BIOS settings and added the suggested line to the grub startup. After rebooting my compute node, I observed an improvement in performance, reaching around 20,000 IOPS. Although there was a modest performance boost, it wasn't as substantial as I had anticipated.<br>Additionally, I configured a ramdisk to establish a baseline for comparison. The results were quite significant, with the ramdisk achieving approximately 72,000 IOPS [1] [2]. However, I had initially expected even higher figures. Regardless, such outcomes would be highly beneficial for my NVMe virtual machines.<br>Nonetheless, I'm at a loss regarding potential further optimizations. I've explored some resources, such as those found at: <a href="https://docs.openstack.org/nova/rocky/user/flavors.html">https://docs.openstack.org/nova/rocky/user/flavors.html</a>, which outline IO limits. However, I am under the impression that these limits might only restrict performance rather than enhancing it. Could you kindly confirm if my understanding is accurate?<br>I extend my gratitude in advance for any forthcoming suggestions. It's possible that I might be searching in the wrong places for solutions.<br>/Jan Wasilewski<br><br><i>References:<br></i></div><div class="gmail-markdown gmail-prose gmail-w-full gmail-break-words gmail-dark:prose-invert gmail-dark"><i>[1] dumpxml config for ramdisk vm: <a href="https://paste.openstack.org/show/b7AgTZBjvSpWMioJzmoA/">https://paste.openstack.org/show/b7AgTZBjvSpWMioJzmoA/</a></i></div><div class="gmail-markdown gmail-prose gmail-w-full gmail-break-words gmail-dark:prose-invert gmail-dark"><i>[2] fio results of vm where ramdisk is a main disk: <a href="https://paste.openstack.org/show/bdII56cavmVwNAIq3axQ/">https://paste.openstack.org/show/bdII56cavmVwNAIq3axQ/</a></i></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">czw., 10 sie 2023 o 14:05 Damian Pietras <<a href="mailto:damian.pietras@hardit.pl">damian.pietras@hardit.pl</a>> napisał(a):<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">HI,<br>
<br>
You wrote "/sys/devices/system/cpu/*/cpuidle/state*/disable output is 0 for all cpus". It means all C-states (power saving states) are _enabled_.<br>
This may cause lower and inconsistent results. I would repeat the test with deeper C-states disabled. I think simplest way to do that is to boot<br>
system (nova compute node) with "|intel_idle.max_cstate=1" added to kernel command line parameters. I had <br>
similar issues with I/O performance inside VMs (but with lower disk <br>
queue depth) and power saving / frequency scaling had greatest influence <br>
on the results and also caused variations in the results between test <br>
runs. If you are out of ideas you could also rule out disk / filesystem <br>
/ RAID configuration influence by temporary mounting tmpfs in <br>
/var/lib/nova/instances so the instances will have RAM-backed volumes. <br>
You need enough RAM for that of course. |<br>
<br>
On 10/08/2023 13:35, Jan Wasilewski wrote:<br>
> Hi,<br>
><br>
> I wanted to express my sincere gratitude for all the help and advice <br>
> you've given me. I followed your suggestions and carried out a bunch <br>
> of tests, but unfortunately, the performance boost I was hoping for <br>
> hasn't materialized.<br>
><br>
> Let me break down the configurations I've tried and the results I've <br>
> got. Just to give you some context, all my tests were done using two <br>
> INTEL SSDPE2MD400G4 NVMe disks and Ubuntu 20.04LTS as the OS on the <br>
> compute node. You can find all the nitty-gritty details in [1] and <br>
> [2]. Additionally, I've shared the results of the fio tests directly <br>
> executed on the RAID directory within the compute node in [3].<br>
><br>
> Then, I expanded my testing to instances, and here's what I found:<br>
><br>
> 1. I tested things out with the default settings and Ubuntu 22.04 LTS<br>
> image. The iOPS results were hovering around 18-18.5k. Check out<br>
> [4] and [5] for the specifics.<br>
> 2. I tweaked the nova.conf file with two changes: force_raw_images =<br>
> true and images_type = flat. Unfortunately, this only brought the<br>
> iOPS down a bit, to just under 18k. You can see more in [6] and [7].<br>
> 3. I made an extra change in nova.conf by switching the cpu_model<br>
> from SandyBridge to IvyBridge. This change dropped the iOPS<br>
> further, to around 17k. Details are in [8] and [9].<br>
> 4. Lastly, I played around with image properties, setting<br>
> hw_scsi_model=virtio-scsi and hw_disk_bus=scsi. However, this also<br>
> resulted in around 17k iOPS. You can find out more in [10] and [11].<br>
><br>
> It's a bit disheartening that none of these changes seemed to have the <br>
> impact I was aiming for. So, I'm starting to think there might be a <br>
> crucial piece of the puzzle that I'm missing here. If you have any <br>
> ideas or insights, I'd be incredibly grateful for your input.<br>
><br>
> Thanks once more for all your help and support.<br>
><br>
> /Jan Wasilewski<br>
><br>
><br>
> /References:<br>
> /<br>
> /[1] Disk details and raid details: <br>
> <a href="https://paste.openstack.org/show/bRyLPZ6TDHpIEKadLC7z//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bRyLPZ6TDHpIEKadLC7z//</a><br>
> /[2] Compute node and nova details: <br>
> <a href="https://paste.openstack.org/show/bcGw3Glm6U0r1kUsg8nU//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bcGw3Glm6U0r1kUsg8nU//</a><br>
> /[3] fio results executed in raid directory inside compute node: <br>
> <a href="https://paste.openstack.org/show/bN0EkBjoAP2Ig5PSSfy3//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bN0EkBjoAP2Ig5PSSfy3//</a><br>
> /[4] dumpxml of instance from test 1: <br>
> <a href="https://paste.openstack.org/show/bVSq8tz1bSMdiYXcF3IP//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bVSq8tz1bSMdiYXcF3IP//</a><br>
> /[5] fio results from test 1: <br>
> <a href="https://paste.openstack.org/show/bKlxom8Yl7NtHO8kO53a//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bKlxom8Yl7NtHO8kO53a//</a><br>
> /[6] dumpxml of instance from test 2: <br>
> <a href="https://paste.openstack.org/show/bN2JN9DXT4DGKNZnzkJ8//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bN2JN9DXT4DGKNZnzkJ8//</a><br>
> /[7] fio results from test 2: <br>
> <a href="https://paste.openstack.org/show/b7GXIVI2Cv0qkVLQaAF3//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/b7GXIVI2Cv0qkVLQaAF3//</a><br>
> /[8] dumpxml of instance from test 3: <br>
> <a href="https://paste.openstack.org/show/b0821V4IUq8N7YPb73sg//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/b0821V4IUq8N7YPb73sg//</a><br>
> /[9] fio results from test 3: <br>
> <a href="https://paste.openstack.org/show/bT1Erfxq4XTj0ubTTgdj//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bT1Erfxq4XTj0ubTTgdj//</a><br>
> /[10] dumpxml of instance from test 4: <br>
> <a href="https://paste.openstack.org/show/bjTXM0do1xgzmVZO02Q7//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bjTXM0do1xgzmVZO02Q7//</a><br>
> /[11] fio results from test 4: <br>
> <a href="https://paste.openstack.org/show/bpbVJntkR5aNke3trtRd//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bpbVJntkR5aNke3trtRd//</a><br>
><br>
><br>
> śr., 9 sie 2023 o 19:56 Damian Pietras <<a href="mailto:damian.pietras@hardit.pl" target="_blank">damian.pietras@hardit.pl</a>> <br>
> napisał(a):<br>
><br>
> I would suggest to:<br>
><br>
> - make sure that "none" I/O scheduler is used inside VM (e.g.<br>
> /sys/block/sda/queue/scheduler). I assume quite recent kernel,<br>
> otherwise "noop".<br>
><br>
> - make sure that host has CPU C-States above C1 disabled (check<br>
> values of all /sys/devices/system/cpu/*/cpuidle/state*/disable for<br>
> while [..]/name is different than "POLL", C1, C1E) or use some<br>
> tool that disables that.<br>
><br>
> - Use raw images instead of qcow2: in [libvirt] section of<br>
> nova.conf set force_raw_images=True and images_type=flat and<br>
> recreate the instance<br>
><br>
> Is the difference so big also when you lower I/O depth (for<br>
> example to 1) or increase block size (for example to 64k) ?<br>
><br>
><br>
> On 09/08/2023 10:02, Jan Wasilewski wrote:<br>
>> Hi,<br>
>><br>
>> I am reaching out to inquire about the performance of our local<br>
>> storage setup. Currently, I am conducting tests using NVMe disks;<br>
>> however, the results appear to be underwhelming.<br>
>><br>
>> In terms of my setup, I have recently incorporated two NVMe disks<br>
>> into my compute node. These disks have been configured as RAID1<br>
>> under md127 and subsequently mounted at /var/lib/nova/instances<br>
>> [1]. During benchmarking using the fio tool within this<br>
>> directory, I am achieving approximately 160,000 IOPS [2]. This<br>
>> figure serves as a satisfactory baseline and reference point for<br>
>> upcoming VM tests.<br>
>><br>
>> As the next phase, I have established a flavor that employs a<br>
>> root disk for my virtual machine [3]. Regrettably, the resulting<br>
>> performance yields around 18,000 IOPS, which is nearly ten times<br>
>> poorer than the compute node results [4]. While I expected some<br>
>> degradation, a tenfold decrease seems excessive. Realistically, I<br>
>> anticipated no more than a twofold reduction compared to the<br>
>> compute node's performance. Hence, I am led to ask: what should<br>
>> be configured to enhance performance?<br>
>><br>
>> I have already experimented with the settings recommended on the<br>
>> Ceph page for image properties [5]; however, these changes did<br>
>> not yield the desired improvements. In addition, I attempted to<br>
>> modify the CPU architecture within the nova.conf file, switching<br>
>> to Cascade Lake architecture, yet this endeavor also proved<br>
>> ineffective. For your convenience, I have included a link to my<br>
>> current dumpxml results [6].<br>
>><br>
>> Your insights and guidance would be greatly appreciated. I am<br>
>> confident that there is a solution to this performance disparity<br>
>> that I may have overlooked. Thank you in advance for your help.<br>
>><br>
>> /Jan Wasilewski<br>
>><br>
>> /References:/<br>
>> /[1] nvme allocation and raid configuration:<br>
>> <a href="https://paste.openstack.org/show/bMMgGqu5I6LWuoQWV7TV//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bMMgGqu5I6LWuoQWV7TV//</a><br>
>> /[2] fio performance inside compute node:<br>
>> <a href="https://paste.openstack.org/show/bcMi4zG7QZwuJZX8nyct//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bcMi4zG7QZwuJZX8nyct//</a><br>
>> /[3] Flavor configuration:<br>
>> <a href="https://paste.openstack.org/show/b7o9hCKilmJI3qyXsP5u//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/b7o9hCKilmJI3qyXsP5u//</a><br>
>> /[4] fio performance inside VM:<br>
>> <a href="https://paste.openstack.org/show/bUjqxfU4nEtSFqTlU8oH//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bUjqxfU4nEtSFqTlU8oH//</a><br>
>> /[5] image properties:<br>
>> <a href="https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties/" rel="noreferrer" target="_blank">https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties/</a><br>
>> /[6] dumpxml of vm:<br>
>> <a href="https://paste.openstack.org/show/bRECcaSMqa8TlrPp0xrT//" rel="noreferrer" target="_blank">https://paste.openstack.org/show/bRECcaSMqa8TlrPp0xrT//</a><br>
><br>
> -- <br>
> Damian Pietras<br>
><br>
-- <br>
Damian Pietras<br>
<br>
</blockquote></div>