On Wed, 2023-08-09 at 13:59 +0200, Sven Kieske wrote:
Hi,
I can't cover everything here, because performance is a huge topic, but here are some questions which I didn't find the answer to:
which nvme is this? is this a consumer device by chance?
which openstack release are you running, which hypervisor os and which guest os and kernel versions?
which deployment method do you use?
*[5] image properties: https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties <https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties
at least the ceph recommendations for virtio-scsi are somewhat outdated as virtio-blk is, depending on the benchmark you look at - faster and also supports discard (it's a well hidden secret, but it's true). discard support depend on your distro and machine type
on rhel-9/centos 9 stream this is only true if using q35 machine type the only pc-i440fx machine type in centos 9 is form centso 7 and that predates that feature landing in qemu so if you use the pc/pc-i440fx manchie type virtio-blk does not support discard. on debian based distos since they ship the upstream qemu machine types this is not an issue. it will work with pc too. in general the reason to use virtio-scsi for ceph is that it allow more volues to be used as each one does not consume a pci devce unlike virtio-blk. so if you need many disks use virtio-scsi if you need less then 30 ish then use virtio-blk. nova does not curretly supprrt multiqueue or iothread which are likely required to fully use nvme ssds. we did start workign on a proposal for that but the person working on it move to a diffent role so that is likely something we will look at again for the next deveolpment cycle.
I'd test with virtio-blk first and make sure deadline scheduler is used inside the vm and caching should be none.
HTH