Hi, I can't cover everything here, because performance is a huge topic, but here are some questions which I didn't find the answer to: which nvme is this? is this a consumer device by chance? which openstack release are you running, which hypervisor os and which guest os and kernel versions? which deployment method do you use?
*[5] image properties: https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties <https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties
at least the ceph recommendations for virtio-scsi are somewhat outdated as virtio-blk is, depending on the benchmark you look at - faster and also supports discard (it's a well hidden secret, but it's true). I'd test with virtio-blk first and make sure deadline scheduler is used inside the vm and caching should be none. HTH -- Sven Kieske Senior Cloud Engineer Mail: kieske@osism.tech Web: https://osism.tech OSISM GmbH Teckstraße 62 / 70190 Stuttgart / Deutschland Geschäftsführer: Christian Berendt Unternehmenssitz: Stuttgart Amtsgericht: Stuttgart, HRB 756139