[openstack-dev] [nova] increasing the number of allowed volumes attached per instance > 26

Kashyap Chamarthy kchamart at redhat.com
Mon Jun 11 15:06:04 UTC 2018


On Mon, Jun 11, 2018 at 11:55:29AM +0200, Sahid Orentino Ferdjaoui wrote:
> On Fri, Jun 08, 2018 at 11:35:45AM +0200, Kashyap Chamarthy wrote:
> > On Thu, Jun 07, 2018 at 01:07:48PM -0500, Matt Riedemann wrote:

[...]

> > > The 26 volumes thing is a libvirt driver restriction.
> > 
> > The original limitation of 26 disks was because at that time there was
> > no 'virtio-scsi'.  
> > 
> > (With 'virtio-scsi', each of its controller allows upto 256 targets, and
> > each target can use any LUN (Logical Unit Number) from 0 to 16383
> > (inclusive).  Therefore, the maxium allowable disks on a single
> > 'virtio-scsi' controller is 256 * 16384 == 4194304.)  Source[1].
> 
> Not totally true for Nova. Nova handles one virtio-scsi controller per
> guest and plug all the volumes in one target so in theory that would
> be 16384 LUN (only).

Yeah, I could've been clearer that I was only talking the maximum
allowable disks regardless of how Nova handles it.

> But you made a good point the 26 volumes thing is not a libvirt driver
> restriction. For example the QEMU SCSI native implementation handles
> 256 disks.
> 
> About the virtio-blk limitation I made the same finding but Tsuyoshi
> Nagata shared an interesting point saying that virtio-blk is not longer
> limited by the number of PCI slot available. That in recent kernel and
> QEMU version [0].
> 
> I could join what you are suggesting at the bottom and fix the limit
> to 256 disks.

Right, that's for KVM-based hypervisors.  

Eric Fried on IRC said the other day that for IBM POWER hypervisor they
have tested (not with OpenStack) upto 4000 disks.  But I am yet to see
any more concrete details from POWER hypervisor users on this thread.

If people can't seem to reach an agreement on the limits, we may have to
settle with conditionals:

    if kvm|qemu:
        return 256
    elif POWER:
        return 4000
    elif:
        ...

Before that we need concrete data that it is a _reasonble_ limit for
POWER hypervisor (and possibly others).

> [0] https://review.openstack.org/#/c/567472/16/nova/virt/libvirt/blockinfo.py@162

[...]

-- 
/kashyap



More information about the OpenStack-dev mailing list