Open Stack

On Thu, Jun 07, 2018 at 01:07:48PM -0500, Matt Riedemann wrote:
> On 6/7/2018 12:56 PM, melanie witt wrote:
> > Recently, we've received interest about increasing the maximum number of
> > allowed volumes to attach to a single instance > 26. The limit of 26 is
> > because of a historical limitation in libvirt (if I remember correctly)
> > and is no longer limited at the libvirt level in the present day. So,
> > we're looking at providing a way to attach more than 26 volumes to a
> > single instance and we want your feedback.
> 
> The 26 volumes thing is a libvirt driver restriction.

The original limitation of 26 disks was because at that time there was
no 'virtio-scsi'.  

(With 'virtio-scsi', each of its controller allows upto 256 targets, and
each target can use any LUN (Logical Unit Number) from 0 to 16383
(inclusive).  Therefore, the maxium allowable disks on a single
'virtio-scsi' controller is 256 * 16384 == 4194304.)  Source[1].

[...]

> > Some ideas that have been discussed so far include:
> > 
> > A) Selecting a new, higher maximum that still yields reasonable
> > performance on a single compute host (64 or 128, for example). Pros:
> > helps prevent the potential for poor performance on a compute host from
> > attaching too many volumes. Cons: doesn't let anyone opt-in to a higher
> > maximum if their environment can handle it.

Option (A) can still be considered: We can limit it to 256 disks.  Why?

FWIW, I did some digging here:

The upstream libguestfs project after some thorough testing, arrived at
a limit of 256 disks, and suggest the same for Nova.  And if anyone
wants to increase that limit, the proposer should come up with a fully
worked through test plan. :-) (Try doing any meaningful I/O to so many
disks at once, and see how well that works out.)

What more, the libguestfs upstream tests 256 disks, and even _that_
fails sometimes:

    https://bugzilla.redhat.com/show_bug.cgi?id=1478201 -- "kernel runs
    out of memory with 256 virtio-scsi disks"
    
The above bug is fixed now in kernel-4.17.0-0.rc3.git1.2. (And also
required a corresponding fix in QEMU[2], which is available from version
v2.11.0 onwards.)

[...]


[1] https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg02823.html
    -- virtio-scsi limits
[2] https://git.qemu.org/?p=qemu.git;a=commit;h=5c0919d 

-- 
/kashyap