Hello Openstack,
We have Ussuri deployed on a few clouds, and they're all plugged to PureStorage Arrays. We allow users to only use volumes for their servers. It means that each server disk is a LUN attached over ISCSI (with multipath) on the compute node hosting the server. Everything works quite fine, but we have a weird issue when extending volumes attached to running instances. The guests notice the new disk size .. of the last extent.
Say I have a server with a 10gb disk. I add 5gb. On the guest, still 10gb. I add another 5gb, and on the guest I get 15, and so on. I've turned the debug mode on and I could see no error in the log. Looking closer at the log I could catch the culprit:
2022-12-08 17:35:13.998 46195 DEBUG os_brick.initiator.linuxscsi [] Starting size: *76235669504* 2022-12-08 17:35:14.028 46195 DEBUG os_brick.initiator.linuxscsi [] volume size after scsi device rescan *80530636800* extend_volume 2022-12-08 17:35:14.035 46195 DEBUG os_brick.initiator.linuxscsi [] Volume device info = {'device': '/dev/disk/by-path/ip-1...1:3260-iscsi-iqn.2010-06.com.purestorage:flasharray.x-lun-10', 'host': '5', 'channel': '0', 'id': '0', 'lun': '10'} extend_volume 2022-12-08 17:35:14.348 46195 INFO os_brick.initiator.linuxscsi [] Find Multipath device file for volume WWN 3624... 2022-12-08 17:35:14.349 46195 DEBUG os_brick.initiator.linuxscsi [] Checking to see if /dev/disk/by-id/dm-uuid-mpath-3624.. exists yet. wait_for_path 2022-12-08 17:35:14.349 46195 DEBUG os_brick.initiator.linuxscsi [] /dev/disk/by-id/dm-uuid-mpath-3624... has shown up. wait_for_path 2022-12-08 17:35:14.382 46195 INFO os_brick.initiator.linuxscsi [] mpath(/dev/disk/by-id/dm-uuid-mpath-3624) *current size 76235669504* 2022-12-08 17:35:14.412 46195 INFO os_brick.initiator.linuxscsi [] mpath(/dev/disk/by-id/dm-uuid-mpath-3624) *new size 76235669504* 2022-12-08 17:35:14.413 46195 DEBUG oslo_concurrency.lockutils [] Lock "extend_volume" released by "os_brick.initiator.connectors.iscsi.ISCSIConnector.extend_volume" :: held 2.062s inner 2022-12-08 17:35:14.459 46195 DEBUG os_brick.initiator.connectors.iscsi [] <== extend_volume: return (2217ms) *76235669504* trace_logging_wrapper 2022-12-08 17:35:14.461 46195 DEBUG nova.virt.libvirt.volume.iscsi [] Extend iSCSI Volume /dev/dm-28; new_size=*76235669504* extend_volume 2022-12-08 17:35:14.462 46195 DEBUG nova.virt.libvirt.driver [] Resizing target device /dev/dm-28 to *76235669504* _resize_attached_volume
The logs clearly shows that the rescan confirm the new size but when interrogating multipath, it does not. But requesting multipath few seconds after on the command line shows the new size as well. It explains the behaviour.
I'm running Ubuntu 18.04 with multipath 0.7.4-2ubuntu3.2. The os-brick code for multipath is far more basic than the one in master branch. Maybe the multipath version installed is too recent for os-brick.
Thanks for the help
Jerome