Open Stack

Hi,

Is someone using iscsi multipath in Ubuntu 16 / Mitaka? Things that work OK
for us in Ubuntu 14 / Icehouse seem to be broken.

Basically, the WWID of the volume is not automatically placed in
/etc/multipath/wwids, and if we add it manually, we can attach with
multipath, but the detach never cleans up iscsi sessions nor device mapper
multipath maps.

(using multipath-tools 0.5.0+git1.656f8865-5ubuntu2.1, and only one path
for testing purposes)



- first attach, gives "can't get udev device" error ->
https://bugs.launchpad.net/os-brick/+bug/1585940

          Failed to execute command multipath -l
/dev/disk/by-path/ip-10.167.15.101:3260-iscsi-iqn.1992-08.
com.netapp:sn.1873815289-lun-0

root at dev-e01-host07:~# multipath -l /dev/disk/by-path/ip-10.167.
15.101:3260-iscsi-iqn.1992-08.com.netapp:sn.1873815289-lun-0
Aug 12 15:38:30 |
ip-10.167.15.101:3260-iscsi-iqn.1992-08.com.netapp:sn.1873815289-lun-0:
can't get udev device


- once patched, we get a non-multipath attach: in nova-compute.log we see
"couldn't find a valid multipath device path for <id>"

       issuing "multipath -v3", we get "wwid <id> not in wwids file,
skipping <device>"

root at dev-e01-host07:~# multipath -v3
Aug 12 15:39:47 | libdevmapper version 1.02.110 (2015-10-30)
Aug 12 15:39:47 | DM multipath kernel driver v1.11.0
Aug 12 15:39:47 | sdb: prio = ontap controller setting)
Aug 12 15:39:47 | sdb: prio args = "" controller setting)
Aug 12 15:39:47 | sdb: ontap prio = 2
...
...
Aug 12 15:39:47 | dm-0: blacklisted, udev property missing
===== paths list =====
uuid                              hcil    dev dev_t pri dm_st chk_st
vend/prod
360a9800041763061792b492f4e564179 3:0:0:0 sdb 8:16  2   undef faulty
NETAPP,LU
*Aug 12 15:39:47 | wwid 360a9800041763061792b492f4e564179 not in wwids
file, skipping sdb*
Aug 12 15:39:47 | sdb: orphan path, only one path
Aug 12 15:39:47 | ontap prioritizer refcount 1
Aug 12 15:39:47 | tur checker refcount 1


- if we manually add the WWID to /etc/multipath/wwids, the attach is done
with multipath:

root at dev-e01-host07:~# virsh domblklist 10
Target     Source
------------------------------------------------
vda        /var/lib/nova/instances/8440e7a8-92d9-448d-ac17-5f04f032849c/disk
vdb        /dev/disk/by-id/dm-uuid-mpath-360a9800041763061792b492f4e564230

root at dev-e01-host07:~# multipath -ll
360a9800041763061792b492f4e564230 dm-1 NETAPP,LUN
size=1.0G features='2 queue_if_no_path retain_attached_hw_handler'
hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
  `- 3:0:0:0 sdb 8:16 active ready running


- but the detach operation doesn't remove the multipath map and doesn't
logout of iscsi sessions

[1]+  Done                    nova volume-detach
8440e7a8-92d9-448d-ac17-5f04f032849c 2f717d41-894d-472a-851c-aaa03b593364

root at dev-e01-host07:~# multipath -ll
360a9800041763061792b492f4e564230 dm-1 ##,##
size=1.0G features='2 queue_if_no_path retain_attached_hw_handler'
hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=enabled
  `- #:#:#:# sdb 8:16 failed undef running

root at dev-e01-host07:~# iscsiadm -m session
tcp: [2] 10.167.15.101:3260,2000 iqn.1992-08.com.netapp:sn.1873815289
(non-flash)



in nova-compute.log, we see "Couldn't find multipath device
/dev/mapper/360a9800041763061792b492f4e564230"



instance: 8440e7a8-92d9-448d-ac17-5f04f032849c] Detach volume
951264a3-19d9-4f7e-8bbb-a2eabdcae044 from mountpoint /dev/vdb
 -] Generated XML ('<disk type="block" device="disk">\n  <driver
name="qemu" type="raw" cache="none" io="native"/>\n  <source
dev="/dev/disk/by-id/dm-uuid-mpath-360a9
 -] Waiting for function nova.virt.libvirt.guest._do_wait_and_retry_detach
to return. func
/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py:384
83 - - -] calling os-brick to detach iSCSI Volume disconnect_volume
/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py:95
 - -] Lock "connect_volume" acquired by
"os_brick.initiator.connector.disconnect_volume" :: waited 0.000s inner
/usr/lib/python2.7/dist-packages/oslo_concurrency/lock
3 - - -] Running cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf multipath -r execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:3
3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r"
returned: 0 in 0.167s execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py
 - - -] multipath ['-r']: stdout=reload: 360a9800041763061792b492f4e564230
undef NETAPP,LUN#012size=1.0G features='1 queue_if_no_path' hwhandler='0'
wp=undef#012`-+-
3 - - -] Running cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf multipath -ll /dev/sdb execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/process
3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll
/dev/sdb" returned: 0 in 0.191s execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/proce
 - - -] multipath ['-ll', u'/dev/sdb']:
stdout=360a9800041763061792b492f4e564230 dm-1 NETAPP,LUN#012size=1.0G
features='2 queue_if_no_path retain_attached_hw_handler'
 - - -] remove multipath device /dev/sdb remove_multipath_device
/usr/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:123
3 - - -] Running cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf multipath -l /dev/sdb execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processu
3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l
/dev/sdb" returned: 0 in 0.156s execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/proces
83 - - -] Couldn't find multipath device
/dev/mapper/360a9800041763061792b492f4e564230
 - - -] Disconnect multipath device
/dev/mapper/360a9800041763061792b492f4e564230
_disconnect_volume_multipath_iscsi
/usr/lib/python2.7/dist-packages/os_brick/initiat
3 - - -] Running cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf multipath -ll execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:
3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll"
returned: 0 in 0.158s execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.p
 - - -] multipath ['-ll']: stdout=360a9800041763061792b492f4e564230 dm-1
NETAPP,LUN#012size=1.0G features='2 queue_if_no_path
retain_attached_hw_handler' hwhandler='0
3 - - -] Running cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p
10.167.15.101:3260 execute /usr/lib/python2.7/di
3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m
discovery -t sendtargets -p 10.167.15.101:3260" returned: 0 in 0.168s
execute /usr/lib/python2.7/
 - - -] iscsiadm ['-m', 'discovery', '-t', 'sendtargets', '-p',
u'10.167.15.101:3260']: stdout=10.167.15.101:3260,2000
iqn.1992-08.com.netapp:sn.1873815289#012 stderr
3 - - -] Running cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf multipath -r execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:3
3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r"
returned: 0 in 0.165s execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py
 - - -] multipath ['-r']: stdout=reload: 360a9800041763061792b492f4e564230
undef NETAPP,LUN#012size=1.0G features='1 queue_if_no_path' hwhandler='0'
wp=undef#012`-+-
 - -] Lock "connect_volume" released by
"os_brick.initiator.connector.disconnect_volume" :: held 1.022s inner
/usr/lib/python2.7/dist-packages/oslo_concurrency/lockut
83 - - -] Disconnected iSCSI Volume vdb disconnect_volume
/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py:101
3 - - -] Running cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf cat /etc/iscsi/initiatorname.iscsi execute
/usr/lib/python2.7/dist-packages/oslo_concurr
3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf cat
/etc/iscsi/initiatorname.iscsi" returned: 0 in 0.186s execute
/usr/lib/python2.7/dist-packages/oslo_concu
3 - - -] Running cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf systool -c fc_host -v execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processu
3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host
-v" returned: 96 in 0.140s execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/proce
3 - - -] u'sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host
-v' failed. Not Retrying. execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processut
 - - -] systool is not installed
3 - - -] Running cmd (subprocess): sudo nova-rootwrap
/etc/nova/rootwrap.conf systool -c fc_host -v execute
/usr/lib/python2.7/dist-packages/oslo_concurrency/processu



Any idea of what could be happening?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20160812/f3e5d373/attachment.html>