Please ignore my second mail. I found the issue with the cleanup. Before issuing "multipath -f ", os-brick executes "multipath -r" (reload). This command is triggering an error in multipathd, which erases /dev/mapper/<idmap>, and subsequent check for this folder in os-brick fails. <div><br></div><div>The error in multipathd is:</div><div><br></div><div><div> Aug 12 18:18:56 dev-e01-host07 multipath: dm-1: failed to get udev uid: Invalid argument</div><div> Aug 12 18:18:56 dev-e01-host07 multipath: dm-1: failed to get sysfs uid: Invalid argument</div><div> Aug 12 18:18:56 dev-e01-host07 multipath: dm-1: failed to get sgio uid: No such file or directory</div><div><br></div><div><br></div>Maybe this is related to missing wwid in /etc/multipath/wwids?</div><div><br></div><div>We are running kernel 4.6</div><div><br></div><div>Any hint will be appreciated</div><div><br></div><div>Gustavo </div><div><br></div><div><br></div><div>On Friday, 12 August 2016, Gustavo Randich <<a href="mailto:gustavo.randich@gmail.com">gustavo.randich@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Regarding the cleanup in detach, I found that the python-os-brick package in Ubuntu 16 is rather old: 1.2.0<div><br></div><div>I found that the function find_multipath_device in os_brick/initiator/<wbr>linuxscsi.py was refactored her <a href="https://github.com/openstack/os-brick/commit/3ea86f7d605d7109f4e13c1566ba1a14dc856443" target="_blank">https://github.com/openstack/<wbr>os-brick/commit/<wbr>3ea86f7d605d7109f4e13c1566ba1a<wbr>14dc856443</a><br></div><div><br></div><div>I partially applied those changes and could make it work (clean up)</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 12, 2016 at 5:47 PM, Gustavo Randich <span dir="ltr"><<a href="javascript:_e(%7B%7D,'cvml','gustavo.randich@gmail.com');" target="_blank">gustavo.randich@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>Is someone using iscsi multipath in Ubuntu 16 / Mitaka? Things that work OK for us in Ubuntu 14 / Icehouse seem to be broken.</div><div><br></div><div>Basically, the WWID of the volume is not automatically placed in /etc/multipath/wwids, and if we add it manually, we can attach with multipath, but the detach never cleans up iscsi sessions nor device mapper multipath maps.</div><div><br></div><div>(using multipath-tools 0.5.0+git1.656<wbr>f8865-5ubuntu2.1, and only one path for testing purposes)</div><div><br></div><div><br></div><div><br></div><div>- first attach, gives "can't get udev device" error -> <a href="https://bugs.launchpad.net/os-brick/+bug/1585940" target="_blank">https://bugs.launchpad.net/os-<wbr>brick/+bug/1585940</a></div><div><br></div><div> Failed to execute command multipath -l /dev/disk/by-path/ip-10.167.15<wbr>.101:<a href="http://3260-iscsi-iqn.1992-08.co" target="_blank">3260-iscsi-iqn.1992-08.co</a><wbr>m.netapp:sn.1873815289-lun-0<br></div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>root@dev-e01-host07:~# multipath -l /dev/disk/by-path/ip-10.167.15<wbr>.101:<a href="http://3260-iscsi-iqn.1992-08.co" target="_blank">3260-iscsi-iqn.1992-08.co</a><wbr>m.netapp:sn.1873815289-lun-0</div><div>Aug 12 15:38:30 | ip-10.167.15.101:3260-iscsi-iq<wbr>n.1992-08.com.netapp:sn.187381<wbr>5289-lun-0: can't get udev device</div></blockquote><div><br></div><div><div>- once patched, we get a non-multipath attach: in nova-compute.log we see "couldn't find a valid multipath device path for <id>"</div><div><br></div><div> issuing "multipath -v3", we get "wwid <id> not in wwids file, skipping <device>"</div></div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>root@dev-e01-host07:~# multipath -v3</div></div><div><div>Aug 12 15:39:47 | libdevmapper version 1.02.110 (2015-10-30)</div></div><div><div>Aug 12 15:39:47 | DM multipath kernel driver v1.11.0</div></div><div><div>Aug 12 15:39:47 | sdb: prio = ontap controller setting)<br></div></div><div><div>Aug 12 15:39:47 | sdb: prio args = "" controller setting)</div></div><div><div>Aug 12 15:39:47 | sdb: ontap prio = 2</div></div><div>...</div><div>...</div><div><div>Aug 12 15:39:47 | dm-0: blacklisted, udev property missing</div></div><div><div>===== paths list =====</div></div><div><div>uuid hcil dev dev_t pri dm_st chk_st vend/prod</div></div><div><div>360a9800041763061792b492f4e564<wbr>179 3:0:0:0 sdb 8:16 2 undef faulty NETAPP,LU</div></div><div><div><b>Aug 12 15:39:47 | wwid 360a9800041763061792b492f4e564<wbr>179 not in wwids file, skipping sdb</b></div></div><div><div>Aug 12 15:39:47 | sdb: orphan path, only one path</div></div><div><div>Aug 12 15:39:47 | ontap prioritizer refcount 1</div></div><div><div>Aug 12 15:39:47 | tur checker refcount 1</div></div></blockquote><div><br></div><div>- if we manually add the WWID to /etc/multipath/wwids, the attach is done with multipath:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>root@dev-e01-host07:~# virsh domblklist 10</div></div><div><div>Target Source</div></div><div><div>------------------------------<wbr>------------------</div></div><div><div>vda /var/lib/nova/instances/8440e<wbr>7a8-92d9-448d-ac17-5f04f032849<wbr>c/disk</div></div><div><div>vdb /dev/disk/by-id/dm-uuid-mpath<wbr>-360a9800041763061792b492f4e56<wbr>4230</div></div><div><div><br></div></div><div><div>root@dev-e01-host07:~# multipath -ll</div></div><div><div>360a9800041763061792b492f4e564<wbr>230 dm-1 NETAPP,LUN</div></div><div><div>size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='0' wp=rw</div></div><div><div>`-+- policy='round-robin 0' prio=2 status=active</div></div><div><div> `- 3:0:0:0 sdb 8:16 active ready running</div></div></blockquote><div><br></div><div>- but the detach operation doesn't remove the multipath map and doesn't logout of iscsi sessions</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>[1]+ Done nova volume-detach 8440e7a8-92d9-448d-ac17-5f04f0<wbr>32849c 2f717d41-894d-472a-851c-aaa03b<wbr>593364</div><div><br></div><div><div>root@dev-e01-host07:~# multipath -ll</div><div>360a9800041763061792b492f4e564<wbr>230 dm-1 ##,##</div><div>size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='0' wp=rw</div><div>`-+- policy='round-robin 0' prio=0 status=enabled</div><div> `- #:#:#:# sdb 8:16 failed undef running</div></div><div><br></div><div>root@dev-e01-host07:~# iscsiadm -m session</div><div>tcp: [2] <a href="http://10.167.15.101:3260" target="_blank">10.167.15.101:3260</a>,2000 iqn.1992-08.com.netapp:sn.1873<wbr>815289 (non-flash)</div></blockquote><div><br></div><div><br></div><div>in nova-compute.log, we see "Couldn't find multipath device /dev/mapper/360a98000417630617<wbr>92b492f4e564230"</div><div><br></div><div><br></div><div><br></div><div><div>instance: 8440e7a8-92d9-448d-ac17-5f04f0<wbr>32849c] Detach volume 951264a3-19d9-4f7e-8bbb-a2eabd<wbr>cae044 from mountpoint /dev/vdb</div><div> -] Generated XML ('<disk type="block" device="disk">\n <driver name="qemu" type="raw" cache="none" io="native"/>\n <source dev="/dev/disk/by-id/dm-uuid-m<wbr>path-360a9</div><div> -] Waiting for function nova.virt.libvirt.guest._do_wa<wbr>it_and_retry_detach to return. func /usr/lib/python2.7/dist-packag<wbr>es/oslo_service/loopingcall.<wbr>py:384</div><div>83 - - -] calling os-brick to detach iSCSI Volume disconnect_volume /usr/lib/python2.7/dist-packag<wbr>es/nova/virt/libvirt/volume/<wbr>iscsi.py:95</div><div> - -] Lock "connect_volume" acquired by "os_brick.initiator.connector.<wbr>disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/lock</div><div>3 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processuti<wbr>ls.py:3</div><div>3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r" returned: 0 in 0.167s execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processuti<wbr>ls.py</div><div> - - -] multipath ['-r']: stdout=reload: 360a9800041763061792b492f4e564<wbr>230 undef NETAPP,LUN#012size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=undef#012`-+- </div><div>3 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll /dev/sdb execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/process</div><div>3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll /dev/sdb" returned: 0 in 0.191s execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/proce</div><div> - - -] multipath ['-ll', u'/dev/sdb']: stdout=360a9800041763061792b49<wbr>2f4e564230 dm-1 NETAPP,LUN#012size=1.0G features='2 queue_if_no_path retain_attached_hw_handler'</div><div> - - -] remove multipath device /dev/sdb remove_multipath_device /usr/lib/python2.7/dist-packag<wbr>es/os_brick/initiator/linuxscs<wbr>i.py:123</div><div>3 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l /dev/sdb execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processu</div><div>3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l /dev/sdb" returned: 0 in 0.156s execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/proces</div><div>83 - - -] Couldn't find multipath device /dev/mapper/360a98000417630617<wbr>92b492f4e564230</div><div> - - -] Disconnect multipath device /dev/mapper/360a98000417630617<wbr>92b492f4e564230 _disconnect_volume_multipath_i<wbr>scsi /usr/lib/python2.7/dist-packag<wbr>es/os_brick/initiat</div><div>3 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processuti<wbr>ls.py:</div><div>3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll" returned: 0 in 0.158s execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processuti<wbr>ls.p</div><div> - - -] multipath ['-ll']: stdout=360a9800041763061792b49<wbr>2f4e564230 dm-1 NETAPP,LUN#012size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='0</div><div>3 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p <a href="http://10.167.15.101:3260" target="_blank">10.167.15.101:3260</a> execute /usr/lib/python2.7/di</div><div>3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p <a href="http://10.167.15.101:3260" target="_blank">10.167.15.101:3260</a>" returned: 0 in 0.168s execute /usr/lib/python2.7/</div><div> - - -] iscsiadm ['-m', 'discovery', '-t', 'sendtargets', '-p', u'10.167.15.101:3260']: stdout=<a href="http://10.167.15.101:3260" target="_blank">10.167.15.101:3260</a>,2000 iqn.1992-08.com.netapp:sn.1873<wbr>815289#012 stderr</div><div>3 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processuti<wbr>ls.py:3</div><div>3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r" returned: 0 in 0.165s execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processuti<wbr>ls.py</div><div> - - -] multipath ['-r']: stdout=reload: 360a9800041763061792b492f4e564<wbr>230 undef NETAPP,LUN#012size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=undef#012`-+- </div><div> - -] Lock "connect_volume" released by "os_brick.initiator.connector.<wbr>disconnect_volume" :: held 1.022s inner /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/lockut</div><div>83 - - -] Disconnected iSCSI Volume vdb disconnect_volume /usr/lib/python2.7/dist-packag<wbr>es/nova/virt/libvirt/volume/<wbr>iscsi.py:101</div><div>3 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf cat /etc/iscsi/initiatorname.iscsi execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurr</div><div>3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf cat /etc/iscsi/initiatorname.iscsi<wbr>" returned: 0 in 0.186s execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concu</div><div>3 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host -v execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processu</div><div>3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host -v" returned: 96 in 0.140s execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/proce</div><div>3 - - -] u'sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host -v' failed. Not Retrying. execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processut</div><div> - - -] systool is not installed</div><div>3 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host -v execute /usr/lib/python2.7/dist-packag<wbr>es/oslo_concurrency/processu</div></div><div><br></div><div><br></div><div><br></div><div>Any idea of what could be happening?</div><div><br></div><div>Thanks</div><div><br></div></div>
</blockquote></div><br></div>
</blockquote></div>