Open Stack

Please ignore my second mail. I found the issue with the cleanup. Before
issuing "multipath -f ", os-brick executes "multipath -r" (reload). This
command is triggering an error in multipathd, which erases
/dev/mapper/<idmap>, and subsequent check for this folder in os-brick
fails.

The error in multipathd is:

    Aug 12 18:18:56 dev-e01-host07 multipath: dm-1: failed to get udev uid:
Invalid argument
    Aug 12 18:18:56 dev-e01-host07 multipath: dm-1: failed to get sysfs
uid: Invalid argument
    Aug 12 18:18:56 dev-e01-host07 multipath: dm-1: failed to get sgio uid:
No such file or directory


Maybe this is related to missing wwid in /etc/multipath/wwids?

We are running kernel 4.6

Any hint will be appreciated

Gustavo


On Friday, 12 August 2016, Gustavo Randich <gustavo.randich at gmail.com>
wrote:

> Regarding the cleanup in detach, I found that the python-os-brick package
> in Ubuntu 16 is rather old: 1.2.0
>
> I found that the function find_multipath_device in os_brick/initiator/linuxscsi.py
> was refactored her https://github.com/openstack/os-brick/commit/
> 3ea86f7d605d7109f4e13c1566ba1a14dc856443
>
> I partially applied those changes and could make it work (clean up)
>
>
>
>
>
> On Fri, Aug 12, 2016 at 5:47 PM, Gustavo Randich <
> gustavo.randich at gmail.com
> <javascript:_e(%7B%7D,'cvml','gustavo.randich at gmail.com');>> wrote:
>
>> Hi,
>>
>> Is someone using iscsi multipath in Ubuntu 16 / Mitaka? Things that work
>> OK for us in Ubuntu 14 / Icehouse seem to be broken.
>>
>> Basically, the WWID of the volume is not automatically placed in
>> /etc/multipath/wwids, and if we add it manually, we can attach with
>> multipath, but the detach never cleans up iscsi sessions nor device mapper
>> multipath maps.
>>
>> (using multipath-tools 0.5.0+git1.656f8865-5ubuntu2.1, and only one path
>> for testing purposes)
>>
>>
>>
>> - first attach, gives "can't get udev device" error ->
>> https://bugs.launchpad.net/os-brick/+bug/1585940
>>
>>           Failed to execute command multipath -l
>> /dev/disk/by-path/ip-10.167.15.101:3260-iscsi-iqn.1992-08.co
>> m.netapp:sn.1873815289-lun-0
>>
>> root at dev-e01-host07:~# multipath -l /dev/disk/by-path/ip-10.167.15.101:
>> 3260-iscsi-iqn.1992-08.com.netapp:sn.1873815289-lun-0
>> Aug 12 15:38:30 | ip-10.167.15.101:3260-iscsi-iq
>> n.1992-08.com.netapp:sn.1873815289-lun-0: can't get udev device
>>
>>
>> - once patched, we get a non-multipath attach: in nova-compute.log we see
>> "couldn't find a valid multipath device path for <id>"
>>
>>        issuing "multipath -v3", we get "wwid <id> not in wwids file,
>> skipping <device>"
>>
>> root at dev-e01-host07:~# multipath -v3
>> Aug 12 15:39:47 | libdevmapper version 1.02.110 (2015-10-30)
>> Aug 12 15:39:47 | DM multipath kernel driver v1.11.0
>> Aug 12 15:39:47 | sdb: prio = ontap controller setting)
>> Aug 12 15:39:47 | sdb: prio args = "" controller setting)
>> Aug 12 15:39:47 | sdb: ontap prio = 2
>> ...
>> ...
>> Aug 12 15:39:47 | dm-0: blacklisted, udev property missing
>> ===== paths list =====
>> uuid                              hcil    dev dev_t pri dm_st chk_st
>> vend/prod
>> 360a9800041763061792b492f4e564179 3:0:0:0 sdb 8:16  2   undef faulty
>> NETAPP,LU
>> *Aug 12 15:39:47 | wwid 360a9800041763061792b492f4e564179 not in wwids
>> file, skipping sdb*
>> Aug 12 15:39:47 | sdb: orphan path, only one path
>> Aug 12 15:39:47 | ontap prioritizer refcount 1
>> Aug 12 15:39:47 | tur checker refcount 1
>>
>>
>> - if we manually add the WWID to /etc/multipath/wwids, the attach is done
>> with multipath:
>>
>> root at dev-e01-host07:~# virsh domblklist 10
>> Target     Source
>> ------------------------------------------------
>> vda        /var/lib/nova/instances/8440e7a8-92d9-448d-ac17-5f04f032849
>> c/disk
>> vdb        /dev/disk/by-id/dm-uuid-mpath-360a9800041763061792b492f4e56
>> 4230
>>
>> root at dev-e01-host07:~# multipath -ll
>> 360a9800041763061792b492f4e564230 dm-1 NETAPP,LUN
>> size=1.0G features='2 queue_if_no_path retain_attached_hw_handler'
>> hwhandler='0' wp=rw
>> `-+- policy='round-robin 0' prio=2 status=active
>>   `- 3:0:0:0 sdb 8:16 active ready running
>>
>>
>> - but the detach operation doesn't remove the multipath map and doesn't
>> logout of iscsi sessions
>>
>> [1]+  Done                    nova volume-detach
>> 8440e7a8-92d9-448d-ac17-5f04f032849c 2f717d41-894d-472a-851c-aaa03b593364
>>
>> root at dev-e01-host07:~# multipath -ll
>> 360a9800041763061792b492f4e564230 dm-1 ##,##
>> size=1.0G features='2 queue_if_no_path retain_attached_hw_handler'
>> hwhandler='0' wp=rw
>> `-+- policy='round-robin 0' prio=0 status=enabled
>>   `- #:#:#:# sdb 8:16 failed undef running
>>
>> root at dev-e01-host07:~# iscsiadm -m session
>> tcp: [2] 10.167.15.101:3260,2000 iqn.1992-08.com.netapp:sn.1873815289
>> (non-flash)
>>
>>
>>
>> in nova-compute.log, we see "Couldn't find multipath device
>> /dev/mapper/360a9800041763061792b492f4e564230"
>>
>>
>>
>> instance: 8440e7a8-92d9-448d-ac17-5f04f032849c] Detach volume
>> 951264a3-19d9-4f7e-8bbb-a2eabdcae044 from mountpoint /dev/vdb
>>  -] Generated XML ('<disk type="block" device="disk">\n  <driver
>> name="qemu" type="raw" cache="none" io="native"/>\n  <source
>> dev="/dev/disk/by-id/dm-uuid-mpath-360a9
>>  -] Waiting for function nova.virt.libvirt.guest._do_wait_and_retry_detach
>> to return. func /usr/lib/python2.7/dist-packag
>> es/oslo_service/loopingcall.py:384
>> 83 - - -] calling os-brick to detach iSCSI Volume disconnect_volume
>> /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py:95
>>  - -] Lock "connect_volume" acquired by "os_brick.initiator.connector.disconnect_volume"
>> :: waited 0.000s inner /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/lock
>> 3 - - -] Running cmd (subprocess): sudo nova-rootwrap
>> /etc/nova/rootwrap.conf multipath -r execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/processutils.py:3
>> 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r"
>> returned: 0 in 0.167s execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/processutils.py
>>  - - -] multipath ['-r']: stdout=reload: 360a9800041763061792b492f4e564230
>> undef NETAPP,LUN#012size=1.0G features='1 queue_if_no_path' hwhandler='0'
>> wp=undef#012`-+-
>> 3 - - -] Running cmd (subprocess): sudo nova-rootwrap
>> /etc/nova/rootwrap.conf multipath -ll /dev/sdb execute
>> /usr/lib/python2.7/dist-packages/oslo_concurrency/process
>> 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll
>> /dev/sdb" returned: 0 in 0.191s execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/proce
>>  - - -] multipath ['-ll', u'/dev/sdb']: stdout=360a9800041763061792b492f4e564230
>> dm-1 NETAPP,LUN#012size=1.0G features='2 queue_if_no_path
>> retain_attached_hw_handler'
>>  - - -] remove multipath device /dev/sdb remove_multipath_device
>> /usr/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:123
>> 3 - - -] Running cmd (subprocess): sudo nova-rootwrap
>> /etc/nova/rootwrap.conf multipath -l /dev/sdb execute
>> /usr/lib/python2.7/dist-packages/oslo_concurrency/processu
>> 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l
>> /dev/sdb" returned: 0 in 0.156s execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/proces
>> 83 - - -] Couldn't find multipath device /dev/mapper/360a98000417630617
>> 92b492f4e564230
>>  - - -] Disconnect multipath device /dev/mapper/360a9800041763061792b492f4e564230
>> _disconnect_volume_multipath_iscsi /usr/lib/python2.7/dist-packag
>> es/os_brick/initiat
>> 3 - - -] Running cmd (subprocess): sudo nova-rootwrap
>> /etc/nova/rootwrap.conf multipath -ll execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/processutils.py:
>> 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll"
>> returned: 0 in 0.158s execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/processutils.p
>>  - - -] multipath ['-ll']: stdout=360a9800041763061792b492f4e564230 dm-1
>> NETAPP,LUN#012size=1.0G features='2 queue_if_no_path
>> retain_attached_hw_handler' hwhandler='0
>> 3 - - -] Running cmd (subprocess): sudo nova-rootwrap
>> /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p
>> 10.167.15.101:3260 execute /usr/lib/python2.7/di
>> 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m
>> discovery -t sendtargets -p 10.167.15.101:3260" returned: 0 in 0.168s
>> execute /usr/lib/python2.7/
>>  - - -] iscsiadm ['-m', 'discovery', '-t', 'sendtargets', '-p',
>> u'10.167.15.101:3260']: stdout=10.167.15.101:3260,2000
>> iqn.1992-08.com.netapp:sn.1873815289#012 stderr
>> 3 - - -] Running cmd (subprocess): sudo nova-rootwrap
>> /etc/nova/rootwrap.conf multipath -r execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/processutils.py:3
>> 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r"
>> returned: 0 in 0.165s execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/processutils.py
>>  - - -] multipath ['-r']: stdout=reload: 360a9800041763061792b492f4e564230
>> undef NETAPP,LUN#012size=1.0G features='1 queue_if_no_path' hwhandler='0'
>> wp=undef#012`-+-
>>  - -] Lock "connect_volume" released by "os_brick.initiator.connector.disconnect_volume"
>> :: held 1.022s inner /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/lockut
>> 83 - - -] Disconnected iSCSI Volume vdb disconnect_volume
>> /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py:101
>> 3 - - -] Running cmd (subprocess): sudo nova-rootwrap
>> /etc/nova/rootwrap.conf cat /etc/iscsi/initiatorname.iscsi execute
>> /usr/lib/python2.7/dist-packages/oslo_concurr
>> 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf cat
>> /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.186s execute
>> /usr/lib/python2.7/dist-packages/oslo_concu
>> 3 - - -] Running cmd (subprocess): sudo nova-rootwrap
>> /etc/nova/rootwrap.conf systool -c fc_host -v execute
>> /usr/lib/python2.7/dist-packages/oslo_concurrency/processu
>> 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c
>> fc_host -v" returned: 96 in 0.140s execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/proce
>> 3 - - -] u'sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host
>> -v' failed. Not Retrying. execute /usr/lib/python2.7/dist-packag
>> es/oslo_concurrency/processut
>>  - - -] systool is not installed
>> 3 - - -] Running cmd (subprocess): sudo nova-rootwrap
>> /etc/nova/rootwrap.conf systool -c fc_host -v execute
>> /usr/lib/python2.7/dist-packages/oslo_concurrency/processu
>>
>>
>>
>> Any idea of what could be happening?
>>
>> Thanks
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20160812/de96e73f/attachment.html>