Open Stack

On 07/02, Ian Wienand wrote:
> Hello,
>
> I'm trying to diagnose what has gone wrong with Fedora 29 in our gate
> devstack test; it seems there is a problem with the iscsi setup and
> consequently the volume based tempest tests all fail.  AFAICS we end
> up with nova hitting parsing errors inside os_brick's iscsi querying
> routines; so it seems whatever error path we've hit is outside the
> usual as it's made it pretty far down the stack.
>
> I have a rather haphazard bug report going on at
>
>  https://bugs.launchpad.net/os-brick/+bug/1814849
>
> as I've tried to trace it down.  At this point, it's exceeding the
> abilities of my cinder/nova/lvm/iscsi/how-this-all-hangs-together
> knowledge.
>
> The final comment there has a link the devstack logs and a few bits
> and pieces of gleaned off the host (which I have on hold and can
> examine) which is hopefully useful to someone skilled in the art.
>
> I'm hoping ultimately it's a rather simple case of a missing package
> or config option; I would greatly appreciate any input so we can get
> this test stable.
>
> Thanks,
>
> -i
>

Hi Ian,

Well, the system from the pastebin [1] doesn't look too good.  DB and
LIO are out of sync.  You can see that the database says that there must
be 3 exports and maps available, yet you only see 1 in LIO.

It is werid that there are things missing from the logs:

In method _get_connection_devices we have:

    LOG.debug('Getting connected devices for (ips,iqns,luns)=%s', 1
              ips_iqns_luns)
    nodes = self._get_iscsi_nodes()

And we can see the message in the logs [2], but then we don't see the
call to iscsiadm that happens as the first instruction in
_get_iscsi_nodes:

    out, err = self._execute('iscsiadm', '-m', 'node', run_as_root=True,
                             root_helper=self._root_helper,
                             check_exit_code=False)

And we only see the error coming from parsing the output of that command
that is not logged.

I believe Matthew is right in his assessment, the problem is the output
from "iscsiadm -m node", there is a missing space between the first 2
columns in the output [4].

This looks like an issue in Open iSCSI, not in OS-Brick, Cinder, or
Nova.

And checking their code, it looks like this is the patch that fixes it
[5], so it needs to be added to F29 iscsi-initiator-utils package.

Cheers,
Gorka.


[1]: http://paste.openstack.org/show/744723/
[2]: http://logs.openstack.org/59/619259/2/check/devstack-platform-fedora-latest/3eaee4d/controller/logs/screen-n-cpu.txt.gz?#_Feb_06_00_10_05_234149
[3]: https://bugs.launchpad.net/os-brick/+bug/1814849/comments/9
[4]: http://paste.openstack.org/show/744724/
[5]: https://github.com/open-iscsi/open-iscsi/commit/baa0cb45cfcf10a81283c191b0b236cd1a2f66ee