Open Stack

On Mon, Feb 11, 2019 at 11:12:29AM +0100, Gorka Eguileor wrote:
> It is werid that there are things missing from the logs:
> 
> In method _get_connection_devices we have:
> 
>     LOG.debug('Getting connected devices for (ips,iqns,luns)=%s', 1
>               ips_iqns_luns)
>     nodes = self._get_iscsi_nodes()
> 
> And we can see the message in the logs [2], but then we don't see the
> call to iscsiadm that happens as the first instruction in
> _get_iscsi_nodes:
> 
>     out, err = self._execute('iscsiadm', '-m', 'node', run_as_root=True,
>                              root_helper=self._root_helper,
>                              check_exit_code=False)
> 
> And we only see the error coming from parsing the output of that command
> that is not logged.

Yes, I wonder if this is related to a rootwrap stdout/stderr capturing
or something?

> I believe Matthew is right in his assessment, the problem is the output
> from "iscsiadm -m node", there is a missing space between the first 2
> columns in the output [4].
> 
> This looks like an issue in Open iSCSI, not in OS-Brick, Cinder, or
> Nova.
> 
> And checking their code, it looks like this is the patch that fixes it
> [5], so it needs to be added to F29 iscsi-initiator-utils package.

Thank you!  This excellent detective work has solved the problem.  I
did a copr build with that patch [1] and got a good tempest run [2].
Amazing how much trouble a " " can cause.

I have filed an upstream bug on the package

 https://bugzilla.redhat.com/show_bug.cgi?id=1676365

Anyway, it has led to a series of patches you may be interested in,
which I think would help future debugging efforts

 https://review.openstack.org/636078 : fix for quoting of devstack
   args (important for follow-ons)

 https://review.openstack.org/636079 : export all journal logs.
   Things like iscsid were logging to the journal, but we weren't
   capturing them.  Includes instructions on how to use the exported
   journal [3]

 https://review.openstack.org/636080 : add a tcpdump service.  With
   this you can easily packet capture during a devstack run.
   e.g. https://review.openstack.org/636082 captures all iscsi traffic
   and stores it [4]

 https://review.openstack.org/636081 : iscsid debug option, which
   uses a systemd override to turn up debug logging.

Reviews welcome :)

Thanks,

-i

[1] https://github.com/open-iscsi/open-iscsi/commit/baa0cb45cfcf10a81283c191b0b236cd1a2f66ee.patch
[2] http://logs.openstack.org/82/636082/9/check/devstack-platform-fedora-latest/e2fac10/
[3] http://logs.openstack.org/82/636082/9/check/devstack-platform-fedora-latest/e2fac10/controller/logs/devstack.journal.README.txt
[4] http://logs.openstack.org/82/636082/9/check/devstack-platform-fedora-latest/e2fac10/controller/logs/tcpdump.pcap.gz