On Mon, Feb 11, 2019 at 11:12:29AM +0100, Gorka Eguileor wrote:
It is werid that there are things missing from the logs:
In method _get_connection_devices we have:
LOG.debug('Getting connected devices for (ips,iqns,luns)=%s', 1 ips_iqns_luns) nodes = self._get_iscsi_nodes()
And we can see the message in the logs [2], but then we don't see the call to iscsiadm that happens as the first instruction in _get_iscsi_nodes:
out, err = self._execute('iscsiadm', '-m', 'node', run_as_root=True, root_helper=self._root_helper, check_exit_code=False)
And we only see the error coming from parsing the output of that command that is not logged.
Yes, I wonder if this is related to a rootwrap stdout/stderr capturing or something?
I believe Matthew is right in his assessment, the problem is the output from "iscsiadm -m node", there is a missing space between the first 2 columns in the output [4].
This looks like an issue in Open iSCSI, not in OS-Brick, Cinder, or Nova.
And checking their code, it looks like this is the patch that fixes it [5], so it needs to be added to F29 iscsi-initiator-utils package.
Thank you! This excellent detective work has solved the problem. I did a copr build with that patch [1] and got a good tempest run [2]. Amazing how much trouble a " " can cause. I have filed an upstream bug on the package https://bugzilla.redhat.com/show_bug.cgi?id=1676365 Anyway, it has led to a series of patches you may be interested in, which I think would help future debugging efforts https://review.openstack.org/636078 : fix for quoting of devstack args (important for follow-ons) https://review.openstack.org/636079 : export all journal logs. Things like iscsid were logging to the journal, but we weren't capturing them. Includes instructions on how to use the exported journal [3] https://review.openstack.org/636080 : add a tcpdump service. With this you can easily packet capture during a devstack run. e.g. https://review.openstack.org/636082 captures all iscsi traffic and stores it [4] https://review.openstack.org/636081 : iscsid debug option, which uses a systemd override to turn up debug logging. Reviews welcome :) Thanks, -i [1] https://github.com/open-iscsi/open-iscsi/commit/baa0cb45cfcf10a81283c191b0b2... [2] http://logs.openstack.org/82/636082/9/check/devstack-platform-fedora-latest/... [3] http://logs.openstack.org/82/636082/9/check/devstack-platform-fedora-latest/... [4] http://logs.openstack.org/82/636082/9/check/devstack-platform-fedora-latest/...