[cinder][nova] Running parallel iSCSI/LVM c-vol backends is causing random failures in CI

Lee Yarwood lyarwood at redhat.com
Wed Mar 10 09:46:26 UTC 2021


On Tue, 9 Mar 2021 at 23:11, Luigi Toscano <ltoscano at redhat.com> wrote:
> On Wednesday, 10 March 2021 00:00:29 CET Sean Mooney wrote:
> > On Tue, 2021-03-09 at 23:50 +0100, Luigi Toscano wrote:
> > > On Tuesday, 9 March 2021 23:40:08 CET Lee Yarwood wrote:
> > > > On Tue, 9 Mar 2021 at 22:24, Jeremy Stanley <fungi at yuggoth.org> wrote:
> > > > > On 2021-03-09 22:19:48 +0000 (+0000), Jeremy Stanley wrote:
> > > > > > On 2021-03-09 22:18:12 +0000 (+0000), Jeremy Stanley wrote:
> > > > > > > On 2021-03-09 22:08:18 +0000 (+0000), Lee Yarwood wrote:
> > > > > > > > I reported the following bug last week but I've yet to get any
> > > > > > > > real
> > > > > > > > feedback after asking a few times in irc.
> > > > > > > >
> > > > > > > > Running parallel iSCSI/LVM c-vol backends is causing random
> > > > > > > > failures
> > > > > > > > in CI
> > > > > > > > https://bugs.launchpad.net/cinder/+bug/1917750
> > > > > > > >
> > > > > > > > AFAICT tgtadm is causing this behaviour. As I've stated in the
> > > > > > > > bug
> > > > > > > > with Fedora 32 and lioadm I don't see the WWN conflict between
> > > > > > > > the
> > > > > > > > two
> > > > > > > > backends. Does anyone know if using lioadm is an option on
> > > > > > > > Focal?
> > > > > > >
> > > > > > > https://docs.openstack.org/cinder/latest/admin/blockstorage-lio-is
> > > > > > > csi-> > > > support.html seems to indicate that you just need to
> > > > > > > set it in configuration. The package that document mentions looks
> > > > > > > like a distro package
> > > > > > > recommendation (does not exist under that name on PyPI) and the
> > > > > > > equivalent lib on PyPI is included in cinder's requirements.txt
> > > > > > > file, but I don't see either mentioned in the devstack source tree
> > > > > > > so maybe that needs to be installed for DevStack-based jobs to
> > > > > > > take
> > > > > > > advantage of it.
> > > > > >
> > > > > > Oh, and this would be the package to add from Ubuntu Focal:
> > > > > >
> > > > > > https://packages.ubuntu.com/focal/python3-rtslib-fb
> > > > >
> > > > > Nevermind, DevStack installs the projects with pip, so the one in
> > > > > cinder's requirements.txt should already be present. In that case,
> > > > > yeah, just set it in the config?
> > > >
> > > > Yes correct, my question was more to see if there were any known
> > > > issues with using lioadm on Focal. Anyway I've pushed the following
> > > > WIP change for devstack to switch over to lioadm when using Focal:
> > > >
> > > > WIP cinder: Default CINDER_ISCSI_HELPER to lioadm on Ubuntu
> > > > https://review.opendev.org/c/openstack/devstack/+/779624
> > >
> > > For the record, we use the default (tgt) on the main cinder gates, as well
> > > as a lioadm job (defined in cinder-tempest-plugin). If I read the code
> > > correctly, that change would break tgt for everyone on ubuntu.
> > >
> > > Please raise this in the cinder meeting (Wednesday).
> >
> > i dont think we can wait that long im pretty sure this is causing this error
> > form talking to lee earlier to day
> > http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:
> > %5C%22Unable%20to%20detach%20the%20device%20from%20the%20live%20config%5C%22
> > %20AND%20loglevel:%20ERROR
> >
> > i guess it almost wednesday already but its startin to casue issues in
> > multiple poject gates right before code freeze so we need to adress this
> > before we end up rechecking things over and over on many patches.

If M3 wasn't *tomorrow* I'd agree but I don't think anyone wants to
change the default iSCSI target this close to feature freeze. I've
also been unable to reproduce the above failure with a multi LVM/iSCSI
+ tgtadm setup so I'm not entirely confident about the switch to
lioadm actually resolving that issue at the moment.

> But then don't error out if someone tries to use tgtadm, which is what would
> happen if that patch was merged (if I didn't misread the code).

There's nothing stopping someone from declaring
CINDER_ISCSI_HELPER=tgtadm to override the default so I'm not sure
what you're suggesting.

To that end I've posted the following to ensure the single host tgtadm
jobs in the cinder-tempest-plugin use the correct target:

Set CINDER_ISCSI_HELPER explicitly for tgtadm job
https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/779697

If you know of anymore please let me know!

Cheers,

Lee




More information about the openstack-discuss mailing list