Open Stack

Hey Everyone,

Thought I'd reach out to the ML to see if somebody might have some insight
or suggestions to a problem I've been trying to solve.

The short summary is:

During a dsvm-full run in the gate there are times when /dev/sdX devices on
the system may be created/deleted.  The trick here though is that on the
Cinder side with LVM we're doing a lot of things that rely on VGS and LVS
calls (including periodic tasks that are run).  Part of the scan routine
unfortunately is for LVM to go through and open any block devices that is
sees on the system and read them to see if it's an LVM device.

The problem here is that the timing in the gate tests when everything is on
a single node can result in udev not being quite up to date and the LVM
scan process attempts to open a device that is no longer valid.  In this
case (which we hit a few times on every single gate test), LVM blocks on
the Open until the device times out and gives:
    -1 ENXIO (No such device or address)

The problem is this can take up to almost a full minute for the timeout, so
we have a few tests that take upwards of 150 seconds that actually should
complete in about 30 seconds.  In addition this causes things like
concurrent lvcreate cmds to block as well.  Note this is kind of
inefficient anyway (even if all of the devices are valid), so there is a
case to be made for not doing it if possible.

Nothing fails typically in this scenario, things are just "slow".

I thought this would be easy to fix a while back by adding a local lvm.conf
with a device filter.  It turns out however that the device filter only
filters out items AFTER the vgs or lvs, it doesn't filter out the opens.
For that you need either:
1. global_filter
2. lvmetad service enabled

The problem with '#1' is that the global_filter parameter is only honored
on a global level NOT just in a local lvm.conf like we have currently.  To
use that though we would have to set things such that Cinder was the only
thing using LVM (not sure if that's doable or not).

The problem with '#2' is that Trusty doesn't seem to have lvmetad in it's
lvm2 packages until 2.02.111 (which isn't introduced until Vivid).

I'm wondering if anybody knows of a backport or another method to get
lvmetad capability in Ubuntu Trusty?

OR

What are some thoughts regarding the addition of a global_filter in
/etc/lvm.conf?  We'd have to make a number of modifications to any services
in devstack that setup LVM to make sure their PV's are added to the
filter.  This might not be a big deal because most everyone uses loopback
files so we could just to a loop regex and hit everything in one shot (I
think).  But this means that anybody using LVM in devstack for something
else is going to need to understand what's going on and add their devices
to the filter.

Ok... so not really the short version after all, but I'm wondering if
anybody has any ideas that maybe I'm missing here.  I'll likely proceed
with the idea of a global filter later this week if I don't hear any strong
objections, or even better maybe somebody knows how to get lvmetad on
Trusty which I *think* would be ideal for a number of other reasons.

Thanks,
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150316/6657c96b/attachment.html>