Open Stack

Can the bits that make those devices invalid and udev out of date call udev
admin --settle to just block till things are upto date and hopefully the
subseqent vg and pv scans quicker?

On Monday, March 16, 2015, John Griffith <john.griffith8 at gmail.com> wrote:

> Hey Everyone,
>
> Thought I'd reach out to the ML to see if somebody might have some insight
> or suggestions to a problem I've been trying to solve.
>
> The short summary is:
>
> During a dsvm-full run in the gate there are times when /dev/sdX devices
> on the system may be created/deleted.  The trick here though is that on the
> Cinder side with LVM we're doing a lot of things that rely on VGS and LVS
> calls (including periodic tasks that are run).  Part of the scan routine
> unfortunately is for LVM to go through and open any block devices that is
> sees on the system and read them to see if it's an LVM device.
>
> The problem here is that the timing in the gate tests when everything is
> on a single node can result in udev not being quite up to date and the LVM
> scan process attempts to open a device that is no longer valid.  In this
> case (which we hit a few times on every single gate test), LVM blocks on
> the Open until the device times out and gives:
>     -1 ENXIO (No such device or address)
>
> The problem is this can take up to almost a full minute for the timeout,
> so we have a few tests that take upwards of 150 seconds that actually
> should complete in about 30 seconds.  In addition this causes things like
> concurrent lvcreate cmds to block as well.  Note this is kind of
> inefficient anyway (even if all of the devices are valid), so there is a
> case to be made for not doing it if possible.
>
> Nothing fails typically in this scenario, things are just "slow".
>
> I thought this would be easy to fix a while back by adding a local
> lvm.conf with a device filter.  It turns out however that the device filter
> only filters out items AFTER the vgs or lvs, it doesn't filter out the
> opens.  For that you need either:
> 1. global_filter
> 2. lvmetad service enabled
>
> The problem with '#1' is that the global_filter parameter is only honored
> on a global level NOT just in a local lvm.conf like we have currently.  To
> use that though we would have to set things such that Cinder was the only
> thing using LVM (not sure if that's doable or not).
>
> The problem with '#2' is that Trusty doesn't seem to have lvmetad in it's
> lvm2 packages until 2.02.111 (which isn't introduced until Vivid).
>
> I'm wondering if anybody knows of a backport or another method to get
> lvmetad capability in Ubuntu Trusty?
>
> OR
>
> What are some thoughts regarding the addition of a global_filter in
> /etc/lvm.conf?  We'd have to make a number of modifications to any services
> in devstack that setup LVM to make sure their PV's are added to the
> filter.  This might not be a big deal because most everyone uses loopback
> files so we could just to a loop regex and hit everything in one shot (I
> think).  But this means that anybody using LVM in devstack for something
> else is going to need to understand what's going on and add their devices
> to the filter.
>
> Ok... so not really the short version after all, but I'm wondering if
> anybody has any ideas that maybe I'm missing here.  I'll likely proceed
> with the idea of a global filter later this week if I don't hear any strong
> objections, or even better maybe somebody knows how to get lvmetad on
> Trusty which I *think* would be ideal for a number of other reasons.
>
> Thanks,
> John
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150316/c48c0202/attachment.html>