<div dir="ltr">Joe, <div><br></div><div>What about running benchmarks (with small load), for all major functions (like snaphshoting, booting/deleting, ..) on every patch in nova. It can catch a lot of related stuff. </div>
<div><br></div><div><br></div><div>Best regards,</div><div>Boris Pavlovic</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 9, 2014 at 1:56 AM, Michael Still <span dir="ltr"><<a href="mailto:mikal@stillhq.com" target="_blank">mikal@stillhq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The associated bug says this is probably a qemu bug, so I think we<br>
should rephrase that to "we need to start thinking about how to make<br>
sure upstream changes don't break nova".<br>
<br>
Michael<br>
<div class="HOEnZb"><div class="h5"><br>
On Wed, Jul 9, 2014 at 7:50 AM, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>> wrote:<br>
><br>
> On Thu, Jun 26, 2014 at 4:12 AM, Daniel P. Berrange <<a href="mailto:berrange@redhat.com">berrange@redhat.com</a>><br>
> wrote:<br>
>><br>
>> On Thu, Jun 26, 2014 at 07:00:32AM -0400, Sean Dague wrote:<br>
>> > While the Trusty transition was mostly uneventful, it has exposed a<br>
>> > particular issue in libvirt, which is generating ~ 25% failure rate now<br>
>> > on most tempest jobs.<br>
>> ><br>
>> > As can be seen here -<br>
>> ><br>
>> > <a href="https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L294-L297" target="_blank">https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L294-L297</a><br>
>> ><br>
>> ><br>
>> > ... the libvirt live_snapshot code is something that our test pipeline<br>
>> > has never tested before, because it wasn't a new enough libvirt for us<br>
>> > to take that path.<br>
>> ><br>
>> > Right now it's exploding, a lot -<br>
>> > <a href="https://bugs.launchpad.net/nova/+bug/1334398" target="_blank">https://bugs.launchpad.net/nova/+bug/1334398</a><br>
>> ><br>
>> > Snapshotting gets used in Tempest to create images for testing, so image<br>
>> > setup tests are doing a decent number of snapshots. If I had to take a<br>
>> > completely *wild guess*, it's that libvirt can't do 2 live_snapshots at<br>
>> > the same time. It's probably something that most people haven't hit. The<br>
>> > wild guess is based on other libvirt issues we've hit that other people<br>
>> > haven't, and they are basically always a parallel ops triggered problem.<br>
>> ><br>
>> > My 'stop the bleeding' suggested fix is this -<br>
>> > <a href="https://review.openstack.org/#/c/102643/" target="_blank">https://review.openstack.org/#/c/102643/</a> which just effectively disables<br>
>> > this code path for now. Then we can get some libvirt experts engaged to<br>
>> > help figure out the right long term fix.<br>
>><br>
>> Yes, this is a sensible pragmatic workaround for the short term until<br>
>> we diagnose the root cause & fix it.<br>
>><br>
>> > I think there are a couple:<br>
>> ><br>
>> > 1) see if newer libvirt fixes this (1.2.5 just came out), and if so<br>
>> > mandate at some known working version. This would actually take a bunch<br>
>> > of work to be able to test a non packaged libvirt in our pipeline. We'd<br>
>> > need volunteers for that.<br>
>> ><br>
>> > 2) lock snapshot operations in nova-compute, so that we can only do 1 at<br>
>> > a time. Hopefully it's just 2 snapshot operations that is the issue, not<br>
>> > any other libvirt op during a snapshot, so serializing snapshot ops in<br>
>> > n-compute could put the kid gloves on libvirt and make it not break<br>
>> > here. This also needs some volunteers as we're going to be playing a<br>
>> > game of progressive serialization until we get to a point where it looks<br>
>> > like the failures go away.<br>
>> ><br>
>> > 3) Roll back to precise. I put this idea here for completeness, but I<br>
>> > think it's a terrible choice. This is one isolated, previously untested<br>
>> > (by us), code path. We can't stay on libvirt 0.9.6 forever, so actually<br>
>> > need to fix this for real (be it in nova's use of libvirt, or libvirt<br>
>> > itself).<br>
>><br>
>> Yep, since we *never* tested this code path in the gate before, rolling<br>
>> back to precise would not even really be a fix for the problem. It would<br>
>> merely mean we're not testing the code path again, which is really akin<br>
>> to sticking our head in the sand.<br>
>><br>
>> > But for right now, we should stop the bleeding, so that nova/libvirt<br>
>> > isn't blocking everyone else from merging code.<br>
>><br>
>> Agreed, we should merge the hack and treat the bug as release blocker<br>
>> to be resolve prior to Juno GA.<br>
><br>
><br>
><br>
> How can we prevent libvirt issues like this from landing in trunk in the<br>
> first place? If we don't figure out a way to prevent this from landing the<br>
> first place I fear we will keep repeating this same pattern of failure.<br>
><br>
>><br>
>><br>
>> Regards,<br>
>> Daniel<br>
>> --<br>
>> |: <a href="http://berrange.com" target="_blank">http://berrange.com</a> -o- <a href="http://www.flickr.com/photos/dberrange/" target="_blank">http://www.flickr.com/photos/dberrange/</a><br>
>> :|<br>
>> |: <a href="http://libvirt.org" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" target="_blank">http://virt-manager.org</a><br>
>> :|<br>
>> |: <a href="http://autobuild.org" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" target="_blank">http://search.cpan.org/~danberr/</a><br>
>> :|<br>
>> |: <a href="http://entangle-photo.org" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" target="_blank">http://live.gnome.org/gtk-vnc</a><br>
>> :|<br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Rackspace Australia<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>