<div dir="ltr"><div dir="ltr"><br></div><div>Thank you, Sylvain, for all these inputs !</div><div><br></div><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 30, 2023 at 7:10 PM Sylvain Bauza <<a href="mailto:sbauza@redhat.com">sbauza@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le jeu. 30 mars 2023 à 06:16, Takashi Kajinami <<a href="mailto:tkajinam@redhat.com" target="_blank">tkajinam@redhat.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello,</div><div><br></div><div><br></div><div>Since we migrated our jobs from Ubuntu Focal to Ubuntu Jammy, heat gate jobs have</div><div>become very flaky. Further investigation revealed that the issue is related to something</div><div>in libvirt from Ubuntu Jammy and that prevents detaching devices from instances[1].<br></div><div><br></div><div>The same problem appears in different jobs[2] and we workaround the problem by disabling</div><div>some affected jobs. In heat we also disabled some flaky tests but because of this we no longer</div><div>run basic scenario tests which deploys instance/volume/network in a single stack, which means</div><div>we lost the quite basic test coverage.<br></div><div><br></div><div>My question is, is there anyone in the Nova team working on "fixing" this problem ?</div><div>We might be able to implement some workaround (like checking status of the instances before</div><div>attempting to delete it) but this should be fixed in libvirt side IMO, as this looks like a "regression"</div><div>in Ubuntu Jammy.<br></div><div>Probably we should report a bug against the libvirt package in Ubuntu but I'd like to hear some</div><div>thoughts from the nova team because they are more directly affected by this problem.</div><div><br></div></div></blockquote><div><br></div><div><br></div><div>FWIW, we discussed about it yesterday on our vPTG : <br></div><div><a href="https://etherpad.opendev.org/p/nova-bobcat-ptg#L289" target="_blank">https://etherpad.opendev.org/p/nova-bobcat-ptg#L289</a></div><div><br></div><div>Most of the problems come from the volume detach thing. We also merged some Tempest changes for not trying to cleanup some volumes if the test was OK (thanks Dan for this). We also added more verifications to ask SSH to wait for a bit of time before calling the instance.</div><div>Eventually, as you see in the etherpad, we didn't found any solutions but we'll try to add some canary job for testing multiple times volume attachs/detachs. <br></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div><br></div><div>We'll also continue to discuss on the CI failures during every Nova weekly meetings (Tuesdays@1600UTC on #openstack-nova) and I'll want to ask a cross-project session for the Vancouver pPTG for Tempest/Cinder/Nova and others.</div></div><div class="gmail_quote">I leave other SMEs to reply on your other points, like for c9s.</div></div></blockquote><div><br></div><div>It's good to hear that the issue is still getting attention. I'll catch up the discussion by reading the etherpad<br>and will try to attend follow-up discussions if possible, especially if I can attend Vancouver vPTG.<br><br>I know some changes have been proposed to check ssh-ability to workaround the problem (though</div><div>the comment in the vPTG session indicates that does not fully solve the problem) but it's still annoying</div><div>because we don't really block resource deletions based on instance status (especially its internal status)</div><div>so we eventually need some solutions here to avoid this problem, IMHO.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>I'm now trying to set up a centos stream 9 job in Heat repo to see whether this can be reproduced</div><div>if we use centos stream 9. I've been running that specific scenario test in centos stream 9 jobs</div><div>in puppet repos but I've never seen this issue, so I suspect the issue is really specific to libvirt</div><div>in Jammy.<br></div></div></blockquote><div><br></div><div><br></div><div>Well, maybe I'm wrong, but no, we also have a centos9stream issue for volume detachs : <br></div><div><a href="https://bugs.launchpad.net/nova/+bug/1960346" target="_blank">https://bugs.launchpad.net/nova/+bug/1960346</a></div><div><br></div></div></div></blockquote><div><br></div><div>I just managed to launch a c9s job in heat but it seems the issue is reproducible in c9s as well[1].</div><div>I'll rerun the job a few more times to see how frequent the issue appears in c9s compared to</div><div>ubuntu.</div><div>We do not run many tests in puppet jobs so that might be the reason I've never hit it in</div><div>puppet jobs.</div><div><br></div><div>[1] <a href="https://review.opendev.org/c/openstack/heat/+/879014">https://review.opendev.org/c/openstack/heat/+/879014</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div>[1] <a href="https://bugs.launchpad.net/nova/+bug/1998274" target="_blank">https://bugs.launchpad.net/nova/+bug/1998274</a></div><div>[2] <a href="https://bugs.launchpad.net/nova/+bug/1998148" target="_blank">https://bugs.launchpad.net/nova/+bug/1998148</a></div><div><br></div><div>Thank you,</div><div>Takashi<br></div></div>
</blockquote></div></div>
</blockquote></div></div></div>