<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >Hi Lee,<br><br>I observed some recent failures in the integration tests run on Power machines. The failures indicate the VM failed to boot on ppc. Here are the logs- <a href="https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/42/769942/2/check/tempest-dsvm-full-focal-py3/8811347/job-output.txt">https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/42/769942/2/check/tempest-dsvm-full-focal-py3/8811347/job-output.txt</a><br>Could you please help identify the cause?<br><br>Thanks,<br>Aditi Dukle</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: aditi Dukle/India/Contr/IBM<br>To: lyarwood@redhat.com<br>Cc: Michael J Turek/Poughkeepsie/IBM@IBM, openstack-discuss@lists.openstack.org, Sajauddin Mohammad/India/Contr/IBM@IBM<br>Subject: Re: [EXTERNAL] Re: [nova] unit testing on ppc64le<br>Date: Fri, Jan 22, 2021 12:43 PM<br>
<div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >Hi Lee,<br><br>Thanks for getting this fixed. I re-ran the job using the change <a href="https://review.opendev.org/c/openstack/nova/+/741545/" target="_blank">https://review.opendev.org/c/openstack/nova/+/741545/</a> and we have 0 failures.<br>Here are the test results - <a href="https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/periodic/openstack-tox-py39/2021-01-22-0709-66b8935/job-output.txt" target="_blank">https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/periodic/openstack-tox-py39/2021-01-22-0709-66b8935/job-output.txt</a><br><br>Thanks,<br>Aditi Dukle</div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Lee Yarwood <lyarwood@redhat.com><br>To: aditi Dukle <aditi.Dukle@ibm.com><br>Cc: Michael J Turek <mjturek@us.ibm.com>, Sajauddin Mohammad <Sajauddin.Mohammad@ibm.com>, openstack-discuss@lists.openstack.org<br>Subject: [EXTERNAL] Re: [nova] unit testing on ppc64le<br>Date: Thu, Jan 21, 2021 6:42 PM<br>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >On 21-01-21 09:22:44, Lee Yarwood wrote:<br>> On Tue, 19 Jan 2021 at 20:28, aditi Dukle <aditi.Dukle@ibm.com> wrote:<br>> ><br>> > Hi Mike,<br>> ><br>> > I have started nova unit test jobs in the periodic pipeline(runs everyday at UTC- 12) for each openstack branch as follows:<br>> > periodic:<br>> > jobs:<br>> > - openstack-tox-py27:<br>> > branches:<br>> > - stable/ocata<br>> > - stable/pike<br>> > - stable/queens<br>> > - stable/rocky<br>> > - stable/stein<br>> > - stable/train<br>> > - openstack-tox-py36:<br>> > branches:<br>> > - stable/train<br>> > - stable/ussuri<br>> > - stable/victoria<br>> > - openstack-tox-py37:<br>> > branches:<br>> > - stable/train<br>> > - stable/ussuri<br>> > - openstack-tox-py38:<br>> > branches:<br>> > - stable/victoria<br>> > - openstack-tox-py39:<br>> > branches:<br>> > - master<br>> ><br>> > I have observed a few failures in the unit test cases mostly all related to volume drivers. Please have a look at the 14 test cases that are failing in openstack-tox-py39 job( <a href="https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/periodic/openstack-tox-py39/2021-01-19-0058-38c70ae/job-output.txt" target="_blank">https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/periodic/openstack-tox-py39/2021-01-19-0058-38c70ae/job-output.txt</a> ). Most of the 14 failures report these errors:<br>> > InvalidConnectorProtocol: Invalid InitiatorConnector protocol specified NVME<br>> > InvalidConnectorProtocol: Invalid InitiatorConnector protocol specified SCALEIO<br>> ><br>> > I would need some help in understanding if these connectors(NVME,SCALEIO) are supported on ppc64.<br>><br>> Right os-brick doesn't support these connectors on ppc64 but the real<br>> issue here is with the unit tests that don't mock out calls os-brick,<br>> an out of tree lib. I've filed a bug for this below:<br>><br>> LibvirtNVMEVolumeDriverTestCase and LibvirtScaleIOVolumeDriverTestCase<br>> unit tests fail on ppc64<br>> <a href="https://bugs.launchpad.net/nova/+bug/1912608" target="_blank">https://bugs.launchpad.net/nova/+bug/1912608</a> <br>><br>> I'll push a fix for this shortly.<br><br>libvirt: Stop NVMe and ScaleIO unit tests from calling os-brick<br><a href="https://review.opendev.org/c/openstack/nova/+/771806/" target="_blank">https://review.opendev.org/c/openstack/nova/+/771806/</a> <br> <br>> In addition to these failures I also see stable rescue tests failing,<br>> these should be fixed by the following change:<br>><br>> libvirt: Mock get_arch during some stable rescue unit tests<br>> <a href="https://review.opendev.org/c/openstack/nova/+/769916/" target="_blank">https://review.opendev.org/c/openstack/nova/+/769916/</a> <br><br>I also noticed some additional failures caused by the way in which the<br>libvirt virt driver loads its volume drivers at startup that also<br>attempt to load the underlying os-brick connector. As above this results<br>in volume drivers failing to load and being dropped.<br><br>We've actually had a long standing TODO in the driver to move to loading<br>these drivers on-demand so I've proposed the following:<br><br>libvirt: Load and cache volume drivers on-demand<br><a href="https://review.opendev.org/c/openstack/nova/+/741545/" target="_blank">https://review.opendev.org/c/openstack/nova/+/741545/</a> <br><br>Can you rerun your tests using the above change and I'll try to address<br>any additional failures.<br> <br>> Finally, in an effort to root out any further issues caused by tests<br>> looking up the arch of the test host I've pushed the following change<br>> to poison nova.objects.fields.Architecture.from_host:<br>><br>> tests: Poison nova.objects.fields.Architecture.from_host<br>> <a href="https://review.opendev.org/c/openstack/nova/+/769920" target="_blank">https://review.opendev.org/c/openstack/nova/+/769920</a> <br>><br>> There's a huge amount of fallout from this that I'll try to address in<br>> the coming weeks ahead of M3.<br>><br>> Hope this helps!<br>><br>> Lee<br>><br>> > ----- Original message -----<br>> > From: aditi Dukle/India/Contr/IBM<br>> > To: Michael J Turek/Poughkeepsie/IBM@IBM<br>> > Cc: Sajauddin Mohammad/India/Contr/IBM@IBM, openstack-discuss@lists.openstack.org<br>> > Subject: Re: [EXTERNAL] [nova] unit testing on ppc64le<br>> > Date: Tue, Jan 12, 2021 5:41 PM<br>> ><br>> > Hi Mike,<br>> ><br>> > I have created these unit test jobs - openstack-tox-py27, openstack-tox-py35, openstack-tox-py36, openstack-tox-py37, openstack-tox-py38, openstack-tox-py39<br>> > by referring to the upstream CI( <a href="https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml" target="_blank">https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml</a> ) and these jobs are triggered for every patchset in the Openstack CI.<br>> ><br>> > I checked the code for old CI for Power, we didn't have any unit test jobs that were run for every patchset for nova. We had one "nova-python27" job that was run in a periodic pipeline. So, I wanted to know if we need to run the unit test jobs on ppc for every patchset for nova? and If yes, should these be reporting to the Openstack community?<br>> ><br>> ><br>> > Thanks,<br>> > Aditi Dukle<br>> ><br>> > ----- Original message -----<br>> > From: Michael J Turek/Poughkeepsie/IBM<br>> > To: balazs.gibizer@est.tech, aditi Dukle/India/Contr/IBM@IBM, Sajauddin Mohammad/India/Contr/IBM@IBM<br>> > Cc: openstack-discuss@lists.openstack.org<br>> > Subject: Re: [EXTERNAL] [nova] unit testing on ppc64le<br>> > Date: Sat, Jan 9, 2021 12:52 AM<br>> ><br>> > Thanks for the heads up,<br>> ><br>> > We should have the capacity to add them. At one point I think we ran unit tests for nova but the job may have been culled in the move to zuul v3. I've CC'd the maintainers of the CI, Aditi Dukle and Sajauddin Mohammad.<br>> ><br>> > Aditi and Sajauddin, could we add a job to pkvmci to run unit tests for nova?<br>> ><br>> > Michael Turek<br>> > Software Engineer<br>> > Power Cloud Department<br>> > 1 845 433 1290 Office<br>> > mjturek@us.ibm.com<br>> > He/Him/His<br>> ><br>> > IBM<br>> ><br>> ><br>> ><br>> ><br>> > ----- Original message -----<br>> > From: Balazs Gibizer <balazs.gibizer@est.tech><br>> > To: OpenStack Discuss <openstack-discuss@lists.openstack.org><br>> > Cc: mjturek@us.ibm.com<br>> > Subject: [EXTERNAL] [nova] unit testing on ppc64le<br>> > Date: Fri, Jan 8, 2021 7:59 AM<br>> ><br>> > Hi,<br>> ><br>> > We have a bugreport[1] showing that our unit tests are not passing on<br>> > ppc. In the upstream CI we don't have test capability to run our tests<br>> > on ppc. But we have the IBM Power KVM CI[2] that runs integration tests<br>> > on ppc. I'm wondering if IBM could extend the CI to run nova unit and<br>> > functional tests too. I've added Michael Turek (mjturek@us.ibm.com) to<br>> > CC. Michael is listed as the contact person for the CI.<br>> ><br>> > Cheers,<br>> > gibi<br>> ><br>> > [1]<a href="https://bugs.launchpad.net/nova/+bug/1909972" target="_blank">https://bugs.launchpad.net/nova/+bug/1909972</a> <br>> > [2]<a href="https://wiki.openstack.org/wiki/ThirdPartySystems/IBMPowerKVMCI" target="_blank">https://wiki.openstack.org/wiki/ThirdPartySystems/IBMPowerKVMCI</a> <br>> ><br>> ><br>> ><br>> ><br>> ><br>> ><br>> ><br>> ><br>> ><br><br>--<br>Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76</font><br> </div></blockquote>
<div dir="ltr" > </div></div></blockquote>
<div dir="ltr" > </div></div><BR>