[nova]Integration tests failing on ppc64le master branch

Sean Mooney smooney at redhat.com
Thu Jan 28 14:55:37 UTC 2021


On Thu, 2021-01-28 at 08:10 +0000, aditi Dukle wrote:
> Hi Lee,
> 
> I observed some recent failures in the integration tests run on Power machines. The failures indicate the VM failed to boot on ppc. Here are the
> logs- https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/42/769942/2/check/tempest-dsvm-full-focal-py3/8811347/job-output.txt
> Could you please help identify the cause?

added nova to the subject.

looking at the n-cpu logs the error is pretty clear

Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: ERROR nova.virt.libvirt.guest [None req-ffe2d074-e828-4851-
a8da-3d0aecdb5873 tempest-ServerDiskConfigTestJSON-1142390778 tempest-ServerDiskConfigTestJSON-1142390778-project] Error defining a guest with XML:
<domain type="kvm">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <uuid>97ddda4b-1b4b-4c2d-a77f-087faeae06ae</uuid>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <name>instance-00000021</name>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <memory>131072</memory>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <vcpu>1</vcpu>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <metadata>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <nova:instance
xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <nova:package version="22.1.0"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <nova:name>tempest-ServerDiskConfigTestJSON-server-
1335737010</nova:name>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <nova:creationTime>2021-01-28
00:11:24</nova:creationTime>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <nova:flavor name="m1.nano">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:         <nova:memory>128</nova:memory>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:         <nova:disk>1</nova:disk>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:         <nova:swap>0</nova:swap>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:         <nova:ephemeral>0</nova:ephemeral>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:         <nova:vcpus>1</nova:vcpus>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       </nova:flavor>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <nova:owner>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:         <nova:user
uuid="89b5ea0dad3348d6be9a94d329e41901">tempest-ServerDiskConfigTestJSON-1142390778-project</nova:user>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:         <nova:project
uuid="37329f48690d4985af6f125e7bd9d529">tempest-ServerDiskConfigTestJSON-1142390778</nova:project>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       </nova:owner>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <nova:root type="image" uuid="74cc0df8-6b07-4b4e-a1db-
39777ecf457b"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     </nova:instance>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   </metadata>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <os>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <type machine="pseries-3.1">hvm</type>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <boot dev="hd"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   </os>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <features>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <acpi/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <apic/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   </features>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <cputune>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <shares>1024</shares>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   </cputune>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <clock offset="utc">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <timer name="pit" tickpolicy="delay"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <timer name="rtc" tickpolicy="catchup"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   </clock>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <cpu match="exact">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <topology sockets="1" cores="1" threads="1"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   </cpu>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   <devices>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <disk type="file" device="disk">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <driver name="qemu" type="qcow2" cache="none"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <source file="/opt/stack/data/nova/instances/97ddda4b-
1b4b-4c2d-a77f-087faeae06ae/disk"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <target dev="vda" bus="virtio"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     </disk>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <interface type="bridge">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <mac address="fa:16:3e:30:92:85"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <model type="virtio"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <source bridge="br-int"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <mtu size="1450"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <target dev="tap68762b55-ad"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <virtualport type="openvswitch">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:         <parameters interfaceid="68762b55-adb4-48c6-8f4c-
b5025b2787cd"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       </virtualport>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     </interface>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <serial type="pty">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <log file="/opt/stack/data/nova/instances/97ddda4b-
1b4b-4c2d-a77f-087faeae06ae/console.log" append="off"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     </serial>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <graphics type="vnc" autoport="yes" listen="0.0.0.0"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <video>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <model type="vga"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     </video>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <rng model="virtio">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <backend model="random">/dev/urandom</backend>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     </rng>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <controller type="usb" index="0" model="none"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     <memballoon model="virtio">
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:       <stats period="10"/>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:     </memballoon>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]:   </devices>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </domain>
Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: : libvirt.libvirtError: unsupported configuration: USB is
disabled for this domain, but USB devices are present in the domain XML
Jan 28 00:11:24.746856 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: ERROR nova.virt.libvirt.driver [None req-ffe2d074-e828-4851-
a8da-3d0aecdb5873 tempest-ServerDiskConfigTestJSON-1142390778 tempest-ServerDiskConfigTestJSON-1142390778-project] [instance: 97ddda4b-1b4b-4c2d-a77f-
087faeae06ae] Failed to start libvirt guest: libvirt.libvirtError: unsupported configuration: USB is disabled for this domain, but USB devices are
present in the domain XML

so this is proably a result of https://github.com/openstack/nova/commit/c34a17db6f1ed4469d963a4e6426594602939912
stephen can you take a look

<controller type="usb" index="0" model="none"/> is there
but im not seeing anything use the usb device
im guessing we need to disable one that is auto added by libvirt on ppc

aditi can you provide an xml form a vm booted on ppc normally before this change.
specifcally the fully populated xml retrived form libvirt with virsh dumpxml so we can se what it adds
> 
> Thanks,
> Aditi Dukle
>  
>  
> > ----- Original message -----
> > From: aditi Dukle/India/Contr/IBM
> > To: lyarwood at redhat.com
> > Cc: Michael J Turek/Poughkeepsie/IBM at IBM, openstack-discuss at lists.openstack.org, Sajauddin Mohammad/India/Contr/IBM at IBM
> > Subject: Re: [EXTERNAL] Re: [nova] unit testing on ppc64le
> > Date: Fri, Jan 22, 2021 12:43 PM
> >  
> > Hi Lee,
> > 
> > Thanks for getting this fixed. I re-ran the job using the change https://review.opendev.org/c/openstack/nova/+/741545/ and we have 0 failures.
> > Here are the test results - https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/periodic/openstack-tox-py39/2021-01-22-0709-66b8935/job-
> > output.txt
> > 
> > Thanks,
> > Aditi Dukle
> >  
> > > ----- Original message -----
> > > From: Lee Yarwood <lyarwood at redhat.com>
> > > To: aditi Dukle <aditi.Dukle at ibm.com>
> > > Cc: Michael J Turek <mjturek at us.ibm.com>, Sajauddin Mohammad <Sajauddin.Mohammad at ibm.com>, openstack-discuss at lists.openstack.org
> > > Subject: [EXTERNAL] Re: [nova] unit testing on ppc64le
> > > Date: Thu, Jan 21, 2021 6:42 PM
> > >  
> > > On 21-01-21 09:22:44, Lee Yarwood wrote:
> > > > On Tue, 19 Jan 2021 at 20:28, aditi Dukle <aditi.Dukle at ibm.com> wrote:
> > > > > 
> > > > > Hi Mike,
> > > > > 
> > > > > I have started nova unit test jobs in the periodic pipeline(runs everyday at UTC- 12) for each openstack branch as follows:
> > > > > periodic:
> > > > >       jobs:
> > > > >         - openstack-tox-py27:
> > > > >             branches:
> > > > >               - stable/ocata
> > > > >               - stable/pike
> > > > >               - stable/queens
> > > > >               - stable/rocky
> > > > >               - stable/stein
> > > > >               - stable/train
> > > > >         - openstack-tox-py36:
> > > > >             branches:
> > > > >               - stable/train
> > > > >               - stable/ussuri
> > > > >               - stable/victoria
> > > > >         - openstack-tox-py37:
> > > > >             branches:
> > > > >               - stable/train
> > > > >               - stable/ussuri
> > > > >         - openstack-tox-py38:
> > > > >             branches:
> > > > >               - stable/victoria
> > > > >         - openstack-tox-py39:
> > > > >             branches:
> > > > >               - master
> > > > > 
> > > > > I have observed a few failures in the unit test cases mostly all related to volume drivers. Please have a look at the 14 test cases that are
> > > > > failing in openstack-tox-py39 job( https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/periodic/openstack-tox-py39/2021-01-19-0058-
> > > > > 38c70ae/job-output.txt ).  Most of the 14 failures report these errors:
> > > > > InvalidConnectorProtocol: Invalid InitiatorConnector protocol specified NVME
> > > > > InvalidConnectorProtocol: Invalid InitiatorConnector protocol specified SCALEIO
> > > > > 
> > > > > I would need some help in understanding if these connectors(NVME,SCALEIO) are supported on ppc64.
> > > > 
> > > > Right os-brick doesn't support these connectors on ppc64 but the real
> > > > issue here is with the unit tests that don't mock out calls os-brick,
> > > > an out of tree lib. I've filed a bug for this below:
> > > > 
> > > > LibvirtNVMEVolumeDriverTestCase and LibvirtScaleIOVolumeDriverTestCase
> > > > unit tests fail on ppc64
> > > > https://bugs.launchpad.net/nova/+bug/1912608 
> > > > 
> > > > I'll push a fix for this shortly.
> > > 
> > > libvirt: Stop NVMe and ScaleIO unit tests from calling os-brick
> > > https://review.opendev.org/c/openstack/nova/+/771806/ 
> > >  
> > > > In addition to these failures I also see stable rescue tests failing,
> > > > these should be fixed by the following change:
> > > > 
> > > > libvirt: Mock get_arch during some stable rescue unit tests
> > > > https://review.opendev.org/c/openstack/nova/+/769916/ 
> > > 
> > > I also noticed some additional failures caused by the way in which the
> > > libvirt virt driver loads its volume drivers at startup that also
> > > attempt to load the underlying os-brick connector. As above this results
> > > in volume drivers failing to load and being dropped.
> > > 
> > > We've actually had a long standing TODO in the driver to move to loading
> > > these drivers on-demand so I've proposed the following:
> > > 
> > > libvirt: Load and cache volume drivers on-demand
> > > https://review.opendev.org/c/openstack/nova/+/741545/ 
> > > 
> > > Can you rerun your tests using the above change and I'll try to address
> > > any additional failures.
> > >  
> > > > Finally, in an effort to root out any further issues caused by tests
> > > > looking up the arch of the test host I've pushed the following change
> > > > to poison  nova.objects.fields.Architecture.from_host:
> > > > 
> > > > tests: Poison nova.objects.fields.Architecture.from_host
> > > > https://review.opendev.org/c/openstack/nova/+/769920 
> > > > 
> > > > There's a huge amount of fallout from this that I'll try to address in
> > > > the coming weeks ahead of M3.
> > > > 
> > > > Hope this helps!
> > > > 
> > > > Lee
> > > > 
> > > > > ----- Original message -----
> > > > > From: aditi Dukle/India/Contr/IBM
> > > > > To: Michael J Turek/Poughkeepsie/IBM at IBM
> > > > > Cc: Sajauddin Mohammad/India/Contr/IBM at IBM, openstack-discuss at lists.openstack.org
> > > > > Subject: Re: [EXTERNAL] [nova] unit testing on ppc64le
> > > > > Date: Tue, Jan 12, 2021 5:41 PM
> > > > > 
> > > > > Hi Mike,
> > > > > 
> > > > > I have created these unit test jobs - openstack-tox-py27, openstack-tox-py35, openstack-tox-py36, openstack-tox-py37, openstack-tox-py38,
> > > > > openstack-tox-py39
> > > > > by referring to the upstream CI( https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml ) and these jobs are
> > > > > triggered for every patchset in the Openstack CI.
> > > > > 
> > > > > I checked the code for old CI for Power, we didn't have any unit test jobs that were run for every patchset for nova. We had one "nova-
> > > > > python27" job that was run in a periodic pipeline. So, I wanted to know if we need to run the unit test jobs on ppc for every patchset for
> > > > > nova? and If yes, should these be reporting to the Openstack community?
> > > > > 
> > > > > 
> > > > > Thanks,
> > > > > Aditi Dukle
> > > > > 
> > > > > ----- Original message -----
> > > > > From: Michael J Turek/Poughkeepsie/IBM
> > > > > To: balazs.gibizer at est.tech, aditi Dukle/India/Contr/IBM at IBM, Sajauddin Mohammad/India/Contr/IBM at IBM
> > > > > Cc: openstack-discuss at lists.openstack.org
> > > > > Subject: Re: [EXTERNAL] [nova] unit testing on ppc64le
> > > > > Date: Sat, Jan 9, 2021 12:52 AM
> > > > > 
> > > > > Thanks for the heads up,
> > > > > 
> > > > > We should have the capacity to add them. At one point I think we ran unit tests for nova but the job may have been culled in the move to
> > > > > zuul v3. I've CC'd the maintainers of the CI, Aditi Dukle and Sajauddin Mohammad.
> > > > > 
> > > > > Aditi and Sajauddin, could we add a job to pkvmci to run unit tests for nova?
> > > > > 
> > > > > Michael Turek
> > > > > Software Engineer
> > > > > Power Cloud Department
> > > > > 1 845 433 1290 Office
> > > > > mjturek at us.ibm.com
> > > > > He/Him/His
> > > > > 
> > > > > IBM
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > ----- Original message -----
> > > > > From: Balazs Gibizer <balazs.gibizer at est.tech>
> > > > > To: OpenStack Discuss <openstack-discuss at lists.openstack.org>
> > > > > Cc: mjturek at us.ibm.com
> > > > > Subject: [EXTERNAL] [nova] unit testing on ppc64le
> > > > > Date: Fri, Jan 8, 2021 7:59 AM
> > > > > 
> > > > > Hi,
> > > > > 
> > > > > We have a bugreport[1] showing that our unit tests are not passing on
> > > > > ppc. In the upstream CI we don't have test capability to run our tests
> > > > > on ppc. But we have the IBM Power KVM CI[2] that runs integration tests
> > > > > on ppc. I'm wondering if IBM could extend the CI to run nova unit and
> > > > > functional tests too. I've added Michael Turek (mjturek at us.ibm.com) to
> > > > > CC. Michael is listed as the contact person for the CI.
> > > > > 
> > > > > Cheers,
> > > > > gibi
> > > > > 
> > > > > [1]https://bugs.launchpad.net/nova/+bug/1909972 
> > > > > [2]https://wiki.openstack.org/wiki/ThirdPartySystems/IBMPowerKVMCI 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > 
> > > --
> > > Lee Yarwood                 A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76
> > >  
> >  
>  
> 





More information about the openstack-discuss mailing list