On Thu, 2021-01-28 at 08:10 +0000, aditi Dukle wrote:
Hi Lee,
I observed some recent failures in the integration tests run on Power machines. The failures indicate the VM failed to boot on ppc. Here are the logs- https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/42/769942/2/che... Could you please help identify the cause?
added nova to the subject. looking at the n-cpu logs the error is pretty clear Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: ERROR nova.virt.libvirt.guest [None req-ffe2d074-e828-4851- a8da-3d0aecdb5873 tempest-ServerDiskConfigTestJSON-1142390778 tempest-ServerDiskConfigTestJSON-1142390778-project] Error defining a guest with XML: <domain type="kvm"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <uuid>97ddda4b-1b4b-4c2d-a77f-087faeae06ae</uuid> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <name>instance-00000021</name> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <memory>131072</memory> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <vcpu>1</vcpu> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <metadata> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:package version="22.1.0"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:name>tempest-ServerDiskConfigTestJSON-server- 1335737010</nova:name> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:creationTime>2021-01-28 00:11:24</nova:creationTime> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:flavor name="m1.nano"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:memory>128</nova:memory> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:disk>1</nova:disk> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:swap>0</nova:swap> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:ephemeral>0</nova:ephemeral> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:vcpus>1</nova:vcpus> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </nova:flavor> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:owner> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:user uuid="89b5ea0dad3348d6be9a94d329e41901">tempest-ServerDiskConfigTestJSON-1142390778-project</nova:user> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:project uuid="37329f48690d4985af6f125e7bd9d529">tempest-ServerDiskConfigTestJSON-1142390778</nova:project> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </nova:owner> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <nova:root type="image" uuid="74cc0df8-6b07-4b4e-a1db- 39777ecf457b"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </nova:instance> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </metadata> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <os> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <type machine="pseries-3.1">hvm</type> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <boot dev="hd"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </os> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <features> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <acpi/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <apic/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </features> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <cputune> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <shares>1024</shares> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </cputune> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <clock offset="utc"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <timer name="pit" tickpolicy="delay"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <timer name="rtc" tickpolicy="catchup"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </clock> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <cpu match="exact"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <topology sockets="1" cores="1" threads="1"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </cpu> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <devices> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <disk type="file" device="disk"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <driver name="qemu" type="qcow2" cache="none"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <source file="/opt/stack/data/nova/instances/97ddda4b- 1b4b-4c2d-a77f-087faeae06ae/disk"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <target dev="vda" bus="virtio"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </disk> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <interface type="bridge"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <mac address="fa:16:3e:30:92:85"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <model type="virtio"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <source bridge="br-int"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <mtu size="1450"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <target dev="tap68762b55-ad"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <virtualport type="openvswitch"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <parameters interfaceid="68762b55-adb4-48c6-8f4c- b5025b2787cd"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </virtualport> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </interface> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <serial type="pty"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <log file="/opt/stack/data/nova/instances/97ddda4b- 1b4b-4c2d-a77f-087faeae06ae/console.log" append="off"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </serial> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <graphics type="vnc" autoport="yes" listen="0.0.0.0"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <video> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <model type="vga"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </video> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <rng model="virtio"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <backend model="random">/dev/urandom</backend> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </rng> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <controller type="usb" index="0" model="none"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <memballoon model="virtio"> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: <stats period="10"/> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </memballoon> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </devices> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: </domain> Jan 28 00:11:24.743424 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: : libvirt.libvirtError: unsupported configuration: USB is disabled for this domain, but USB devices are present in the domain XML Jan 28 00:11:24.746856 zuulv3-devstack-focal-test-ppc64-0000031478 nova-compute[192435]: ERROR nova.virt.libvirt.driver [None req-ffe2d074-e828-4851- a8da-3d0aecdb5873 tempest-ServerDiskConfigTestJSON-1142390778 tempest-ServerDiskConfigTestJSON-1142390778-project] [instance: 97ddda4b-1b4b-4c2d-a77f- 087faeae06ae] Failed to start libvirt guest: libvirt.libvirtError: unsupported configuration: USB is disabled for this domain, but USB devices are present in the domain XML so this is proably a result of https://github.com/openstack/nova/commit/c34a17db6f1ed4469d963a4e64265946029... stephen can you take a look <controller type="usb" index="0" model="none"/> is there but im not seeing anything use the usb device im guessing we need to disable one that is auto added by libvirt on ppc aditi can you provide an xml form a vm booted on ppc normally before this change. specifcally the fully populated xml retrived form libvirt with virsh dumpxml so we can se what it adds
Thanks, Aditi Dukle
----- Original message ----- From: aditi Dukle/India/Contr/IBM To: lyarwood@redhat.com Cc: Michael J Turek/Poughkeepsie/IBM@IBM, openstack-discuss@lists.openstack.org, Sajauddin Mohammad/India/Contr/IBM@IBM Subject: Re: [EXTERNAL] Re: [nova] unit testing on ppc64le Date: Fri, Jan 22, 2021 12:43 PM Hi Lee,
Thanks for getting this fixed. I re-ran the job using the change https://review.opendev.org/c/openstack/nova/+/741545/ and we have 0 failures. Here are the test results - https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/periodic/openst... output.txt
Thanks, Aditi Dukle
----- Original message ----- From: Lee Yarwood <lyarwood@redhat.com> To: aditi Dukle <aditi.Dukle@ibm.com> Cc: Michael J Turek <mjturek@us.ibm.com>, Sajauddin Mohammad <Sajauddin.Mohammad@ibm.com>, openstack-discuss@lists.openstack.org Subject: [EXTERNAL] Re: [nova] unit testing on ppc64le Date: Thu, Jan 21, 2021 6:42 PM On 21-01-21 09:22:44, Lee Yarwood wrote:
On Tue, 19 Jan 2021 at 20:28, aditi Dukle <aditi.Dukle@ibm.com> wrote:
Hi Mike,
I have started nova unit test jobs in the periodic pipeline(runs everyday at UTC- 12) for each openstack branch as follows: periodic: jobs: - openstack-tox-py27: branches: - stable/ocata - stable/pike - stable/queens - stable/rocky - stable/stein - stable/train - openstack-tox-py36: branches: - stable/train - stable/ussuri - stable/victoria - openstack-tox-py37: branches: - stable/train - stable/ussuri - openstack-tox-py38: branches: - stable/victoria - openstack-tox-py39: branches: - master
I have observed a few failures in the unit test cases mostly all related to volume drivers. Please have a look at the 14 test cases that are failing in openstack-tox-py39 job( https://oplab9.parqtec.unicamp.br/pub/ppc64el/openstack/nova/periodic/openst... 38c70ae/job-output.txt ). Most of the 14 failures report these errors: InvalidConnectorProtocol: Invalid InitiatorConnector protocol specified NVME InvalidConnectorProtocol: Invalid InitiatorConnector protocol specified SCALEIO
I would need some help in understanding if these connectors(NVME,SCALEIO) are supported on ppc64.
Right os-brick doesn't support these connectors on ppc64 but the real issue here is with the unit tests that don't mock out calls os-brick, an out of tree lib. I've filed a bug for this below:
LibvirtNVMEVolumeDriverTestCase and LibvirtScaleIOVolumeDriverTestCase unit tests fail on ppc64 https://bugs.launchpad.net/nova/+bug/1912608
I'll push a fix for this shortly.
libvirt: Stop NVMe and ScaleIO unit tests from calling os-brick https://review.opendev.org/c/openstack/nova/+/771806/
In addition to these failures I also see stable rescue tests failing, these should be fixed by the following change:
libvirt: Mock get_arch during some stable rescue unit tests https://review.opendev.org/c/openstack/nova/+/769916/
I also noticed some additional failures caused by the way in which the libvirt virt driver loads its volume drivers at startup that also attempt to load the underlying os-brick connector. As above this results in volume drivers failing to load and being dropped.
We've actually had a long standing TODO in the driver to move to loading these drivers on-demand so I've proposed the following:
libvirt: Load and cache volume drivers on-demand https://review.opendev.org/c/openstack/nova/+/741545/
Can you rerun your tests using the above change and I'll try to address any additional failures.
Finally, in an effort to root out any further issues caused by tests looking up the arch of the test host I've pushed the following change to poison nova.objects.fields.Architecture.from_host:
tests: Poison nova.objects.fields.Architecture.from_host https://review.opendev.org/c/openstack/nova/+/769920
There's a huge amount of fallout from this that I'll try to address in the coming weeks ahead of M3.
Hope this helps!
Lee
----- Original message ----- From: aditi Dukle/India/Contr/IBM To: Michael J Turek/Poughkeepsie/IBM@IBM Cc: Sajauddin Mohammad/India/Contr/IBM@IBM, openstack-discuss@lists.openstack.org Subject: Re: [EXTERNAL] [nova] unit testing on ppc64le Date: Tue, Jan 12, 2021 5:41 PM
Hi Mike,
I have created these unit test jobs - openstack-tox-py27, openstack-tox-py35, openstack-tox-py36, openstack-tox-py37, openstack-tox-py38, openstack-tox-py39 by referring to the upstream CI( https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/j... ) and these jobs are triggered for every patchset in the Openstack CI.
I checked the code for old CI for Power, we didn't have any unit test jobs that were run for every patchset for nova. We had one "nova- python27" job that was run in a periodic pipeline. So, I wanted to know if we need to run the unit test jobs on ppc for every patchset for nova? and If yes, should these be reporting to the Openstack community?
Thanks, Aditi Dukle
----- Original message ----- From: Michael J Turek/Poughkeepsie/IBM To: balazs.gibizer@est.tech, aditi Dukle/India/Contr/IBM@IBM, Sajauddin Mohammad/India/Contr/IBM@IBM Cc: openstack-discuss@lists.openstack.org Subject: Re: [EXTERNAL] [nova] unit testing on ppc64le Date: Sat, Jan 9, 2021 12:52 AM
Thanks for the heads up,
We should have the capacity to add them. At one point I think we ran unit tests for nova but the job may have been culled in the move to zuul v3. I've CC'd the maintainers of the CI, Aditi Dukle and Sajauddin Mohammad.
Aditi and Sajauddin, could we add a job to pkvmci to run unit tests for nova?
Michael Turek Software Engineer Power Cloud Department 1 845 433 1290 Office mjturek@us.ibm.com He/Him/His
IBM
----- Original message ----- From: Balazs Gibizer <balazs.gibizer@est.tech> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Cc: mjturek@us.ibm.com Subject: [EXTERNAL] [nova] unit testing on ppc64le Date: Fri, Jan 8, 2021 7:59 AM
Hi,
We have a bugreport[1] showing that our unit tests are not passing on ppc. In the upstream CI we don't have test capability to run our tests on ppc. But we have the IBM Power KVM CI[2] that runs integration tests on ppc. I'm wondering if IBM could extend the CI to run nova unit and functional tests too. I've added Michael Turek (mjturek@us.ibm.com) to CC. Michael is listed as the contact person for the CI.
Cheers, gibi
[1]https://bugs.launchpad.net/nova/+bug/1909972 [2]https://wiki.openstack.org/wiki/ThirdPartySystems/IBMPowerKVMCI
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76