[openstack-dev] [nova] unit tests result in false negatives on system z platform CI

Markus Zoeller mzoeller at de.ibm.com
Wed Apr 1 09:54:16 UTC 2015

During the Kilo development cycle the "KVM/libvirt on system z" platform
made some effort to be supported by the libvirt driver [1]. 

Our first tests in a prototype platform CI showed some false negatives
because some unit tests don't seem to be fully platform independent.
For example the result of test:

    test_get_guest_config_without_qga_through_image_meta [0.016369s] 
    ... FAILED

    Captured traceback:
        Traceback (most recent call last):
          File "nova/tests/unit/virt/libvirt/test_driver.py", line 3112, 
          in test_get_guest_config_without_qga_through_image_meta
            raise mismatch_error
        '<nova.virt.libvirt.config.LibvirtConfigGuestConsole object>' 
        is not an instance of LibvirtConfigGuestSerial

This mismatch makes fully sense if x86 is assumed as default underlying
platform for unit test execution. The root cause (in this case) is that
the call of "libvirt_utils.get_arch()" is not mocked and actually speaks
to the underlying platform. On our system z CI this call returns "s390x"
which hits platform switches in the code (a search for "arch.S390X" will
show you all this platform specific code). 

A first test
A change of this specific test could look like this:

    # git diff nova/tests/unit/virt/libvirt/test_driver.py
    diff --git a/nova/tests/unit/virt/libvirt/test_driver.py 
    index 5fbe5e1..ebcc9ed 100644
    --- a/nova/tests/unit/virt/libvirt/test_driver.py
    +++ b/nova/tests/unit/virt/libvirt/test_driver.py
    @@ -3091,7 +3091,9 @@ class LibvirtConnTestCase(test.NoDBTestCase):

    -    def test_get_guest_config_without_qga_through_image_meta(self):
    +    @mock.patch.object(libvirt_driver.libvirt_utils, 'get_arch',
    +                       return_value=arch.X86_64)
    +    def test_get_guest_config_without_qga_through_image_meta(self, 
             self.flags(virt_type='kvm', group='libvirt')

             drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True)

Open questions
We currently discovered around 30 test cases (mostly the class 
"LibvirtConnTestCase") which could be treated that way, which seems to 
be cumbersome from my point of view. I'm looking for a way to express 
the assumption that x86 should be the default platform in the unit tests
and prevent calls to the underlying system. This has to be rewritable if
platform specific code like in [2] has to be tested. 

I'd like to discuss how that could be achieved in a maintainable way.

[1] https://blueprints.launchpad.net/nova/+spec/libvirt-kvm-systemz
[2] test_driver.py; test_get_guest_config_with_type_kvm_on_s390;

More information about the OpenStack-dev mailing list