Open Stack

Hi all,

The Neutron community has recently started contributing Tempest
scenario tests in the Neutron tree and I'd like to discuss a general
issue we're hitting. Mainly, we're talking about required hardware for
some features and/or VM images that support more features, which makes
it hard (or impossible) to test certain features at the gate
end-to-end. Specifically, tests that require SR-IOV or OVS-DPDK, and
tests relating to VLAN Aware VMs.

For this reason, we would like to bring up the idea of introducing in
a new concept to the "testing arena" - skip-only tests. These will be
tests that will be merged upstream and skipped unless some
pre-conditions are met (example: "@skip_unless(has_sriov=True)"). The
key idea here though, is that we don't expect upstream gates to add
support for these pre-conditions any time soon. Instead, the tests
will be merged upstream and used on the various downstream gates most
of us have (with specific hardware/deployment that match our
respective interests).

Another way to look at it is: "I want to contribute tests that require
specific hardware/software/deployments, and I want it to be robust and
shared amongst the community as per the Open Source way, even though
it won't be actually run on the upstream gates". Obviously, we would
continue to investigate ways to "unskip" these kind of tests where
possible.

Examples where this would be extremely helpful include (but aren't limited to):

1. SR-IOV - Sure, the Mellanox CI currently tests SR-IOV in VF mode,
however this isn't enough; SR-IOV in PF mode isn't checked and
additional interesting use-cases (such as QoS + SR-IOV) aren't tested
in Mellanox' CI (let alone any other upstream gate). Also, considering
the current QoS test - in order to test it specifically with SR-IOV,
one must be able to create a port with vnic_type='direct'.

2. VLAN Aware VMs - while tested (or planned to be tested) upstream in
the unit/functional/api/fullstack levels, it isn't tested (nor is it
planned to be, afaik) in a tempest scenario. The reason is simple: VMs
require 802.1Q [1] in order to support VLAN Aware VMs, and it isn't
currently in the Cirros image. While we're currently blocking on these
tests until the Cirros images are updated, we could easily test the
feature using our respective images downstream. This is also a good
example where eventually these tests can be un-skipped when the
upstream Cirros images are updated.

3. SCTP tests for security groups - also blocked because Cirros
doesn't carry a netcat which supports this.

...And so on and so forth.


Here's a brief QA section that should include some common questions/complaints:

Q: The VLAN Aware VMs issue can be solved by adding Cirros support for
802.1Q - why should we in the mean time merge tests that need to be
maintained?
A: No one has guarantees when the new Cirros will be released and that
it will work with all the other gate jobs and tests. Though we have
already started pushing for a new version with support, we have
(downstream-wise) other VMs that do carry the support required - we
are just missing the tests.

Q: So write the tests downstream - why should I care?
A: We want to collaborate on the tests upstream so we (and you) can
gain the standard benefits of improved and better maintained code.
This will also (hopefully) help removing duplication of efforts in the
testing scene.

Q: So how will we know if a downstream test fails?
A: This is a currently open question. One suggestion we had was that
any time these skip-only tests are explicitly changed, the author has
to show logs of a successful run. While we don't see how this can be
relevant to certain trivial project-wide changes (and such changes can
be exempt from such proofs), explicit modification of tests, logic or
infra that involve these tests will need to prove that they actually
work downstream.


We are now open for ideas about this idea.

[1]: https://en.wikipedia.org/wiki/IEEE_802.1Q

-- 
John Schwarz,
Senior Software Engineer,
Red Hat.