Open Stack

On Sat, Mar 11, 2017 at 10:45 PM Matt Riedemann <mriedemos at gmail.com> wrote:

On 3/10/2017 3:02 PM, Andrea Frittoli wrote:
>
> We had a couple of sessions related to this topic at the PTG [0][1].
>
> We agreed that we want to still maintain integration tests only in
> Tempest, which means that API micro versions that have no integration
> impact can be tested via functional tests.

To be clear, "integration" here means tests that span operations across
multiple services, correct? Like a compute test that first creates a
port in the networking service and a volume in the block storage service
and then uses those to create a server and maybe take a snapshot of it
which is then verified was uploaded to the image service.

Yes, indeed.



The non-integration things are self-contained in a single service, like
if all you need to do is create an aggregate, show it's details and
validate the response, at a particular microversion, we can just do that
in nova functional tests, and it's not necessary in Tempest.

It might be worth having a definition of this policy in the Tempest docs
so when people ask this question again you can just point at the docs.


I agree, we need to go a better job at documenting this.
I just want to avoid having a black and white rule that would not work.

To me tests that should be in Tempest are tests that make sense to run
against any change in any repo, and the reasons for that can be many:
- the test involves multiple services (more that "it uses a token though")
- the test covers a feature which is key for interoperability
- the test must be reliable and not take too long

Based on this I don't it's possible to completely avoid the discussion about
which test should in Tempest and which not, it's something to be considered
on a test by test basis, based on a set of guidelines.

We have an initial definition of scope in [0], but it's probably worth to
elaborate
more on it. I'll put up a patch so that the discussion on it can continue in
gerrit.

[0]
https://docs.openstack.org/developer/tempest/test-removal.html#tempest-scope


>
> In terms of which versions we test in the gate, for nova we always run
> with min_microversion = None and max_microversion = latest, which means
> that all tests will be executed.
> Since micro versions are incremental, and each micro version usually
> involves no or one test on Tempest side, I think it will be a while
> before this becomes an issue for the common gate.

We test max_microversion=latest only on master. On the devstack stable
branch we cap the max_microversion, e.g.:

Thanks, good point.



https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339

--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170313/4132c5ac/attachment-0001.html>