Open Stack

On Mon, Mar 13, 2017 at 6:52 AM Ghanshyam Mann <ghanshyammann at gmail.com>
wrote:

> On Sun, Mar 12, 2017 at 7:40 AM, Matt Riedemann <mriedemos at gmail.com>
> wrote:
>
> On 3/10/2017 3:02 PM, Andrea Frittoli wrote:
>
>
> We had a couple of sessions related to this topic at the PTG [0][1].
>
> We agreed that we want to still maintain integration tests only in
> Tempest, which means that API micro versions that have no integration
> impact can be tested via functional tests.
>
>
> To be clear, "integration" here means tests that span operations across
> multiple services, correct? Like a compute test that first creates a port
> in the networking service and a volume in the block storage service and
> then uses those to create a server and maybe take a snapshot of it which is
> then verified was uploaded to the image service.
>
> The non-integration things are self-contained in a single service, like if
> all you need to do is create an aggregate, show it's details and validate
> the response, at a particular microversion, we can just do that in nova
> functional tests, and it's not necessary in Tempest.
>
> It might be worth having a definition of this policy in the Tempest docs
> so when people ask this question again you can just point at the docs.
>
>
> ​Yes, that will be helpful to have those agreement in doc​. I am adding
> those in existing microversion testing doc [0] and later will be
> refactoring to give them better and more visible shape.
>
> Tempest maintain the info about what microversion tests are is implemented
> on Tempest side which will be helpful for projects to check the coverage
> [1]. Cinder one was missed which I added in [2].
>
> ​As summary:
> Tempest Scope:
> - Only Integration tests for microversion is allowed in Tempest
>
If a microversion is important for interoperability it may end up in
Tempest even if doesn't involve too much integration


> - Exception for non integration tests to fill the schema gap if exists.
>
Yeah when we fill schema gaps we need at least one test for the top version
to verify that the latest implemented schema is valid


>
> Project Scope:
> - Remaining tests coverage of microversion​ should be on Projects side.
>       - Project can cover those as functional tests etc.
>       - Nova currently have at least one functional tests per each
> microversion and plan to extend coverage like [3]
>       - IMO, only running tests with 'latest' microversion is not enough,
> each microversion should be tested with positive and negative testing
> (w.r.t at least immediate previous microversion).
>
>
>
>
> In terms of which versions we test in the gate, for nova we always run
> with min_microversion = None and max_microversion = latest, which means
> that all tests will be executed.
> Since micro versions are incremental, and each micro version usually
> involves no or one test on Tempest side, I think it will be a while
> before this becomes an issue for the common gate.
>
>
> We test max_microversion=latest only on master. On the devstack stable
> branch we cap the max_microversion, e.g.:
>
>
> https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339
>
> --
>
> Thanks,
>
> Matt
>
>
>
>
> ..0 https://review.openstack.org/#/c/444727/1
> ..1
> https://docs.openstack.org/developer/tempest/microversion_testing.html#microversion-tests-implemented-in-tempest
>
> ..2 https://review.openstack.org/#/c/444711/ ​
> ​..3
> https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests
>  ​
>
> ​​Thanks
> gmann​
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170313/632449c7/attachment.html>