Open Stack

On Sun, Mar 12, 2017 at 7:40 AM, Matt Riedemann <mriedemos at gmail.com> wrote:

> On 3/10/2017 3:02 PM, Andrea Frittoli wrote:
>
>>
>> We had a couple of sessions related to this topic at the PTG [0][1].
>>
>> We agreed that we want to still maintain integration tests only in
>> Tempest, which means that API micro versions that have no integration
>> impact can be tested via functional tests.
>>
>
> To be clear, "integration" here means tests that span operations across
> multiple services, correct? Like a compute test that first creates a port
> in the networking service and a volume in the block storage service and
> then uses those to create a server and maybe take a snapshot of it which is
> then verified was uploaded to the image service.
>
> The non-integration things are self-contained in a single service, like if
> all you need to do is create an aggregate, show it's details and validate
> the response, at a particular microversion, we can just do that in nova
> functional tests, and it's not necessary in Tempest.
>
> It might be worth having a definition of this policy in the Tempest docs
> so when people ask this question again you can just point at the docs.
>
>
​Yes, that will be helpful to have those agreement in doc​. I am adding
those in existing microversion testing doc [0] and later will be
refactoring to give them better and more visible shape.

Tempest maintain the info about what microversion tests are is implemented
on Tempest side which will be helpful for projects to check the coverage
[1]. Cinder one was missed which I added in [2].

​As summary:
Tempest Scope:
- Only Integration tests for microversion is allowed in Tempest
- Exception for non integration tests to fill the schema gap if exists.

Project Scope:
- Remaining tests coverage of microversion​ should be on Projects side.
      - Project can cover those as functional tests etc.
      - Nova currently have at least one functional tests per each
microversion and plan to extend coverage like [3]
      - IMO, only running tests with 'latest' microversion is not enough,
each microversion should be tested with positive and negative testing
(w.r.t at least immediate previous microversion).



>
>> In terms of which versions we test in the gate, for nova we always run
>> with min_microversion = None and max_microversion = latest, which means
>> that all tests will be executed.
>> Since micro versions are incremental, and each micro version usually
>> involves no or one test on Tempest side, I think it will be a while
>> before this becomes an issue for the common gate.
>>
>
> We test max_microversion=latest only on master. On the devstack stable
> branch we cap the max_microversion, e.g.:
>
> https://github.com/openstack-dev/devstack/blob/stable/newton
> /lib/tempest#L339
>
> --
>
> Thanks,
>
> Matt
>
>


..0 https://review.openstack.org/#/c/444727/1
..1
https://docs.openstack.org/developer/tempest/microversion_testing.html#microversion-tests-implemented-in-tempest

..2 https://review.openstack.org/#/c/444711/ ​
​..3
https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests
 ​

​​Thanks
gmann​



> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170313/2f6139f3/attachment.html>