[openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)

Dmitry Tantsur dtantsur at redhat.com
Wed Jun 17 06:52:26 UTC 2015

On 06/17/2015 06:54 AM, Ken'ichi Ohmichi wrote:
> 2015-06-17 12:38 GMT+09:00 Yuiko Takada <yuikotakada0313 at gmail.com>:
>> Then, as you and Matt and Dimitry talked about this on IRC few days ago,
>> We can add Ironic/Ironic-inspector tests into Tempest still, right?
>> So that I've started to implement a test in Tempest,
>> but I'm facing another issue.
>> As you know, Ironic API has microversions, and Ironic-inspector can run
>> with microversion > 1.6.
>> But currently there is no feature testing specific Ironic API microversions
>> on Tempest, right?
>> So that, we have to think about some solutions.
>> (1) Make testing specific Ironic API microversions on Tempest possible
>> adam_g is posting this patch set.
>> https://review.openstack.org/166386
>> (2)Using tempest_lib instead of adding tests into Tempest
>> Is tempest_lib available already?
>> Or do we need to wait for something will be merged?
> I guess the above question seems multiple factors are mixed.
> You want to test ironic-inspector behaviors by
>    * using ironic-inspector REST APIs directly without Ironic
>    * using Ironic REST APIs which need newer microversion
> right?

Hi, thanks for clarifying, let me jump in :)

The former is more or less covered by functional testing, so I'd like us 
to concentrate on the latter, and run it voting on inspector repo and 
non-voting on Ironic for the time being.

> For the first test, you can implement without considering microversion.
> The test just calls ironic-inspector REST APIs directly and checks its behavior.
> You can implement the test on Tempest/ironic-inspector repository.
> Current tempest-lib seems enough to implement tests in
> ironic-inspector repository as features, but it is better to wait for
> Tempest's external interface spec[1] approval.
> It is trying to define directory structure of Tempest-like tests on
> each project repository and Tempest will discover tests based on the
> directory structure and run them.
> So if implementing tests on ironic-inspector repository before the
> spec approval, you will need to change the directory structure again
> in the future.

This "wait" part bothers me to some extend, because gate absence badly 
affects us for some time, but fine. Thanks for heads up anyway.

> For the second test, microversions support is necessary on Tempest
> side and adam_g's patch seems good for implementing it.
> My main concern of microversions tests is how to run multiple
> microversions on the gate.
> We have discussed that in Nova design session of Vancouver summit and
> the conclusion is
>   * Minimum microversion
>   * Maximum microversion
>   * Interesting microversions
> as the gate test.

Facepalm. That's what I was talking about (and what we actually ended up 
in Ironic with): we're introducing a ton of non-tested (and thus 
presumably broken) microversions, because it's cool to do. Ok, that's 
another thread :)

> IMO "Interesting microversions" would be the last microversions of
> each release(Kilo, Liberty, ..) I feel.

With Ironic intermediate release, it will be more, I estimate it to be 
5-6 per year, but of course I can't tell for sure.

> I have qa-spec[2] for testing microversions on the gate, but that is
> not complete yet.
> That will affect how to specify/run microversions tests on Tempest.
> So now I'm not sure yet the way to specify microversion on current
> adam_g's patch is the best.
> So my recommendation/hope is that we concentrate on Tempest's external
> interface spec[1] and make it better together, then we can implement
> Tempest-like tests on each repository after that.
> As the next step, we will test microversions on the same way between
> projects based on conclusion of the spec[2].

What I'd prefer us to start with is a gate test, which just sets 
devstack with our plugin and runs a shell script testing a couple of 
basic things. This will be a HUGE leap forward for inspector, compared 
to only limited functional testing we have now.

So maybe we should start with it, and keep an eye on the tempest-lib 
stuff, wdyt?

>> (3)Make Ironic-inspector available even if microversion < 1.6
>> Dmitry is posting this patch set.
>> https://review.openstack.org/192196
>> # I don't mean asking you to review this, don't worry :p
> I've reviewed it already :)
> Thanks
> Ken Ohmichi
> ---
> [1]: https://review.openstack.org/#/c/184992/
> [2]: https://review.openstack.org/#/c/169126/
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

More information about the OpenStack-dev mailing list