[openstack-dev] [QA] [Ironic] [Inspector] Where should integration tests for non-core projects live now? (Was: Toward 2.0.0 release)
dtantsur at redhat.com
Wed Jun 17 06:52:26 UTC 2015
On 06/17/2015 06:54 AM, Ken'ichi Ohmichi wrote:
> 2015-06-17 12:38 GMT+09:00 Yuiko Takada <yuikotakada0313 at gmail.com>:
>> Then, as you and Matt and Dimitry talked about this on IRC few days ago,
>> We can add Ironic/Ironic-inspector tests into Tempest still, right?
>> So that I've started to implement a test in Tempest,
>> but I'm facing another issue.
>> As you know, Ironic API has microversions, and Ironic-inspector can run
>> with microversion > 1.6.
>> But currently there is no feature testing specific Ironic API microversions
>> on Tempest, right?
>> So that, we have to think about some solutions.
>> (1) Make testing specific Ironic API microversions on Tempest possible
>> adam_g is posting this patch set.
>> (2)Using tempest_lib instead of adding tests into Tempest
>> Is tempest_lib available already?
>> Or do we need to wait for something will be merged?
> I guess the above question seems multiple factors are mixed.
> You want to test ironic-inspector behaviors by
> * using ironic-inspector REST APIs directly without Ironic
> * using Ironic REST APIs which need newer microversion
Hi, thanks for clarifying, let me jump in :)
The former is more or less covered by functional testing, so I'd like us
to concentrate on the latter, and run it voting on inspector repo and
non-voting on Ironic for the time being.
> For the first test, you can implement without considering microversion.
> The test just calls ironic-inspector REST APIs directly and checks its behavior.
> You can implement the test on Tempest/ironic-inspector repository.
> Current tempest-lib seems enough to implement tests in
> ironic-inspector repository as features, but it is better to wait for
> Tempest's external interface spec approval.
> It is trying to define directory structure of Tempest-like tests on
> each project repository and Tempest will discover tests based on the
> directory structure and run them.
> So if implementing tests on ironic-inspector repository before the
> spec approval, you will need to change the directory structure again
> in the future.
This "wait" part bothers me to some extend, because gate absence badly
affects us for some time, but fine. Thanks for heads up anyway.
> For the second test, microversions support is necessary on Tempest
> side and adam_g's patch seems good for implementing it.
> My main concern of microversions tests is how to run multiple
> microversions on the gate.
> We have discussed that in Nova design session of Vancouver summit and
> the conclusion is
> * Minimum microversion
> * Maximum microversion
> * Interesting microversions
> as the gate test.
Facepalm. That's what I was talking about (and what we actually ended up
in Ironic with): we're introducing a ton of non-tested (and thus
presumably broken) microversions, because it's cool to do. Ok, that's
another thread :)
> IMO "Interesting microversions" would be the last microversions of
> each release(Kilo, Liberty, ..) I feel.
With Ironic intermediate release, it will be more, I estimate it to be
5-6 per year, but of course I can't tell for sure.
> I have qa-spec for testing microversions on the gate, but that is
> not complete yet.
> That will affect how to specify/run microversions tests on Tempest.
> So now I'm not sure yet the way to specify microversion on current
> adam_g's patch is the best.
> So my recommendation/hope is that we concentrate on Tempest's external
> interface spec and make it better together, then we can implement
> Tempest-like tests on each repository after that.
> As the next step, we will test microversions on the same way between
> projects based on conclusion of the spec.
What I'd prefer us to start with is a gate test, which just sets
devstack with our plugin and runs a shell script testing a couple of
basic things. This will be a HUGE leap forward for inspector, compared
to only limited functional testing we have now.
So maybe we should start with it, and keep an eye on the tempest-lib
>> (3)Make Ironic-inspector available even if microversion < 1.6
>> Dmitry is posting this patch set.
>> # I don't mean asking you to review this, don't worry :p
> I've reviewed it already :)
> Ken Ohmichi
> : https://review.openstack.org/#/c/184992/
> : https://review.openstack.org/#/c/169126/
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
More information about the OpenStack-dev