Open Stack

On Thursday, 4 May 2017 15:41:04 CEST Dan Prince wrote:
> On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote:
> > ----- Original Message -----
> > 

> > > 
> > > Running a subset of Tempest tests isn't the same thing as designing
> > > (and owning) your own test suite that targets the things that mean
> > > the
> > > most to our community (namely speed and coverage). Even giving up
> > > 5-10
> > > minutes of runtime...just to be able to run Tempest isn't something
> > > that some of us would be willing to do.
> > 
> > As I mentioned, you can do it with Tempest (the library). You can
> > have your own test suite that does exactly what you are asking
> > (namely, a set of scenario tests based on Heat which targets the
> > TripleO use case) in a Tempest plugin and there is no absolute reason
> > that those tests should add 5-10 minutes of runtime compared to
> > pingtest. 
> > 
> > It/they would be exactly pingtest, only implemented using a different
> > library and running with a different runner, with the *exact* same
> > run time. 
> > 
> > Obvious advantages: only one technology used to run tests, so if
> > anyone else want to run additional tests, there is no need to
> > maintain two code paths; reuse on a big and proven library of test
> > and test runner tools.
> 
> I like the idea of getting pingtest out of tripleo.sh as more of a
> stand alone tool. I would support an effort that re-implemented it...
> and using tempest-lib would be totally fine. And as you point out one
> could even combine these tests with a more common "Tempest" run that
> incorporates the scenarios, etc.

That's the idea, yes: anyone would be able to consume it easily with the other 
tests; just a regexp away.


> To me the message is clear that we DO NOT want to consume the normal
> Tempest scenarios in TripleO upstream CI at this point. Sure there is
> overlap there, but the focus of those tests is just plain different...
> speed isn't a primary concern there as it is for us so I don't think we
> should do it now. And probably not ever unless the CI job time is less
> than an hour. Like even if we were able to tune a set of stock Tempest
> smoke tests today to our liking unless TripleO proper gates on the
> runtime of those not increasing we'd be at risk of breaking our CI
> queues as the wall time would potentially get too long. In this regard
> this entire thread is poorly named I think in that we are no longer
> talking about 'pingtest vs. tempest' but rather the implementation
> details of how we reimplement our existing pingtest to better suite the
> community.
> 
> So ++ for the idea of experimenting with the use of tempest.lib. But
> stay away from the idea of using Tempest smoke tests and the like for
> TripleO I think ATM.

That would be good!

> 
> Its also worth noting there is some risk when maintaining your own in-
> tree Tempest tests [1]. If I understood that thread correctly that
> breakage wouldn't have occurred if the stable branch tests were gating
> Tempest proper... which is a very hard thing to do if we have our own
> in-tree stuff. So there is a cost to doing what you suggest here, but
> probably one that we'd be willing to accept.

About this, the idea is not to put the Tempest plugin in-tree, but keep it in 
a separate repository (and keep it branchless like Tempest; as you test the 
API. We did this for Sahara tests since liberty with good results. 
Moreover, there is a proposed global goal for Queen to decouple the Tempest 
plugin in separate repositories:
https://review.openstack.org/#/c/369749/

-- 
Luigi