Open Stack

On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote:
> ----- Original Message -----
> > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote:
> > > (cross-posting)
> > 
> > > Instead of running the Pingtest, we would execute a Tempest
> > > Scenario
> > > that boot an instance from volume (like Pingstest is already
> > > doing)
> > > and see how it goes (in term of coverage and runtime).
> > > I volunteer to kick-off the work with someone more expert than I
> > > am
> > > with quickstart (Arx maybe?).
> > > 
> > > Another iteration could be to start building an easy interface to
> > > select which Tempest tests we want a TripleO CI job to run and
> > > plug
> > > it
> > > to our CI tooling (tripleo-quickstart I presume).
> > 
> > Running a subset of Tempest tests isn't the same thing as designing
> > (and owning) your own test suite that targets the things that mean
> > the
> > most to our community (namely speed and coverage). Even giving up
> > 5-10
> > minutes of runtime...just to be able to run Tempest isn't something
> > that some of us would be willing to do.
> 
> As I mentioned, you can do it with Tempest (the library). You can
> have your own test suite that does exactly what you are asking
> (namely, a set of scenario tests based on Heat which targets the
> TripleO use case) in a Tempest plugin and there is no absolute reason
> that those tests should add 5-10 minutes of runtime compared to
> pingtest. 
> 
> It/they would be exactly pingtest, only implemented using a different
> library and running with a different runner, with the *exact* same
> run time. 
> 
> Obvious advantages: only one technology used to run tests, so if
> anyone else want to run additional tests, there is no need to
> maintain two code paths; reuse on a big and proven library of test
> and test runner tools.

I like the idea of getting pingtest out of tripleo.sh as more of a
stand alone tool. I would support an effort that re-implemented it...
and using tempest-lib would be totally fine. And as you point out one
could even combine these tests with a more common "Tempest" run that
incorporates the scenarios, etc.

To me the message is clear that we DO NOT want to consume the normal
Tempest scenarios in TripleO upstream CI at this point. Sure there is
overlap there, but the focus of those tests is just plain different...
speed isn't a primary concern there as it is for us so I don't think we
should do it now. And probably not ever unless the CI job time is less
than an hour. Like even if we were able to tune a set of stock Tempest
smoke tests today to our liking unless TripleO proper gates on the
runtime of those not increasing we'd be at risk of breaking our CI
queues as the wall time would potentially get too long. In this regard
this entire thread is poorly named I think in that we are no longer
talking about 'pingtest vs. tempest' but rather the implementation
details of how we reimplement our existing pingtest to better suite the
community.

So ++ for the idea of experimenting with the use of tempest.lib. But
stay away from the idea of using Tempest smoke tests and the like for
TripleO I think ATM.

Its also worth noting there is some risk when maintaining your own in-
tree Tempest tests [1]. If I understood that thread correctly that
breakage wouldn't have occurred if the stable branch tests were gating
Tempest proper... which is a very hard thing to do if we have our own
in-tree stuff. So there is a cost to doing what you suggest here, but
probably one that we'd be willing to accept.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116172.
html

Dan

> 
> Ciao