Open Stack

On Thu, May 4, 2017 at 9:41 AM, Dan Prince <dprince at redhat.com> wrote:
> On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote:
>> ----- Original Message -----
>> > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote:
>> > > (cross-posting)
>> >
>> > > Instead of running the Pingtest, we would execute a Tempest
>> > > Scenario
>> > > that boot an instance from volume (like Pingstest is already
>> > > doing)
>> > > and see how it goes (in term of coverage and runtime).
>> > > I volunteer to kick-off the work with someone more expert than I
>> > > am
>> > > with quickstart (Arx maybe?).
>> > >
>> > > Another iteration could be to start building an easy interface to
>> > > select which Tempest tests we want a TripleO CI job to run and
>> > > plug
>> > > it
>> > > to our CI tooling (tripleo-quickstart I presume).
>> >
>> > Running a subset of Tempest tests isn't the same thing as designing
>> > (and owning) your own test suite that targets the things that mean
>> > the
>> > most to our community (namely speed and coverage). Even giving up
>> > 5-10
>> > minutes of runtime...just to be able to run Tempest isn't something
>> > that some of us would be willing to do.
>>
>> As I mentioned, you can do it with Tempest (the library). You can
>> have your own test suite that does exactly what you are asking
>> (namely, a set of scenario tests based on Heat which targets the
>> TripleO use case) in a Tempest plugin and there is no absolute reason
>> that those tests should add 5-10 minutes of runtime compared to
>> pingtest.
>>
>> It/they would be exactly pingtest, only implemented using a different
>> library and running with a different runner, with the *exact* same
>> run time.
>>
>> Obvious advantages: only one technology used to run tests, so if
>> anyone else want to run additional tests, there is no need to
>> maintain two code paths; reuse on a big and proven library of test
>> and test runner tools.
>
> I like the idea of getting pingtest out of tripleo.sh as more of a
> stand alone tool. I would support an effort that re-implemented it...
> and using tempest-lib would be totally fine. And as you point out one
> could even combine these tests with a more common "Tempest" run that
> incorporates the scenarios, etc.

I don't understand why we would re-implement the pingtest in a tempest plugin.
Could you please tell us what is the technical difference between what
does this scenario :
https://github.com/openstack/tempest/blob/master/tempest/scenario/test_volume_boot_pattern.py

And this pingtest:
https://github.com/openstack/tripleo-heat-templates/blob/master/ci/pingtests/tenantvm_floatingip.yaml

They both create a volume Cinder, snapshot it in Glance and and spawn
a Nova server from the volume.

What one does that the other one doesn't?

> To me the message is clear that we DO NOT want to consume the normal
> Tempest scenarios in TripleO upstream CI at this point. Sure there is
> overlap there, but the focus of those tests is just plain different...

I haven't seen strong pushback in this thread except you.
I'm against overlap in general and this one is pretty obvious. Why
would we maintain a tripleo-specific Tempest scenario while existing
ones would work for us. Please give me a technical reason in what is
not good enough in the existing scenarios.

> speed isn't a primary concern there as it is for us so I don't think we
> should do it now. And probably not ever unless the CI job time is less
> than an hour. Like even if we were able to tune a set of stock Tempest
> smoke tests today to our liking unless TripleO proper gates on the
> runtime of those not increasing we'd be at risk of breaking our CI
> queues as the wall time would potentially get too long. In this regard
> this entire thread is poorly named I think in that we are no longer
> talking about 'pingtest vs. tempest' but rather the implementation
> details of how we reimplement our existing pingtest to better suite the
> community.

What I would like to see if we're going to use Tempest in our gate, is
to run at least one TripleO jobs as voting in Tempest.

Tempest folks: I need your support here. We have been running Puppet
jobs as non-voting and we have seen a quite number of patches that
broke us because folks were ignore the jobs. If we switch TripleO to
use more Tempest, being in your gate might be required. We'll run the
fastest and  more stable job that we have to make sure the impact for
you is minimal.

> So ++ for the idea of experimenting with the use of tempest.lib. But
> stay away from the idea of using Tempest smoke tests and the like for
> TripleO I think ATM.
>
> Its also worth noting there is some risk when maintaining your own in-
> tree Tempest tests [1]. If I understood that thread correctly that
> breakage wouldn't have occurred if the stable branch tests were gating
> Tempest proper... which is a very hard thing to do if we have our own
> in-tree stuff. So there is a cost to doing what you suggest here, but
> probably one that we'd be willing to accept.

I'm not sure we have the resources to write and maintain our own
in-tree tempest plugin, tbh.

> [1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116172.
> html
>
> Dan
>

It sounds like we haven't reached full consensus yet but at least we
agreed on experimenting with Tempest.

While we make progress on the discussion, I'll start working with some
folks as experimental jobs, so we can make progress in the meantime.

Thanks,
-- 
Emilien Macchi