Open Stack

On Thu, May 4, 2017 at 7:13 PM Emilien Macchi <emilien at redhat.com> wrote:

> On Thu, May 4, 2017 at 9:41 AM, Dan Prince <dprince at redhat.com> wrote:
> > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote:
> >> ----- Original Message -----
> >> > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote:
> >> > > (cross-posting)
> >> >
> >> > > Instead of running the Pingtest, we would execute a Tempest
> >> > > Scenario
> >> > > that boot an instance from volume (like Pingstest is already
> >> > > doing)
> >> > > and see how it goes (in term of coverage and runtime).
> >> > > I volunteer to kick-off the work with someone more expert than I
> >> > > am
> >> > > with quickstart (Arx maybe?).
> >> > >
> >> > > Another iteration could be to start building an easy interface to
> >> > > select which Tempest tests we want a TripleO CI job to run and
> >> > > plug
> >> > > it
> >> > > to our CI tooling (tripleo-quickstart I presume).
> >> >
> >> > Running a subset of Tempest tests isn't the same thing as designing
> >> > (and owning) your own test suite that targets the things that mean
> >> > the
> >> > most to our community (namely speed and coverage). Even giving up
> >> > 5-10
> >> > minutes of runtime...just to be able to run Tempest isn't something
> >> > that some of us would be willing to do.
> >>
> >> As I mentioned, you can do it with Tempest (the library). You can
> >> have your own test suite that does exactly what you are asking
> >> (namely, a set of scenario tests based on Heat which targets the
> >> TripleO use case) in a Tempest plugin and there is no absolute reason
> >> that those tests should add 5-10 minutes of runtime compared to
> >> pingtest.
> >>
> >> It/they would be exactly pingtest, only implemented using a different
> >> library and running with a different runner, with the *exact* same
> >> run time.
> >>
> >> Obvious advantages: only one technology used to run tests, so if
> >> anyone else want to run additional tests, there is no need to
> >> maintain two code paths; reuse on a big and proven library of test
> >> and test runner tools.
> >
> > I like the idea of getting pingtest out of tripleo.sh as more of a
> > stand alone tool. I would support an effort that re-implemented it...
> > and using tempest-lib would be totally fine. And as you point out one
> > could even combine these tests with a more common "Tempest" run that
> > incorporates the scenarios, etc.
>
> I don't understand why we would re-implement the pingtest in a tempest
> plugin.
> Could you please tell us what is the technical difference between what
> does this scenario :
>
> https://github.com/openstack/tempest/blob/master/tempest/scenario/test_volume_boot_pattern.py
>
> And this pingtest:
>
> https://github.com/openstack/tripleo-heat-templates/blob/master/ci/pingtests/tenantvm_floatingip.yaml
>
> They both create a volume Cinder, snapshot it in Glance and and spawn
> a Nova server from the volume.
>
> What one does that the other one doesn't?
>
> > To me the message is clear that we DO NOT want to consume the normal
> > Tempest scenarios in TripleO upstream CI at this point. Sure there is
> > overlap there, but the focus of those tests is just plain different...
>
> I haven't seen strong pushback in this thread except you.
> I'm against overlap in general and this one is pretty obvious. Why
> would we maintain a tripleo-specific Tempest scenario while existing
> ones would work for us. Please give me a technical reason in what is
> not good enough in the existing scenarios.
>
> > speed isn't a primary concern there as it is for us so I don't think we
> > should do it now. And probably not ever unless the CI job time is less
> > than an hour. Like even if we were able to tune a set of stock Tempest
> > smoke tests today to our liking unless TripleO proper gates on the
> > runtime of those not increasing we'd be at risk of breaking our CI
> > queues as the wall time would potentially get too long. In this regard
> > this entire thread is poorly named I think in that we are no longer
> > talking about 'pingtest vs. tempest' but rather the implementation
> > details of how we reimplement our existing pingtest to better suite the
> > community.
>
> What I would like to see if we're going to use Tempest in our gate, is
> to run at least one TripleO jobs as voting in Tempest.
>
> Tempest folks: I need your support here. We have been running Puppet
> jobs as non-voting and we have seen a quite number of patches that
> broke us because folks were ignore the jobs. If we switch TripleO to
> use more Tempest, being in your gate might be required. We'll run the
> fastest and  more stable job that we have to make sure the impact for
> you is minimal.
>
>
For TripleO I guess the risks of failures might be about deploying tempest
tests and their dependencies more than else.

We're increasing our stable API surface constantly, and I believe the number
of breakages have decreased over time - I don't have data to support this
though.

>From a test pov, any cloud deployed by TripleO should be able to run a few
basic scenarios and I would say the likelihood of Tempest breaking that is
rather
low.

We could start with a non-voting job like for puppet and take it from there.


> So ++ for the idea of experimenting with the use of tempest.lib. But
> > stay away from the idea of using Tempest smoke tests and the like for
> > TripleO I think ATM.
> >
> > Its also worth noting there is some risk when maintaining your own in-
> > tree Tempest tests [1]. If I understood that thread correctly that
> > breakage wouldn't have occurred if the stable branch tests were gating
> > Tempest proper... which is a very hard thing to do if we have our own
> > in-tree stuff. So there is a cost to doing what you suggest here, but
> > probably one that we'd be willing to accept.
>
> I'm not sure we have the resources to write and maintain our own
> in-tree tempest plugin, tbh.
>
> > [1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116172.
> > html
> >
> > Dan
> >
>
> It sounds like we haven't reached full consensus yet but at least we
> agreed on experimenting with Tempest.
>
> While we make progress on the discussion, I'll start working with some
> folks as experimental jobs, so we can make progress in the meantime.
>
> Thanks,
> --
> Emilien Macchi
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170505/f52108f5/attachment.html>