Open Stack

On Thu, May 4, 2017 at 11:11 PM Dan Prince <dprince at redhat.com> wrote:

> On Thu, 2017-05-04 at 14:11 -0400, Emilien Macchi wrote:
> > On Thu, May 4, 2017 at 9:41 AM, Dan Prince <dprince at redhat.com>
> > wrote:
> > > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote:
> > > > ----- Original Message -----
> > > > > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote:
> > > > > > (cross-posting)
> > > > > > Instead of running the Pingtest, we would execute a Tempest
> > > > > > Scenario
> > > > > > that boot an instance from volume (like Pingstest is already
> > > > > > doing)
> > > > > > and see how it goes (in term of coverage and runtime).
> > > > > > I volunteer to kick-off the work with someone more expert
> > > > > > than I
> > > > > > am
> > > > > > with quickstart (Arx maybe?).
> > > > > >
> > > > > > Another iteration could be to start building an easy
> > > > > > interface to
> > > > > > select which Tempest tests we want a TripleO CI job to run
> > > > > > and
> > > > > > plug
> > > > > > it
> > > > > > to our CI tooling (tripleo-quickstart I presume).
> > > > >
> > > > > Running a subset of Tempest tests isn't the same thing as
> > > > > designing
> > > > > (and owning) your own test suite that targets the things that
> > > > > mean
> > > > > the
> > > > > most to our community (namely speed and coverage). Even giving
> > > > > up
> > > > > 5-10
> > > > > minutes of runtime...just to be able to run Tempest isn't
> > > > > something
> > > > > that some of us would be willing to do.
> > > >
> > > > As I mentioned, you can do it with Tempest (the library). You can
> > > > have your own test suite that does exactly what you are asking
> > > > (namely, a set of scenario tests based on Heat which targets the
> > > > TripleO use case) in a Tempest plugin and there is no absolute
> > > > reason
> > > > that those tests should add 5-10 minutes of runtime compared to
> > > > pingtest.
> > > >
> > > > It/they would be exactly pingtest, only implemented using a
> > > > different
> > > > library and running with a different runner, with the *exact*
> > > > same
> > > > run time.
> > > >
> > > > Obvious advantages: only one technology used to run tests, so if
> > > > anyone else want to run additional tests, there is no need to
> > > > maintain two code paths; reuse on a big and proven library of
> > > > test
> > > > and test runner tools.
> > >
> > > I like the idea of getting pingtest out of tripleo.sh as more of a
> > > stand alone tool. I would support an effort that re-implemented
> > > it...
> > > and using tempest-lib would be totally fine. And as you point out
> > > one
> > > could even combine these tests with a more common "Tempest" run
> > > that
> > > incorporates the scenarios, etc.
> >
> > I don't understand why we would re-implement the pingtest in a
> > tempest plugin.
> > Could you please tell us what is the technical difference between
> > what
> > does this scenario :
> > https://github.com/openstack/tempest/blob/master/tempest/scenario/tes
> > t_volume_boot_pattern.py
> >
> > And this pingtest:
> > https://github.com/openstack/tripleo-heat-templates/blob/master/ci/pi
> > ngtests/tenantvm_floatingip.yaml
> >
> > They both create a volume Cinder, snapshot it in Glance and and spawn
> > a Nova server from the volume.
> >
> > What one does that the other one doesn't?
>
> I don't think these are the same things. Does the Tempest test even
> create a floating IP? And in the case of pingtest we also cover Heat
> API in the overcloud (also valuable coverage). And even if they could
> be made to match today is there any guarantee that they would diverge
> in the future or maintain the same speed goals as that test lives in
> Tempest (and most TripleO cores don't review there).
>
> The main difference that I care about is it is easier for us to
> maintain and fix the pingtest varient at this point. We care a lot
> about our CI, and like I said before increasing the runtime isn't
> something we could easily tolerate. I'm willing to entertain reuse so
> long as it also allows us the speed and control we desire.
>
> >
> > > To me the message is clear that we DO NOT want to consume the
> > > normal
> > > Tempest scenarios in TripleO upstream CI at this point. Sure there
> > > is
> > > overlap there, but the focus of those tests is just plain
> > > different...
> >
> > I haven't seen strong pushback in this thread except you.
>
> Perhaps most cores haven't weighed in on this issue because moving to
> Tempest(-lib) isn't the most pressing issue ATM. We have a lot of
> architectural changes happening at the moment for example and that is
> why I only replied to this thread this week.
>
> > I'm against overlap in general and this one is pretty obvious. Why
> > would we maintain a tripleo-specific Tempest scenario while existing
> > ones would work for us. Please give me a technical reason in what is
> > not good enough in the existing scenarios.
>
> We maintain it because we care about speed I think.


We (QA) care about speed as well :)

Even when deploying with devstack, we are rather resource and time
contained. Granted, not to the level of TripleO perhaps.
We use a "slow" tag to mark tests that take longer to run, which could
be useful for you as well.


> Also, the re-use
> you talk about here has a fairly large cost to us in that we'd be
> introducing new dependencies and code reviews for TripleO. All of this
> has a cost too... when it really comes down to it I think the simpler
> implementation of ping test suites what we do and need in TripleO quite
> fine already.
>
> My assumption was that some of the people chiming into this thread
> would do the heavy lifting to give us a tempest.lib parity test that
> covers the same things we do today (in a similar runtime). Call it an
> experiment I guess... just to see what it would look like if we went
> down this path. If it works out... and everyone likes it great. If not
> then perhaps we still keep some of the simplicity we have today with
> pingtest. But I'm still not convinced that this test lives in Tempest
> proper because then we'd loose our guarantees of speed (runtime) and
> coverage as it is out of our control.
>
> Dan
>
>
> >
> > > speed isn't a primary concern there as it is for us so I don't
> > > think we
> > > should do it now. And probably not ever unless the CI job time is
> > > less
> > > than an hour. Like even if we were able to tune a set of stock
> > > Tempest
> > > smoke tests today to our liking unless TripleO proper gates on the
> > > runtime of those not increasing we'd be at risk of breaking our CI
> > > queues as the wall time would potentially get too long. In this
> > > regard
> > > this entire thread is poorly named I think in that we are no longer
> > > talking about 'pingtest vs. tempest' but rather the implementation
> > > details of how we reimplement our existing pingtest to better suite
> > > the
> > > community.
> >
> > What I would like to see if we're going to use Tempest in our gate,
> > is
> > to run at least one TripleO jobs as voting in Tempest.
> >
> > Tempest folks: I need your support here. We have been running Puppet
> > jobs as non-voting and we have seen a quite number of patches that
> > broke us because folks were ignore the jobs. If we switch TripleO to
> > use more Tempest, being in your gate might be required. We'll run the
> > fastest and  more stable job that we have to make sure the impact for
> > you is minimal.
> >
> > > So ++ for the idea of experimenting with the use of tempest.lib.
> > > But
> > > stay away from the idea of using Tempest smoke tests and the like
> > > for
> > > TripleO I think ATM.
> > >
> > > Its also worth noting there is some risk when maintaining your own
> > > in-
> > > tree Tempest tests [1]. If I understood that thread correctly that
> > > breakage wouldn't have occurred if the stable branch tests were
> > > gating
> > > Tempest proper... which is a very hard thing to do if we have our
> > > own
> > > in-tree stuff. So there is a cost to doing what you suggest here,
> > > but
> > > probably one that we'd be willing to accept.
> >
> > I'm not sure we have the resources to write and maintain our own
> > in-tree tempest plugin, tbh.
> >
> > > [1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116
> > > 172.
> > > html
> > >
> > > Dan
> > >
> >
> > It sounds like we haven't reached full consensus yet but at least we
> > agreed on experimenting with Tempest.
> >
> > While we make progress on the discussion, I'll start working with
> > some
> > folks as experimental jobs, so we can make progress in the meantime.
> >
> > Thanks,
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170505/a3b3b42d/attachment.html>