[openstack-dev] [tripleo] pingtest vs tempest

Dan Prince dprince at redhat.com
Thu May 4 22:08:29 UTC 2017


On Thu, 2017-05-04 at 14:11 -0400, Emilien Macchi wrote:
> On Thu, May 4, 2017 at 9:41 AM, Dan Prince <dprince at redhat.com>
> wrote:
> > On Thu, 2017-05-04 at 03:11 -0400, Luigi Toscano wrote:
> > > ----- Original Message -----
> > > > On Wed, 2017-05-03 at 17:53 -0400, Emilien Macchi wrote:
> > > > > (cross-posting)
> > > > > Instead of running the Pingtest, we would execute a Tempest
> > > > > Scenario
> > > > > that boot an instance from volume (like Pingstest is already
> > > > > doing)
> > > > > and see how it goes (in term of coverage and runtime).
> > > > > I volunteer to kick-off the work with someone more expert
> > > > > than I
> > > > > am
> > > > > with quickstart (Arx maybe?).
> > > > > 
> > > > > Another iteration could be to start building an easy
> > > > > interface to
> > > > > select which Tempest tests we want a TripleO CI job to run
> > > > > and
> > > > > plug
> > > > > it
> > > > > to our CI tooling (tripleo-quickstart I presume).
> > > > 
> > > > Running a subset of Tempest tests isn't the same thing as
> > > > designing
> > > > (and owning) your own test suite that targets the things that
> > > > mean
> > > > the
> > > > most to our community (namely speed and coverage). Even giving
> > > > up
> > > > 5-10
> > > > minutes of runtime...just to be able to run Tempest isn't
> > > > something
> > > > that some of us would be willing to do.
> > > 
> > > As I mentioned, you can do it with Tempest (the library). You can
> > > have your own test suite that does exactly what you are asking
> > > (namely, a set of scenario tests based on Heat which targets the
> > > TripleO use case) in a Tempest plugin and there is no absolute
> > > reason
> > > that those tests should add 5-10 minutes of runtime compared to
> > > pingtest.
> > > 
> > > It/they would be exactly pingtest, only implemented using a
> > > different
> > > library and running with a different runner, with the *exact*
> > > same
> > > run time.
> > > 
> > > Obvious advantages: only one technology used to run tests, so if
> > > anyone else want to run additional tests, there is no need to
> > > maintain two code paths; reuse on a big and proven library of
> > > test
> > > and test runner tools.
> > 
> > I like the idea of getting pingtest out of tripleo.sh as more of a
> > stand alone tool. I would support an effort that re-implemented
> > it...
> > and using tempest-lib would be totally fine. And as you point out
> > one
> > could even combine these tests with a more common "Tempest" run
> > that
> > incorporates the scenarios, etc.
> 
> I don't understand why we would re-implement the pingtest in a
> tempest plugin.
> Could you please tell us what is the technical difference between
> what
> does this scenario :
> https://github.com/openstack/tempest/blob/master/tempest/scenario/tes
> t_volume_boot_pattern.py
> 
> And this pingtest:
> https://github.com/openstack/tripleo-heat-templates/blob/master/ci/pi
> ngtests/tenantvm_floatingip.yaml
> 
> They both create a volume Cinder, snapshot it in Glance and and spawn
> a Nova server from the volume.
> 
> What one does that the other one doesn't?

I don't think these are the same things. Does the Tempest test even
create a floating IP? And in the case of pingtest we also cover Heat
API in the overcloud (also valuable coverage). And even if they could
be made to match today is there any guarantee that they would diverge
in the future or maintain the same speed goals as that test lives in
Tempest (and most TripleO cores don't review there).

The main difference that I care about is it is easier for us to
maintain and fix the pingtest varient at this point. We care a lot
about our CI, and like I said before increasing the runtime isn't
something we could easily tolerate. I'm willing to entertain reuse so
long as it also allows us the speed and control we desire.

> 
> > To me the message is clear that we DO NOT want to consume the
> > normal
> > Tempest scenarios in TripleO upstream CI at this point. Sure there
> > is
> > overlap there, but the focus of those tests is just plain
> > different...
> 
> I haven't seen strong pushback in this thread except you.

Perhaps most cores haven't weighed in on this issue because moving to
Tempest(-lib) isn't the most pressing issue ATM. We have a lot of
architectural changes happening at the moment for example and that is
why I only replied to this thread this week.

> I'm against overlap in general and this one is pretty obvious. Why
> would we maintain a tripleo-specific Tempest scenario while existing
> ones would work for us. Please give me a technical reason in what is
> not good enough in the existing scenarios.

We maintain it because we care about speed I think. Also, the re-use
you talk about here has a fairly large cost to us in that we'd be
introducing new dependencies and code reviews for TripleO. All of this
has a cost too... when it really comes down to it I think the simpler
implementation of ping test suites what we do and need in TripleO quite
fine already.

My assumption was that some of the people chiming into this thread
would do the heavy lifting to give us a tempest.lib parity test that
covers the same things we do today (in a similar runtime). Call it an
experiment I guess... just to see what it would look like if we went
down this path. If it works out... and everyone likes it great. If not
then perhaps we still keep some of the simplicity we have today with
pingtest. But I'm still not convinced that this test lives in Tempest
proper because then we'd loose our guarantees of speed (runtime) and
coverage as it is out of our control.

Dan


> 
> > speed isn't a primary concern there as it is for us so I don't
> > think we
> > should do it now. And probably not ever unless the CI job time is
> > less
> > than an hour. Like even if we were able to tune a set of stock
> > Tempest
> > smoke tests today to our liking unless TripleO proper gates on the
> > runtime of those not increasing we'd be at risk of breaking our CI
> > queues as the wall time would potentially get too long. In this
> > regard
> > this entire thread is poorly named I think in that we are no longer
> > talking about 'pingtest vs. tempest' but rather the implementation
> > details of how we reimplement our existing pingtest to better suite
> > the
> > community.
> 
> What I would like to see if we're going to use Tempest in our gate,
> is
> to run at least one TripleO jobs as voting in Tempest.
> 
> Tempest folks: I need your support here. We have been running Puppet
> jobs as non-voting and we have seen a quite number of patches that
> broke us because folks were ignore the jobs. If we switch TripleO to
> use more Tempest, being in your gate might be required. We'll run the
> fastest and  more stable job that we have to make sure the impact for
> you is minimal.
> 
> > So ++ for the idea of experimenting with the use of tempest.lib.
> > But
> > stay away from the idea of using Tempest smoke tests and the like
> > for
> > TripleO I think ATM.
> > 
> > Its also worth noting there is some risk when maintaining your own
> > in-
> > tree Tempest tests [1]. If I understood that thread correctly that
> > breakage wouldn't have occurred if the stable branch tests were
> > gating
> > Tempest proper... which is a very hard thing to do if we have our
> > own
> > in-tree stuff. So there is a cost to doing what you suggest here,
> > but
> > probably one that we'd be willing to accept.
> 
> I'm not sure we have the resources to write and maintain our own
> in-tree tempest plugin, tbh.
> 
> > [1] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116
> > 172.
> > html
> > 
> > Dan
> > 
> 
> It sounds like we haven't reached full consensus yet but at least we
> agreed on experimenting with Tempest.
> 
> While we make progress on the discussion, I'll start working with
> some
> folks as experimental jobs, so we can make progress in the meantime.
> 
> Thanks,



More information about the OpenStack-dev mailing list