[openstack-dev] [tripleo] pingtest vs tempest
arxcruz at redhat.com
Thu May 4 12:04:15 UTC 2017
On Wed, May 3, 2017 at 11:53 PM, Emilien Macchi <emilien at redhat.com> wrote:
> I've seen a bunch of interesting thoughts here.
> The most relevant feedback I've seen so far:
> - TripleO folks want to keep testing fast and efficient.
> - Tempest folks understand this problematic and is willing to collaborate.
> I propose that we move forward and experiment the usage of Tempest in
> TripleO CI for one job that could be experimental or non-voting to
> Instead of running the Pingtest, we would execute a Tempest Scenario
> that boot an instance from volume (like Pingstest is already doing)
> and see how it goes (in term of coverage and runtime).
> I volunteer to kick-off the work with someone more expert than I am
> with quickstart (Arx maybe?).
Sure, let's work on that :)
> Another iteration could be to start building an easy interface to
> select which Tempest tests we want a TripleO CI job to run and plug it
> to our CI tooling (tripleo-quickstart I presume).
> I also hear some feedback about keeping the pingtest alive for some
> uses cases, and I agree we could keep some CI jobs to run the pingtest
> when it makes more sense (when we want to test Heat for example, or
> just maintain it for developers who used it).
> How does it sounds? Please bring feedback.
> On Tue, Apr 18, 2017 at 7:41 AM, Attila Fazekas <afazekas at redhat.com>
> > On Tue, Apr 18, 2017 at 11:04 AM, Arx Cruz <arxcruz at redhat.com> wrote:
> >> On Tue, Apr 18, 2017 at 10:42 AM, Steven Hardy <shardy at redhat.com>
> >>> On Mon, Apr 17, 2017 at 12:48:32PM -0400, Justin Kilpatrick wrote:
> >>> > On Mon, Apr 17, 2017 at 12:28 PM, Ben Nemec <openstack at nemebean.com>
> >>> > wrote:
> >>> > > Tempest isn't really either of those things. According to another
> >>> > > message
> >>> > > in this thread it takes around 15 minutes to run just the smoke
> >>> > > tests.
> >>> > > That's unacceptable for a lot of our CI jobs.
> >>> >
> >> I rather spend 15 minutes running tempest than add a regression or a new
> >> bug, which already happen in the past.
> > The smoke tests might not be the best test selection anyway, you should
> > some scenario which does
> > for example snapshot of images and volumes. yes, these are the slow ones,
> > but they can run in parallel.
> > Very likely you do not really want to run all tempest test, but 10~20
> > time,
> > sounds reasonable for a sanity test.
> > The tempest config utility also should be extended by some parallel
> > capability,
> > and should be able to use already downloaded (part of the image)
> > Tempest/testr/subunit worker balance is not always the best,
> > technically would be possible to do dynamic balancing, but it would
> > a lot of work.
> > Let me know when it becomes the main concern, I can check what
> can/cannot be
> > done.
> >>> > Ben, is the issue merely the time it takes? Is it the affect that
> >>> > taken has on hardware availability?
> >>> It's both, but the main constraint is the infra job timeout, which is
> >>> about
> >>> 2.5hrs - if you look at our current jobs many regularly get close to
> >>> sometimes exceed this), so we just don't have the time budget available
> >>> to
> >>> run exhasutive tests every commit.
> >> We have green light from infra to increase the job timeout to 5 hours,
> >> do that in our periodic full tempest job.
> > Sounds good, but I am afraid it could hurt more than helping, it could
> > other things get fixed by lot
> > especially if we got some extra flakiness, because of foobar.
> > You cannot have all possible tripleo configs on the gate anyway,
> > so something will pass which will require a quick fix.
> > IMHO the only real solution, is making the before test-run steps faster
> > shorter.
> > Do you have any option to start the tempest running jobs in a more
> > state ?
> > I mean, having more things already done at the start time
> > and just do a fast upgrade at the beginning of the job.
> > Openstack installation can be completed in a `fast` way (~minute) on
> > RHEL/Fedora systems
> > after the yum steps, also if you are able to aggregate all yum step to
> > single
> > command execution (transaction) you generally able to save a lot of time.
> > There is plenty of things what can be made more efficient before the test
> > run,
> > when you start considering everything evil which can be accounted for
> > than 30 sec
> > of time, this can happen soon.
> > For example just executing the cpython interpreter for the openstack
> > commands is above 30 sec,
> > the work what they are doing can be done in much much faster way.
> > Lot of install steps actually does not depends on each other,
> > it allows more things to be done in parallel, we generally can have more
> > core than Ghz.
> >>> > Should we focus on how much testing we can get into N time period?
> >>> > Then how do we decide an optimal N
> >>> > for our constraints?
> >>> Well yeah, but that's pretty much how/why we ended up with pingtest,
> >>> simple, fast, and provides an efficient way to do smoke tests, e.g
> >>> creating
> >>> just one heat resource is enough to prove multiple OpenStack services
> >>> running, as well as the DB/RPC etc etc.
> >>> > I've been working on a full up functional test for OpenStack CI
> >>> > for a long time now, it works but takes
> >>> > more than 10 hours. IF you're interested in results kick through to
> >>> > Kibana here . Let me know off list if you
> >>> > have any issues, the presentation of this data is all experimental
> >>> > still.
> >>> This kind of thing is great, and I'd support more exhaustive testing
> >>> periodic jobs etc, but the reality is we need to focus on "bang for
> >>> e.g the deepest possible coverage in the most minimal amount of time
> >>> our per-commit tests - we rely on the project gates to provide a full
> >>> surface test, and we need to focus on more basic things like "did the
> >>> service
> >>> start", and "is the API accessible". Simple crud operations on a
> >>> of
> >>> the API's is totally fine for this IMO, whether via pingtest or some
> >>> other
> >>> means.
> >> Right now we do have a periodic job running full tempest, with a few
> >> skips, and because of the lack of tempest tests in the patches, it's
> >> pretty hard to keep it stable enough to have a 100% pass, and of course,
> >> also the installation very often fails (like in the last five days).
> >> For example,  is the latest run we have in periodic job that we get
> >> results from tempest, and we have 114 failures that was caused by some
> >> code/change, and I have no idea which one was, just looking at the
> >> I can notice that smoke tests plus minimum basic scenario tests would
> >> these failures and the developer could fix it and make me happy :)
> >> Now I have to spend several hours installing and debugging each one of
> >> those tests to identify where/why it fails.
> >> Before this run, we got 100% pass, but unfortunately I don't have the
> >> results anymore, it was removed already from logs.openstack.org
> >>> Steve
> >>> ____________________________________________________________
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe:
> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> 
> >> http://logs.openstack.org/periodic/periodic-tripleo-ci-
> >> ____________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > ____________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Emilien Macchi
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-dev