Open Stack

On Mon, Apr 17, 2017 at 12:48:32PM -0400, Justin Kilpatrick wrote:
> On Mon, Apr 17, 2017 at 12:28 PM, Ben Nemec <openstack at nemebean.com> wrote:
> > Tempest isn't really either of those things.  According to another message
> > in this thread it takes around 15 minutes to run just the smoke tests.
> > That's unacceptable for a lot of our CI jobs.
> 
> Ben, is the issue merely the time it takes? Is it the affect that time
> taken has on hardware availability?

It's both, but the main constraint is the infra job timeout, which is about
2.5hrs - if you look at our current jobs many regularly get close to (and
sometimes exceed this), so we just don't have the time budget available to
run exhasutive tests every commit.

> Should we focus on how much testing we can get into N time period?
> Then how do we decide an optimal N
> for our constraints?

Well yeah, but that's pretty much how/why we ended up with pingtest, it's
simple, fast, and provides an efficient way to do smoke tests, e.g creating
just one heat resource is enough to prove multiple OpenStack services are
running, as well as the DB/RPC etc etc.

> I've been working on a full up functional test for OpenStack CI builds
> for a long time now, it works but takes
> more than 10 hours. IF you're interested in results kick through to
> Kibana here [0]. Let me know off list if you
> have any issues, the presentation of this data is all experimental still.

This kind of thing is great, and I'd support more exhaustive testing via
periodic jobs etc, but the reality is we need to focus on "bang for buck"
e.g the deepest possible coverage in the most minimal amount of time for
our per-commit tests - we rely on the project gates to provide a full API
surface test, and we need to focus on more basic things like "did the service
start", and "is the API accessible".  Simple crud operations on a subset of
the API's is totally fine for this IMO, whether via pingtest or some other
means.

Steve