Open Stack

On Wed, Apr 5, 2017 at 1:49 PM, Emilien Macchi <emilien at redhat.com> wrote:
[...]

> == Solutions
>
> 1) Switch pingtest to Tempest run on some specific tests, with feature
> parity of what we had with pingtest.
> For example, we could imagine to run the scenarios that deploys VM and
> boot from volume. It would test the same thing as pingtest (details
> can be discussed here).
> Each scenario would run more tests depending on the service that they
> run (scenario001 is telemetry, so it would run some tempest tests for
> Ceilometer, Aodh, Gnocchi, etc).
> We should work at making the tempest run as short as possible, and the
> close as possible from what we have with a pingtest.
[...]

4 months later :-)

We enabled Tempest on the following jobs:

gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container
gate-tripleo-ci-centos-7-scenario002-multinode-oooq-container
gate-tripleo-ci-centos-7-scenario003-multinode-oooq-container
gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container

gate-tripleo-ci-centos-7-scenario001-multinode-oooq
gate-tripleo-ci-centos-7-scenario002-multinode-oooq
gate-tripleo-ci-centos-7-scenario003-multinode-oooq
gate-tripleo-ci-centos-7-scenario004-multinode-oooq

gate-tripleo-ci-centos-7-nonha-multinode-oooq
gate-tripleo-ci-centos-7-containers-multinode

It has a feature parity with what pingtest did.
For example, scenario001 (focused on Telemetry) test boot from volume
and the whole autoscsaling scenario from Telemetry services, so we can
test end-to-end that our users can deploy autocsaling apps using
OpenStack.

We run Tempest on stable/pike and master, and still run pingtest on
stable/newton and stable/ocata (no change is planned for stable
branches).
No change has been planned yet for OVB jobs but we might want to think
about it later.

You can go on http://status.openstack.org/openstack-health/#/job/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container
and see stats as well.
We also watched the amount of time spent on pingtest versus tempest
and we knew tempest would take more time. It's the case and we've
noticed the average is ~2 min more for now. It's acceptable I guess
for now.

Anyway, we hope this effort was useful and we can make progress to the
next steps:
- notify teams when some tests fail
- increase test coverage depending on which services are deployed

-- 
Emilien Macchi