Open Stack

Greetings dear owls,

I would like to bring back an old topic: running tempest in the gate.

== Context

Right now, TripleO gate is running something called pingtest to
validate that the OpenStack cloud is working. It's an Heat stack, that
deploys a Nova server, some volumes, a glance image, a neutron network
and sometimes a little bit more.
To deploy the pingtest, you obviously need Heat deployed in your overcloud.

== Problems:

Although pingtest has been very helpful over the last years:
- easy to understand, it's an Heat template, like an OpenStack user
would do to deploy their apps.
- fast: the stack takes a few minutes to be created and validated

It has some limitations:
- Limitation to what Heat resources support (example: some OpenStack
resources can't be managed from Heat)
- Impossible to run a dynamic workflow (test a live migration for example)

== Solutions

1) Switch pingtest to Tempest run on some specific tests, with feature
parity of what we had with pingtest.
For example, we could imagine to run the scenarios that deploys VM and
boot from volume. It would test the same thing as pingtest (details
can be discussed here).
Each scenario would run more tests depending on the service that they
run (scenario001 is telemetry, so it would run some tempest tests for
Ceilometer, Aodh, Gnocchi, etc).
We should work at making the tempest run as short as possible, and the
close as possible from what we have with a pingtest.

2) Run custom scripts in TripleO CI tooling, called from the pingtest
(heat template), that would run some validations commands (API calls,
etc).
It has been investigated in the past but never implemented AFIK.

3) ?

I tried to make this text short and go straight to the point, please
bring feedback now. I hope we can make progress on $topic during Pike,
so we can increase our testing coverage and detect deployment issues
sooner.

Thanks,
-- 
Emilien Macchi