Open Stack

== Context

In Newton we added new multinode jobs called "scenarios".
The challenged we tried to solve was "how to test the maximum of
services without overloading the nodes that run tests".

Each scenarios deploys a set of services, which allows us to
horizontally scale the number of scenarios to increase the service
testing coverage.
See the result here:
https://github.com/openstack-infra/tripleo-ci#service-testing-matrix

To implement this model, we took example of Puppet OpenStack CI:
https://github.com/openstack/puppet-openstack-integration#description
We even tried to keep consistent the services/scenarios relations, so
it's consistent and easier to maintain.

Everything was fine until we had to add new services during Ocata cycles.
Because tripleo-ci repository is not branched, adding Barbican service
in the TripleO environment for scenario002 would break Newton CI jobs.
During my vacations, the team created a new scenario, scenario004,
that deploys Barbican and that is only run for Ocata jobs.
I don't think we should proceed this way, and let me explain why.

== Problem

How to scale the number of services that we test without increasing
the number of scenarios and therefore the complexity of maintaining
them on long-term.


== Solutions

The list is not exhaustive, feel free to add more.

1) Re-use experience from Puppet OpenStack CI and have environments
that are in a branched repository.
environments.
In Puppet OpenStack CI, the repository that deploys environments
(puppet-openstack-integration) is branched. So if puppet-barbican is
ready to be tested in Ocata, we'll patch
puppet-openstack-integration/master to start testing it and it won't
break stable jobs.
Like this, we were successfully able to maintain a fair number of
scenarios and keep our coverage increasing over each cycle.

I see 2 sub-options here:

a) Move CI environments and pingtest into
tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo
is branched and we could add a README to explain these files are used
in CI and we don't guarantee they would work outside TripleO CI tools.
b) Branch tripleo-ci repository. Personally I don't like this solution
because a lot of patches in this repo are not related to OpenStack
versions, which means we would need to backport most of the things
from master.

2) Introduce branch-based scenario tests -
https://review.openstack.org/#/c/396008/
It duplicates a lot of code and it's imho not really effective, though
this solution would correctly works.

3) Introduce a new scenario each time we have new services (like we
did with scenario004).
By adding new scenarios at each release because we test new services
is imho the wrong choice because:
a) it adds complexity in our we're going to maintain these scenarios.
b) it consumes more CI resources that we would need when some patches
have to run all scenarios jobs.


So I gave my opinion on the solutions, discussion is now open and my
hope is that we find a consensus soon, so we can make progress in our
testing coverage.
Thanks,
-- 
Emilien Macchi