Open Stack

hi Emilien,

thanks for putting some thought into this. We have a similar problem to 
test RGW which was only added in Newton.

On 11/23/2016 03:02 AM, Emilien Macchi wrote:
> == Context
>
> In Newton we added new multinode jobs called "scenarios".
> The challenged we tried to solve was "how to test the maximum of
> services without overloading the nodes that run tests".
>
> Each scenarios deploys a set of services, which allows us to
> horizontally scale the number of scenarios to increase the service
> testing coverage.
> See the result here:
> https://github.com/openstack-infra/tripleo-ci#service-testing-matrix
>
> To implement this model, we took example of Puppet OpenStack CI:
> https://github.com/openstack/puppet-openstack-integration#description
> We even tried to keep consistent the services/scenarios relations, so
> it's consistent and easier to maintain.
>
> Everything was fine until we had to add new services during Ocata cycles.
> Because tripleo-ci repository is not branched, adding Barbican service
> in the TripleO environment for scenario002 would break Newton CI jobs.
> During my vacations, the team created a new scenario, scenario004,
> that deploys Barbican and that is only run for Ocata jobs.
> I don't think we should proceed this way, and let me explain why.
>
> == Problem
>
> How to scale the number of services that we test without increasing
> the number of scenarios and therefore the complexity of maintaining
> them on long-term.
>
>
> == Solutions
>
> The list is not exhaustive, feel free to add more.
>
> 1) Re-use experience from Puppet OpenStack CI and have environments
> that are in a branched repository.
> environments.
> In Puppet OpenStack CI, the repository that deploys environments
> (puppet-openstack-integration) is branched. So if puppet-barbican is
> ready to be tested in Ocata, we'll patch
> puppet-openstack-integration/master to start testing it and it won't
> break stable jobs.
> Like this, we were successfully able to maintain a fair number of
> scenarios and keep our coverage increasing over each cycle.
>
> I see 2 sub-options here:
>
> a) Move CI environments and pingtest into
> tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo
> is branched and we could add a README to explain these files are used
> in CI and we don't guarantee they would work outside TripleO CI tools.
> b) Branch tripleo-ci repository. Personally I don't like this solution
> because a lot of patches in this repo are not related to OpenStack
> versions, which means we would need to backport most of the things
> from master.

I'd +1 this idea. It sounds like we could make the scenarios generic 
enough to be usable also outside CI? Maybe they can serve as samples?
-- 
Giulio Fidente
GPG KEY: 08D733BA