[openstack-dev] [tripleo] CI scenarios design - how to add more services

Giulio Fidente gfidente at redhat.com
Wed Nov 23 21:13:58 UTC 2016

hi Emilien,

thanks for putting some thought into this. We have a similar problem to 
test RGW which was only added in Newton.

On 11/23/2016 03:02 AM, Emilien Macchi wrote:
> == Context
> In Newton we added new multinode jobs called "scenarios".
> The challenged we tried to solve was "how to test the maximum of
> services without overloading the nodes that run tests".
> Each scenarios deploys a set of services, which allows us to
> horizontally scale the number of scenarios to increase the service
> testing coverage.
> See the result here:
> https://github.com/openstack-infra/tripleo-ci#service-testing-matrix
> To implement this model, we took example of Puppet OpenStack CI:
> https://github.com/openstack/puppet-openstack-integration#description
> We even tried to keep consistent the services/scenarios relations, so
> it's consistent and easier to maintain.
> Everything was fine until we had to add new services during Ocata cycles.
> Because tripleo-ci repository is not branched, adding Barbican service
> in the TripleO environment for scenario002 would break Newton CI jobs.
> During my vacations, the team created a new scenario, scenario004,
> that deploys Barbican and that is only run for Ocata jobs.
> I don't think we should proceed this way, and let me explain why.
> == Problem
> How to scale the number of services that we test without increasing
> the number of scenarios and therefore the complexity of maintaining
> them on long-term.
> == Solutions
> The list is not exhaustive, feel free to add more.
> 1) Re-use experience from Puppet OpenStack CI and have environments
> that are in a branched repository.
> environments.
> In Puppet OpenStack CI, the repository that deploys environments
> (puppet-openstack-integration) is branched. So if puppet-barbican is
> ready to be tested in Ocata, we'll patch
> puppet-openstack-integration/master to start testing it and it won't
> break stable jobs.
> Like this, we were successfully able to maintain a fair number of
> scenarios and keep our coverage increasing over each cycle.
> I see 2 sub-options here:
> a) Move CI environments and pingtest into
> tripleo-heat-templates/environments/ci/(scenarios|pingtest). This repo
> is branched and we could add a README to explain these files are used
> in CI and we don't guarantee they would work outside TripleO CI tools.
> b) Branch tripleo-ci repository. Personally I don't like this solution
> because a lot of patches in this repo are not related to OpenStack
> versions, which means we would need to backport most of the things
> from master.

I'd +1 this idea. It sounds like we could make the scenarios generic 
enough to be usable also outside CI? Maybe they can serve as samples?
Giulio Fidente

More information about the OpenStack-dev mailing list