Open Stack

On Tue, Aug 16, 2016 at 4:49 PM, James Slagle <james.slagle at gmail.com> wrote:
> On Mon, Aug 15, 2016 at 4:54 AM, Dmitry Tantsur <dtantsur at redhat.com> wrote:
>> Hi everyone, happy Monday :)
>>
>> I'd like to start the discussion about CI-testing the optional composable
>> services in the CI (I'm primarily interested in Ironic, but I know there are
>> a lot more).
>>
>> Currently every time we change something in an optional service, we have to
>> create a DO-NOT-MERGE patch making the service in question not optional.
>> This approach has several problems:
>>
>> 1. It's not usually done for global refactorings.
>>
>> 2. The current CI does not have any specific tests to check that the
>> services in question actually works at all (e.g. in my experience the CI was
>> green even though nova-compute could not reach ironic).
>>
>> 3. If something breaks, it's hard to track the problem down to a specific
>> patch, as there is no history of gate runs.
>>
>> 4. It does not test the environment files we provide for enabling the
>> service.
>>
>> So, are there any plans to start covering optional services? Maybe at least
>> a non-HA job with all environment files included? It would be cool to also
>> somehow provide additional checks though. Or, in case of ironic, to disable
>> the regular nova compute, so that the ping test runs on an ironic instance.
>
> There are no immediate plans. Although I think the CI testing matrix
> is always open for discussion.
>
> I'm a little skeptical we will be able to deploy all services within
> the job timeout. And if we are, such a job seems better suited as a
> periodic job than in the check queue.
>
> The reason being is that there are already many different services
> that can break TripleO, and I'd rather focus on improving the testing
> of the actual deployment framework itself, instead of testing the
> "whole world" on every patch. We only have so much capacity. For
> example, I'd rather see us testing updates or upgrades on each patch
> instead of all the services.
>
> That being said, if you wanted to add a job that covered Ironic, I'd
> at least start with adding a job in the experimental queue. If it
> proves to be stable, we can always consider moving it to the check
> queue.
>

TripleO CI is having the same problem as Puppet CI had some time ago,
when we wanted to test more and more.

We solved this problem with scenarios.
Everything is documented here:
https://github.com/openstack/puppet-openstack-integration#description

We're testing all scenarios for all commits in
puppet-openstack-integration (equivalent to tripleo-ci) but only some
scenarios for each Puppet module.
Ex: OpenStack Sahara is deployed in scenario003, so a patch in
puppet-sahara will only run scenario003 job. We do that in Zuul
layout.

Because it would be complicated to do the same with TripleO Heat
Templates, we could run the same kind of job in periodic pipeline.
Though for Puppet jobs, we could run the tripleo scenarios in the
check/gate, as we do with the current multinode job.

So here's a summary of my proposal (and I volunteer to actually work
on it, like I did for Puppet CI):
* Call current jobs "compute-kit" which are current nonha/ha (ovb and
multinode) jobs deploying the basic services (undercloud +
Nova/Neutron/Glance/Keystone/Heat/Cinder/Swift and common services,
apache, mysql, etc).
* Create TripleO envs deploying different scenarios (we could start by
scenario001 deploying "compute-kit" + ceph (rbd backend for Nova /
Glance / Cinder). It's an example, feel free to propose something
else. The envs would live in tripleo-ci repo among others ci envs.
* Switch puppet-ceph zuul layout to stop running ovb ha job and run
the scenario001 job in check/gate (with the experimental pipeline
transition, as usual).
* Run scenario001 job in check/gate of tripleo-ci among other jobs.
* Run scenario001 job in periodic pipeline for puppet-tripleo and
tripleo-heat-templates.

Any feedback is welcome, but I think this would be a good start of
scaling our CI jobs coverage.
-- 
Emilien Macchi