Open Stack

On 08/17/2016 03:21 AM, Emilien Macchi wrote:
> On Tue, Aug 16, 2016 at 4:49 PM, James Slagle <james.slagle at gmail.com> wrote:
>> On Mon, Aug 15, 2016 at 4:54 AM, Dmitry Tantsur <dtantsur at redhat.com> wrote:
>>> Hi everyone, happy Monday :)
>>>
>>> I'd like to start the discussion about CI-testing the optional composable
>>> services in the CI (I'm primarily interested in Ironic, but I know there are
>>> a lot more).
>>>
<snip>
>>
>> The reason being is that there are already many different services
>> that can break TripleO, and I'd rather focus on improving the testing
>> of the actual deployment framework itself, instead of testing the
>> "whole world" on every patch. We only have so much capacity. For
>> example, I'd rather see us testing updates or upgrades on each patch
>> instead of all the services.

I don't suggest we test Ironic, I suggest we test the composable 
services installing Ironic, and then just do a sanity-check. Without 
such check an actual failure can go unnoticed, I've already faced with that.

>>
>> That being said, if you wanted to add a job that covered Ironic, I'd
>> at least start with adding a job in the experimental queue. If it
>> proves to be stable, we can always consider moving it to the check
>> queue.
>>
>
> TripleO CI is having the same problem as Puppet CI had some time ago,
> when we wanted to test more and more.
>
> We solved this problem with scenarios.
> Everything is documented here:
> https://github.com/openstack/puppet-openstack-integration#description
>
> We're testing all scenarios for all commits in
> puppet-openstack-integration (equivalent to tripleo-ci) but only some
> scenarios for each Puppet module.
> Ex: OpenStack Sahara is deployed in scenario003, so a patch in
> puppet-sahara will only run scenario003 job. We do that in Zuul
> layout.
>
> Because it would be complicated to do the same with TripleO Heat
> Templates, we could run the same kind of job in periodic pipeline.
> Though for Puppet jobs, we could run the tripleo scenarios in the
> check/gate, as we do with the current multinode job.
>
> So here's a summary of my proposal (and I volunteer to actually work
> on it, like I did for Puppet CI):
> * Call current jobs "compute-kit" which are current nonha/ha (ovb and
> multinode) jobs deploying the basic services (undercloud +
> Nova/Neutron/Glance/Keystone/Heat/Cinder/Swift and common services,
> apache, mysql, etc).
> * Create TripleO envs deploying different scenarios (we could start by
> scenario001 deploying "compute-kit" + ceph (rbd backend for Nova /
> Glance / Cinder). It's an example, feel free to propose something
> else. The envs would live in tripleo-ci repo among others ci envs.
> * Switch puppet-ceph zuul layout to stop running ovb ha job and run
> the scenario001 job in check/gate (with the experimental pipeline
> transition, as usual).
> * Run scenario001 job in check/gate of tripleo-ci among other jobs.
> * Run scenario001 job in periodic pipeline for puppet-tripleo and
> tripleo-heat-templates.
>
> Any feedback is welcome, but I think this would be a good start of
> scaling our CI jobs coverage.
>

I'm not particularly fond of using *only* periodic jobs. I don't think 
it alone solves the problems I've raised, because:
1. Periodic jobs do not give an immediate feedback of whether we break 
something in a THT patch.
2. Periodic jobs do not help tracking which exactly patch was breaking.
3. Periodic jobs results are slightly hidden, so a failure in a 
non-priority service (like Ironic) will probably stay unnoticed for 
quite a while.

However, the current gate system allows to run jobs based on files 
affected. So we can also run a scenario covering ironic on THT 
check/gate if puppet/services/*ironic* is affected, but not in the other 
cases. This won't cover all potential failures, but it would be of great 
help anyway. It should also run in experimental pipeline, so that it can 
be triggered on any patch.

This is in addition to periodic jobs you're proposing, not replacing 
them. WDYT?