Open Stack

On Sat, Aug 6, 2016 at 8:17 PM, Paul Belanger <pabelanger at redhat.com> wrote:
> Greetings,
>
> 5 months ago fungi posted:
>
>   [tripleo] becoming third party CI (was: enabling third party CI)[1]
>
> About having the discussion whether the existing TripleO CI should itself follow
> our third-party integration model instead of the current implementation relying
> on our main community Zuul/Nodepool/Jenkins servers.
>
> The result of the thread had some pros and cons, which I encourge people to
> re-read.
>
> At the Austin summit we continued the topic of moving tripleo-ci into 3rd party
> CI. Again, consensus could not be reached however we made some progress.  I
> would take on the responsibility to help bring tripleo-test-cloud-rh1 more
> inline with openstack-infra tooling.
>
> That includes, but is not limited to:
>
>   - Initial support for centos-7 jenkins slave (tripleo-ci)
>     https://review.openstack.org/#/c/312725/
>   - Add centos-7 to tripleo cloud (project-config)
>     https://review.openstack.org/#/c/311721/
>   - Revert "Revert "Migrate tripleo to centos-7"" (project-config)
>     https://review.openstack.org/#/c/327425/
>   - Revert "Disable tripleo-test-cloud-rh1 until we have AFS mirrors" (project-config)
>     https://review.openstack.org/#/c/349659/
>   - Add tripleo-test-cloud grafana dashboard
>     https://review.openstack.org/#/c/351251/
>
> And various other reviews adding AFS mirrors for centos / epel. Updates to
> tripleo-ci using our openstack-infra AFS mirrors along with providing general
> support for both tripleo-test-cloud-rh1 and tripleo-test-cloud-rh2.
>
> In a short amount of time, we've made great progress with
> tripleo-test-cloud-rh1, helping bring it more inline with openstack-infra
> tooling.  While we are not finished, there is still some private infrastrucuture
> that tripleo-ci is depending on. I am confident in the next 3 months we should
> have that all replaced and using openstack community infrastruture.
>
> However on Friday[2], we started talking about tripleo-test-cloud-rh1 again in
> #openstack-infra and found ourselves revisiting the original email. It is all
> driven from the current effort from tripleo to start using move community clouds
> for running tripleo-ci jobs.  Today, 3 different type of tripleo-ci jobs are now
> run across all our clouds, for example there is a centos-7-2-node jobs. However,
> tripleo-test-cloud-rh1 is only today setup to accept only tripleo-ci jobs. This
> job does not run on tripleo-test-cloud-rh1.
>
> jeblair posted the following statement:
>
>   It feels like the tripleo cloud has been grandfathered in its current state
>   for a while.  I'd just like to make sure we're being fair to everyone.  So if
>   tripleo wants to run tripleo jobs, then i think we should move it to 3rd party
>   ci.  I think that's a fine choice and we can continue to work together
>   (please!) but with better division of reponsibilities.  Or, if we want to
>   revise the idea of a multi-provider hardware platform that's available for all
>   openstack projects, i'm game for that.  It would be great, but more work.
>
> Should we continue the push to move tripleo-test-cloud-rh1 to 3rd party CI
> (removing from nodepool.o.o) or do we start enabling more jobs on
> tripleo-test-cloud-rh1 bringing the cloud even more into openstack-infra?
>
> My personal thoughts, as somebody who's been working on it for the last 4
> months, I still feel tripleo-test-cloud-rh1 should move to 3rd party CI.
> However, with the work done in the last 4 months, I believe
> tripleo-test-cloud-rh1 _could_ start running additional jobs based on the work
> above.
>
> [1] http://lists.openstack.org/pipermail/openstack-dev/2016-March/088988.html
> [2] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-08-05.log.html#t2016-08-05T23:07:35

I'd like to provide some additional context about where we'd like to
go with the CI jobs running on all cloud providers. We've added
centos7 2-node multinode jobs, and we have some single node jobs as
well. What I'd like clarification around is there is
hesitation/concern around TripleO building out additional mutlinode
jobs that run on any cloud.

I'd like to continue down this path of adding more CI jobs that can
run on any nodepool managed cloud provider. To do so, we'd need to
have 3+ node jobs. A 3 node job would let us test with an undercloud
and a 2 node overcloud, while a 4 node job would let us test with an
undercloud and a 3 node overcloud so we could test HA. 4 node HA jobs
seem to work fine in my testing that I've done on
tripleo-test-cloud-rh2, using the same network setup and infra images
that we do for the 2 node job.

tripleo-ci can't move entirely to using these types of multinode jobs
though, because they do not exercise the nova/ironic deployment during
the tests. So, there will always be some subset of tripleo-ci jobs
that still need tripleo-test-cloud-rh2{1,2} as those clouds are
configured with the necessary OpenStack features enabled to allow for
pxe booting of instances.

I'm certainly open and supportive of the idea of running all job types
on the tripleo clouds. The work Paul has done should make this
possible. rh2 is on "loan" to the TripleO project team while rh1 moved
datacenters. But, I'm told it can become permanent if there is a need.
Keep in mind this is not a single cloud with 2 regions. rh1 and rh2
are 2 separate single region clouds, however they offer the equivalent
functionality to run all tripleo-ci and infra job types (barring just
a few outstanding bugs/configuration issues). So, there is failover
for tripleo-ci jobs (if we re-enable rh2).

At the same time, if there are concerns about capacity from Infra's
perspective as we'd like to build out to at least 4-node multinode
jobs, then I think it makes good sense to open up rh1/rh2 to all job
types, so that they help with the capacity.
In that case, it would not make sense to move tripleo-ci to be a 3rd
party CI setup.

If there is plenty of capacity on the existing cloud providers and no
concerns about building out additional multinode jobs, then there
might not be a huge benefit in enabling all job types on rh1/rh2. If
that's the case then we might as well just move them to 3rd party CI.

I suppose it's also possible that we might be pushing too strongly
down the multinode path? Is the general concensus in infra that they'd
like to help enable project teams to eventually add 3 and 4 (and maybe
more) node multinode jobs? If not, then I agree it probably does make
sense to move most of tripleo-ci to be 3rd party, and we'd just keep
our existing check jobs as-is instead of proposing new ones.

My point is: if there's concensus around TripleO building out to 3+
node multinode jobs, then I think tripleo-ci should stay where it is
so that we can also work towards contributing to available capacity,
etc.

If there is not, and tripleo-ci just keeps adding more jobs that will
only run on our own clouds, then I agree with Paul and Ben that
tripleo-ci feels more naturally to be 3rd party CI.

-- 
-- James Slagle
--