Open Stack

On Mon, Aug 8, 2016 at 11:47 AM, James Slagle <james.slagle at gmail.com> wrote:
> On Sat, Aug 6, 2016 at 8:17 PM, Paul Belanger <pabelanger at redhat.com> wrote:
>> Greetings,
>>
>> 5 months ago fungi posted:
>>
>>   [tripleo] becoming third party CI (was: enabling third party CI)[1]
>>
>> About having the discussion whether the existing TripleO CI should itself follow
>> our third-party integration model instead of the current implementation relying
>> on our main community Zuul/Nodepool/Jenkins servers.
>>
>> The result of the thread had some pros and cons, which I encourge people to
>> re-read.
>>
>> At the Austin summit we continued the topic of moving tripleo-ci into 3rd party
>> CI. Again, consensus could not be reached however we made some progress.  I
>> would take on the responsibility to help bring tripleo-test-cloud-rh1 more
>> inline with openstack-infra tooling.
>>
>> That includes, but is not limited to:
>>
>>   - Initial support for centos-7 jenkins slave (tripleo-ci)
>>     https://review.openstack.org/#/c/312725/
>>   - Add centos-7 to tripleo cloud (project-config)
>>     https://review.openstack.org/#/c/311721/
>>   - Revert "Revert "Migrate tripleo to centos-7"" (project-config)
>>     https://review.openstack.org/#/c/327425/
>>   - Revert "Disable tripleo-test-cloud-rh1 until we have AFS mirrors" (project-config)
>>     https://review.openstack.org/#/c/349659/
>>   - Add tripleo-test-cloud grafana dashboard
>>     https://review.openstack.org/#/c/351251/
>>
>> And various other reviews adding AFS mirrors for centos / epel. Updates to
>> tripleo-ci using our openstack-infra AFS mirrors along with providing general
>> support for both tripleo-test-cloud-rh1 and tripleo-test-cloud-rh2.
>>
>> In a short amount of time, we've made great progress with
>> tripleo-test-cloud-rh1, helping bring it more inline with openstack-infra
>> tooling.  While we are not finished, there is still some private infrastrucuture
>> that tripleo-ci is depending on. I am confident in the next 3 months we should
>> have that all replaced and using openstack community infrastruture.
>>
>> However on Friday[2], we started talking about tripleo-test-cloud-rh1 again in
>> #openstack-infra and found ourselves revisiting the original email. It is all
>> driven from the current effort from tripleo to start using move community clouds
>> for running tripleo-ci jobs.  Today, 3 different type of tripleo-ci jobs are now
>> run across all our clouds, for example there is a centos-7-2-node jobs. However,
>> tripleo-test-cloud-rh1 is only today setup to accept only tripleo-ci jobs. This
>> job does not run on tripleo-test-cloud-rh1.
>>
>> jeblair posted the following statement:
>>
>>   It feels like the tripleo cloud has been grandfathered in its current state
>>   for a while.  I'd just like to make sure we're being fair to everyone.  So if
>>   tripleo wants to run tripleo jobs, then i think we should move it to 3rd party
>>   ci.  I think that's a fine choice and we can continue to work together
>>   (please!) but with better division of reponsibilities.  Or, if we want to
>>   revise the idea of a multi-provider hardware platform that's available for all
>>   openstack projects, i'm game for that.  It would be great, but more work.
>>
>> Should we continue the push to move tripleo-test-cloud-rh1 to 3rd party CI
>> (removing from nodepool.o.o) or do we start enabling more jobs on
>> tripleo-test-cloud-rh1 bringing the cloud even more into openstack-infra?
>>
>> My personal thoughts, as somebody who's been working on it for the last 4
>> months, I still feel tripleo-test-cloud-rh1 should move to 3rd party CI.
>> However, with the work done in the last 4 months, I believe
>> tripleo-test-cloud-rh1 _could_ start running additional jobs based on the work
>> above.
>>
>> [1] http://lists.openstack.org/pipermail/openstack-dev/2016-March/088988.html
>> [2] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-08-05.log.html#t2016-08-05T23:07:35
>
> I'd like to provide some additional context about where we'd like to
> go with the CI jobs running on all cloud providers. We've added
> centos7 2-node multinode jobs, and we have some single node jobs as
> well. What I'd like clarification around is there is
> hesitation/concern around TripleO building out additional mutlinode
> jobs that run on any cloud.
>
> I'd like to continue down this path of adding more CI jobs that can
> run on any nodepool managed cloud provider. To do so, we'd need to
> have 3+ node jobs. A 3 node job would let us test with an undercloud
> and a 2 node overcloud, while a 4 node job would let us test with an
> undercloud and a 3 node overcloud so we could test HA. 4 node HA jobs
> seem to work fine in my testing that I've done on
> tripleo-test-cloud-rh2, using the same network setup and infra images
> that we do for the 2 node job.
>
> tripleo-ci can't move entirely to using these types of multinode jobs
> though, because they do not exercise the nova/ironic deployment during
> the tests. So, there will always be some subset of tripleo-ci jobs
> that still need tripleo-test-cloud-rh2{1,2} as those clouds are
> configured with the necessary OpenStack features enabled to allow for
> pxe booting of instances.
>
> I'm certainly open and supportive of the idea of running all job types
> on the tripleo clouds. The work Paul has done should make this
> possible. rh2 is on "loan" to the TripleO project team while rh1 moved
> datacenters. But, I'm told it can become permanent if there is a need.
> Keep in mind this is not a single cloud with 2 regions. rh1 and rh2
> are 2 separate single region clouds, however they offer the equivalent
> functionality to run all tripleo-ci and infra job types (barring just
> a few outstanding bugs/configuration issues). So, there is failover
> for tripleo-ci jobs (if we re-enable rh2).
>
> At the same time, if there are concerns about capacity from Infra's
> perspective as we'd like to build out to at least 4-node multinode
> jobs, then I think it makes good sense to open up rh1/rh2 to all job
> types, so that they help with the capacity.
> In that case, it would not make sense to move tripleo-ci to be a 3rd
> party CI setup.
>
> If there is plenty of capacity on the existing cloud providers and no
> concerns about building out additional multinode jobs, then there
> might not be a huge benefit in enabling all job types on rh1/rh2. If
> that's the case then we might as well just move them to 3rd party CI.
>
> I suppose it's also possible that we might be pushing too strongly
> down the multinode path? Is the general concensus in infra that they'd
> like to help enable project teams to eventually add 3 and 4 (and maybe
> more) node multinode jobs? If not, then I agree it probably does make
> sense to move most of tripleo-ci to be 3rd party, and we'd just keep
> our existing check jobs as-is instead of proposing new ones.
>
> My point is: if there's concensus around TripleO building out to 3+
> node multinode jobs, then I think tripleo-ci should stay where it is
> so that we can also work towards contributing to available capacity,
> etc.

I think there are some values to test 3+ node multinode jobs:

- Testing High-Availability: we currently have pacemaker scenarios,
that require at least 3 controllers and 1 compute to test something
realistic. Though we have nothing in our tests that disable a
controller and run tests again. So unless we have it, I think we could
run our tests on a single controller node.

- Testing Nova live migration (even if it's tested by devstack) in
TripleO, to validate configuration that makes it possible.

- Testing 3+ means at least 2 nodes for overcloud, which is the first
realistic way to test a multi-node overcloud, and validate TripleO
Orchestration, for both initial deployment and upgrades. We are an
installer, we rely on orchestration and configuration management, so I
think we really need this minimal architecture.


Regarding the contribution to OpenStack Infra to give some capacity,
it would be certainly fair from us to do it, big +1.

> If there is not, and tripleo-ci just keeps adding more jobs that will
> only run on our own clouds, then I agree with Paul and Ben that
> tripleo-ci feels more naturally to be 3rd party CI.
>
> --
> -- James Slagle
> --
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi