[openstack-dev] [tripleo] becoming third party CI (was: enabling third party CI)

Dan Prince dprince at redhat.com
Thu Mar 10 21:27:02 UTC 2016


On Thu, 2016-03-10 at 13:45 -0500, Paul Belanger wrote:
> On Thu, Mar 10, 2016 at 05:32:15PM +0000, Jeremy Stanley wrote:
> > 
> > On 2016-03-10 09:50:03 -0500 (-0500), Emilien Macchi wrote:
> > [...]
> > > 
> > > OpenStack Infra provides an easy way to plug CI systems and some
> > > CIs (Nova, Neutron, Cinder, etc) already gate on some third party
> > > systems. I was wondering if we would not be interested to
> > > investigate this area and maybe ask to our third party drivers
> > > contributors (Bigswitch, Nuage, Midonet, Cisco, Netapp, etc) to
> > > run on their own hardware TripleO CI jobs running their specific
> > > environment to enable the drivers. This CI would be plugged to
> > > TripleO CI, and would provide awesome feedback.
> > [...]
> > 
> > It's also worth broadening the discussion to reassess whether the
> > existing TripleO CI should itself follow our third-party
> > integration
> > model instead of the current implementation relying on our main
> > community Zuul/Nodepool/Jenkins servers. When this was first
> > implemented, there was a promise of adding more regions for
> > robustness and of being able to use the surplus resources
> > maintained
> > in the TripleO CI clouds to augment our generic CI workload. It's
> > been years now and these things have not really come to pass; if
> > anything, that system and its operators are still struggling to
> > keep
> > a single region up and operational and providing enough resources
> > to
> > handle the current TripleO test load.
> > 
> > The majority of unplanned whole-provider outages we've experienced
> > in Nodepool have been from the TripleO cloud going completely
> > offline (sometimes for a week or more straight), by far the
> > longest-running jobs we have are running there (which substantially
> > hampers our ability to do things like gracefully restart our
> > Jenkins
> > masters without aborting running jobs), and ultimately the benefits
> > to TripleO for this model are very minimal anyway (different
> > pipelines means the jobs aren't effectively even voting, much less
> > gating).
> > 
> > I'm not trying to slam the TripleO cloud operators, I think they're
> > doing an amazing job given the limitations they're working under
> > and
> > much of their work has provided inspiration for our Infra-Cloud
> > project too. They're helpful and responsive and a joy to
> > collaborate
> > with, but ultimately I think TripleO might actually realize more
> > benefit from adding a Zuul/Nodepool/Jenkins of their own to this
> > (we've massively streamlined the Puppet for maintaining these
> > recently and have very thorough deployment and operational
> > documentation) rather than dealing with the issues which arise from
> > being half-integrated into one they don't control.
> > 
> > I've been meaning to bring that up for discussion for a while, just
> > keep forgetting, but this thread seems like a good segue into the
> > topic.
> I tend to agree here, I think a lot of great work has been done to
> allow new 3rd
> party CI system to come online.  Especially considering we have the
> puppet-openstackci[1] module.
> 
> However, I would also like to see tripleO move more inline with our
> existing CI
> tooling, if possible.  I know that wouldn't happen over night, but
> would at
> least give better insight into how the CI is working.

TripleO uses more of OpenStack tooling than just about any project I
know of. We do have some unique requirements related to the fact that
our CI actually PXE boots instances in a cloud. Something like this:

http://blog.nemebean.com/tags/quintupleo

We have plans on the table to potentially split our Heat stack (or make
it more configurable) such that we can test the configuration side on
normal cloud instances. I'm all for the effort in this and it would get
us closer to what I think you are talking about is "normal".

Like it our not our CI does catch things that nobody else is catching.
Quirky deployment things happen and until someone gets nested virt
working on commodity cloud servers (well) I think we have a case for
our own CI cloud.

> 
> Now that we have the infracloud too, it might be worth talking about
> doing the
> samething with TripleO hardware, again if possible. There is likely
> corner cases
> where it wouldn't work, but would be interesting to talk about it.

The corner case would be does it allow us to PXE boot an instance (thus
allowing provisioning via Ironic, etc.).

We could certainly entertain the option of creating our own OVB cloud
and managing it alongside of infracloud if that is what you are asking.
 I don't think it is the best fit for TripleO today given our unique
requirements at the moment.

Dan


> 
> [1] https://github.com/openstack-infra/puppet-openstackci
> 
> _____________________________________________________________________
> _____
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list