Open Stack

On Thu, Mar 10, 2016 at 05:32:15PM +0000, Jeremy Stanley wrote:
> On 2016-03-10 09:50:03 -0500 (-0500), Emilien Macchi wrote:
> [...]
> > OpenStack Infra provides an easy way to plug CI systems and some
> > CIs (Nova, Neutron, Cinder, etc) already gate on some third party
> > systems. I was wondering if we would not be interested to
> > investigate this area and maybe ask to our third party drivers
> > contributors (Bigswitch, Nuage, Midonet, Cisco, Netapp, etc) to
> > run on their own hardware TripleO CI jobs running their specific
> > environment to enable the drivers. This CI would be plugged to
> > TripleO CI, and would provide awesome feedback.
> [...]
> 
> It's also worth broadening the discussion to reassess whether the
> existing TripleO CI should itself follow our third-party integration
> model instead of the current implementation relying on our main
> community Zuul/Nodepool/Jenkins servers. When this was first
> implemented, there was a promise of adding more regions for
> robustness and of being able to use the surplus resources maintained
> in the TripleO CI clouds to augment our generic CI workload. It's
> been years now and these things have not really come to pass; if
> anything, that system and its operators are still struggling to keep
> a single region up and operational and providing enough resources to
> handle the current TripleO test load.
> 
> The majority of unplanned whole-provider outages we've experienced
> in Nodepool have been from the TripleO cloud going completely
> offline (sometimes for a week or more straight), by far the
> longest-running jobs we have are running there (which substantially
> hampers our ability to do things like gracefully restart our Jenkins
> masters without aborting running jobs), and ultimately the benefits
> to TripleO for this model are very minimal anyway (different
> pipelines means the jobs aren't effectively even voting, much less
> gating).
> 
> I'm not trying to slam the TripleO cloud operators, I think they're
> doing an amazing job given the limitations they're working under and
> much of their work has provided inspiration for our Infra-Cloud
> project too. They're helpful and responsive and a joy to collaborate
> with, but ultimately I think TripleO might actually realize more
> benefit from adding a Zuul/Nodepool/Jenkins of their own to this
> (we've massively streamlined the Puppet for maintaining these
> recently and have very thorough deployment and operational
> documentation) rather than dealing with the issues which arise from
> being half-integrated into one they don't control.
> 
> I've been meaning to bring that up for discussion for a while, just
> keep forgetting, but this thread seems like a good segue into the
> topic.
I tend to agree here, I think a lot of great work has been done to allow new 3rd
party CI system to come online.  Especially considering we have the
puppet-openstackci[1] module.

However, I would also like to see tripleO move more inline with our existing CI
tooling, if possible.  I know that wouldn't happen over night, but would at
least give better insight into how the CI is working.

Now that we have the infracloud too, it might be worth talking about doing the
samething with TripleO hardware, again if possible. There is likely corner cases
where it wouldn't work, but would be interesting to talk about it.

[1] https://github.com/openstack-infra/puppet-openstackci