Open Stack

On 2016-03-10 09:50:03 -0500 (-0500), Emilien Macchi wrote:
[...]
> OpenStack Infra provides an easy way to plug CI systems and some
> CIs (Nova, Neutron, Cinder, etc) already gate on some third party
> systems. I was wondering if we would not be interested to
> investigate this area and maybe ask to our third party drivers
> contributors (Bigswitch, Nuage, Midonet, Cisco, Netapp, etc) to
> run on their own hardware TripleO CI jobs running their specific
> environment to enable the drivers. This CI would be plugged to
> TripleO CI, and would provide awesome feedback.
[...]

It's also worth broadening the discussion to reassess whether the
existing TripleO CI should itself follow our third-party integration
model instead of the current implementation relying on our main
community Zuul/Nodepool/Jenkins servers. When this was first
implemented, there was a promise of adding more regions for
robustness and of being able to use the surplus resources maintained
in the TripleO CI clouds to augment our generic CI workload. It's
been years now and these things have not really come to pass; if
anything, that system and its operators are still struggling to keep
a single region up and operational and providing enough resources to
handle the current TripleO test load.

The majority of unplanned whole-provider outages we've experienced
in Nodepool have been from the TripleO cloud going completely
offline (sometimes for a week or more straight), by far the
longest-running jobs we have are running there (which substantially
hampers our ability to do things like gracefully restart our Jenkins
masters without aborting running jobs), and ultimately the benefits
to TripleO for this model are very minimal anyway (different
pipelines means the jobs aren't effectively even voting, much less
gating).

I'm not trying to slam the TripleO cloud operators, I think they're
doing an amazing job given the limitations they're working under and
much of their work has provided inspiration for our Infra-Cloud
project too. They're helpful and responsive and a joy to collaborate
with, but ultimately I think TripleO might actually realize more
benefit from adding a Zuul/Nodepool/Jenkins of their own to this
(we've massively streamlined the Puppet for maintaining these
recently and have very thorough deployment and operational
documentation) rather than dealing with the issues which arise from
being half-integrated into one they don't control.

I've been meaning to bring that up for discussion for a while, just
keep forgetting, but this thread seems like a good segue into the
topic.
-- 
Jeremy Stanley