Open Stack

On 17 March 2016 at 16:59, Ben Nemec <openstack at nemebean.com> wrote:
> On 03/10/2016 05:24 PM, Jeremy Stanley wrote:
>> On 2016-03-10 16:09:44 -0500 (-0500), Dan Prince wrote:
>>> This seems to be the week people want to pile it on TripleO. Talking
>>> about upstream is great but I suppose I'd rather debate major changes
>>> after we branch Mitaka. :/
>> [...]
>>
>> I didn't mean to pile on TripleO, nor did I intend to imply this was
>> something which should happen ASAP (or even necessarily at all), but
>> I do want to better understand what actual benefit is currently
>> derived from this implementation vs. a more typical third-party CI
>> (which lots of projects are doing when they find their testing needs
>> are not met by the constraints of our generic test infrastructure).
>>
>>> With regards to Jenkins restarts I think it is understood that our job
>>> times are long. How often do you find infra needs to restart Jenkins?
>>
>> We're restarting all 8 of our production Jenkins masters weekly at a
>> minimum, but generally more often when things are busy (2-3 times a
>> week). For many months we've been struggling with a thread leak for
>> which their development team has not seen as a priority to even
>> triage our bug report effectively. At this point I think we've
>> mostly given up on expecting it to be solved by anything other than
>> our upcoming migration off of Jenkins, but that's another topic
>> altogether.
>>
>>> And regardless of that what if we just said we didn't mind the
>>> destructiveness of losing a few jobs now and then (until our job
>>> times are under the line... say 1.5 hours or so). To be clear I'd
>>> be fine with infra pulling the rug on running jobs if this is the
>>> root cause of the long running jobs in TripleO.
>>
>> For manual Jenkins restarts this is probably doable (if additional
>> hassle), but I don't know whether that's something we can easily
>> shoehorn into our orchestrated/automated restarts.
>>
>>> I think the "benefits are minimal" is bit of an overstatement. The
>>> initial vision for TripleO CI stands and I would still like to see
>>> individual projects entertain the option to use us in their gates.
>> [...]
>>
>> This is what I'd like to delve deeper into. The current
>> implementation isn't providing you with any mechanism to prevent
>> changes which fail jobs running in the tripleo-test cloud from
>> merging to your repos, is it? You're still having to manually
>> inspect the job results posted by it? How is that particularly
>> different from relying on third-party CI integration?
>>
>> As for other projects making use of the same jobs, right now the
>> only convenience I'm aware of is that they can add check-tripleo
>> pipeline jobs in our Zuul layout file instead of having you add it
>> to yours (which could itself reside in a Git repo under your
>> control, giving you even more flexibility over those choices). In
>> fact, with a third-party CI using its own separate Gerrit account,
>> you would be able to leave clear -1/+1 votes on check results which
>> is not possible with the present solution.
>>
>> So anyway, I'm not saying that I definitely believe the third-party
>> CI route will be better for TripleO, but I'm not (yet) clear on what
>> tangible benefit you're receiving now that you lose by switching to
>> that model.
>>
>
> FWIW, I think third-party CI probably makes sense for TripleO.
> Practically speaking we are third-party CI right now - we run our own
> independent hardware infrastructure, we aren't multi-region, and we
> can't leave a vote on changes.  Since the first two aren't likely to
> change any time soon (although I believe it's still a long-term goal to
> get to a place where we can run in regular infra and just contribute our
> existing CI hardware to the general infra pool, but that's still a long
> way off), and moving to actual third-party CI would get us the ability
> to vote, I think it's worth pursuing.
>
> As an added bit of fun, we have a forced move of our CI hardware coming
> up in the relatively near future, and if we don't want to have multiple
> days (and possibly more, depending on how the move goes) of TripleO CI
> outage we're probably going to need to stand up a new environment in
> parallel anyway.  If we're doing that it might make sense to try hooking
> it in through the third-party infra instead of the way we do it today.
> Hopefully that would allow us to work out the kinks before the old
> environment goes away.
>
> Anyway, I'm sure we'll need a bunch more discussion about this, but I
> wanted to chime in with my two cents.

We need to answer this question soon, I'm currently working on the CI
parts that we need in order of move to OVB[1] and was assuming we
would be maintaining the status quo. What we end up doing would look
very different if we move to 3rd party CI, if using 3rd party CI we
can simply start a vanilla centos instance at use it as an undercloud.
It can then create its own baremetal testenv. If we remain inside the
infra umbrella we'll have a seperate jenkins slave for each job that
has to talk to a cetral broker to get a test env (including a seperate
undercloud).

Doing this in 3rd party ci I think simplyfies things because we'll no
longer need a public cloud and as a result the security measures
requeired to avoid putting cloud credentials on the jenkins slaves
wont be needed.

The down side to this is that we would (as I see it) be taking a step
further away from ever being in the gate, tripleo ci hasn't for a long
time gotten closer to getting into the gate, a large part of the
reason I believe is because deploying trunk with tripleo isn't working
a lot, the vast majority of these are because we're not in the gate,
so we have a chicken and egg problem. We also have other reasons we're
not in the gate, capacity for one, so maybe switching to 3rd party is
ok at least until we address the capacity and can reassess.

[1] - https://review.openstack.org/#/c/295243/


>
> -Ben
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev