[OpenStack-Infra] Deep pipelining with zuul

Jerry Zhao xyzjerry at gmail.com
Wed Feb 24 17:19:49 UTC 2016


I remember that there is a delay for used nodes on nodepool to be 
cleaned, without zuul marking it used, new jobs could be assigned on the 
used node, which would be deleted later so new job would sure fail.

On 02/24/2016 08:58 AM, Doug Wiegley wrote:
> I’m doing something similar, but am just specifying nodepool nodes on the child jobs, with artifact copying in Jenkins. I don’t think you need JClouds at that point. Jobs that use nodepool nodes don’t have to originate with zuul. Or am I missing your restriction?
>
> doug
>
>
>> On Feb 24, 2016, at 10:12 AM, Andrew Grimberg <agrimberg at linuxfoundation.org> wrote:
>>
>> The problem with that is that we would then continue to have to rely on
>> JClouds to do slave management and we have two driving reasons for this
>> desire / need to switch to Zuul and Nodepool (or something similar)
>>
>> 1) Recent versions of the Gerrit Trigger plugin for Jenkins forget, or
>> do not update current triggers on JJB updates without someone either
>> manually clicking save in the job configuration UI or doing a full
>> restart of Jenkins. This is a bad thing for us, we're at nearly 2k jobs
>> managed via JJB many of which are generated by templates and a single
>> change can affect more than 100 jobs
>>
>> 2) JClouds is a single-threaded nightmare inside Jenkins and can easily
>> cause it to hang creation of new slaves, deletion of old slaves and
>> cause the Jenkins queue itself to not start jobs because a single job
>> got hung and JClouds gets confused. This along with it completely
>> forgetting about nodes that it started but they failed to connect for
>> some reason.
>>
>> So, while that sounds like a viable solution, it's not :(
>>
>> -Andy-
>>
>> On 02/24/2016 08:04 AM, Doug Wiegley wrote:
>>> Even if zuul doesn’t support the flow you want, you can also still
>>> use matrix, multijob, and downstream jobs on Jenkins itself to simulate the
>>> same behavior, with zuul just monitoring the parent job. Mix and match
>>> until you get the flow that you need.
>>>
>>> doug
>>>
>>>
>>>> On Feb 24, 2016, at 9:56 AM, Andrew Grimberg <agrimberg at linuxfoundation.org> wrote:
>>>>
>>>> Greetings,
>>>>
>>>> We're working on transitioning our Jenkins system off of Gerrit Trigger
>>>> and JClouds to Zuul and Nodepool and have run into a bit of a snag.
>>>>
>>>> We have some rather deep pipeline dependencies with our current
>>>> configurations where one job produces information and possibly artifacts
>>>> that are then passed on to one or more jobs downstream.
>>>>
>>>> Is this sort of configuration even possible with Zuul? I can't seem to
>>>> find anything in the documentation that would explain how to do this. If
>>>> it's not currently possible, what would need to be done to extend Zuul
>>>> to support something like this?
>>>>
>>>> As a simple example we're looking to something like this (please excuse
>>>> the bad ascii art)
>>>>
>>>>                                / Verify1 \
>>>> Gerrit event -- Zuul trigger --   Verify2  -- PipelineJob1 -- Response
>>>>                                \ Verify3 /
>>>>
>>>>
>>>> What the documentation is leading me to believe is that I can't do that
>>>> extra PipelineJob1 (or multiple in serial / parallel)
>>>>
>>>> -Andy-
>>>>
>>>> -- 
>>>> Andrew J Grimberg
>>>> Systems Administrator
>>>> The Linux Foundation
>>>>
>>>> _______________________________________________
>>>> OpenStack-Infra mailing list
>>>> OpenStack-Infra at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>> -- 
>> Andrew J Grimberg
>> Systems Administrator
>> The Linux Foundation
>>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra




More information about the OpenStack-Infra mailing list