<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 21, 2017 at 12:03 PM, Emilien Macchi <span dir="ltr"><<a href="mailto:emilien@redhat.com" target="_blank">emilien@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Mon, Mar 20, 2017 at 3:29 PM, Paul Belanger <<a href="mailto:pabelanger@redhat.com">pabelanger@redhat.com</a>> wrote:<br>
> On Sun, Mar 19, 2017 at 06:54:27PM +0200, Sagi Shnaidman wrote:<br>
>> Hi, Paul<br>
>> I would say that real worthwhile try starts from "normal" priority, because<br>
>> we want to run promotion jobs more *often*, not more *rarely* which happens<br>
>> with low priority.<br>
>> In addition the initial idea in the first mail was running them each after<br>
>> other almost, not once a day like it happens now or with "low" priority.<br>
>><br>
> As I've said, my main reluctance is is how the gate will react if we create a<br>
> new pipeline with the same priority as our check pipeline. I would much rather<br>
> since on caution, default to 'low', see how things react for a day / week /<br>
> month, then see what it would like like a normal. I want us to be caution about<br>
> adding a new pipeline, as it dynamically changes how our existing pipelines<br>
> function.<br>
><br>
> Further more, this is actually a capacity issue for tripleo-test-cloud-rh1,<br>
> there currently too many jobs running for the amount of hardware. If these jobs<br>
> were running on our donated clouds, we could get away with a low priority<br>
> periodic pipeline.<br>
<br>
</span>multinode jobs are running under donated clouds but as you know ovb not.<br>
We want to keep ovb jobs in our promotion pipeline because they bring<br>
high value to the tests (ironic, ipv6, ssl, probably more).<br>
<br>
Another alternative would be to reduce it to one ovb job (ironic with<br>
introspection + ipv6 + ssl at minimum) and use the 4 multinode jobs<br>
into the promotion pipeline -instead of the 3 ovb.<br></blockquote><div><br></div><div>I'm +1 on using one ovb jobs + 4 multinode jobs.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
current: 3 ovb jobs running every night<br>
proposal: 18 ovb jobs per day<br>
<br>
The addition will cost us 15 jobs into rh1 load. Would it be acceptable?<br>
<span class=""><br>
> Now, allow me to propose another solution.<br>
><br>
> RDO project has their own version of zuul, which has the ability to do periodic<br>
> pipelines. Since tripleo-test-cloud-rh2 is still around, and has OVB ability, I<br>
> would suggest configuring this promoting pipeline within RDO, as to not affect<br>
> the capacity of tripleo-test-cloud-rh1. This now means, you can continuously<br>
> enqueue jobs at a rate of 4 hours, priority shouldn't matter as you are the only<br>
> jobs running on tripleo-test-cloud-rh2, resulting in faster promotions.<br>
<br>
</span>Using RDO would also be an option. I'm just not sure about our<br>
available resources, maybe other can reply on this one.<br></blockquote><div><br></div><div>The purpose of the periodic jobs are two fold.</div><div>1. ensure the latest built packages work</div><div>2. ensure the tripleo check gates continue to work with out error </div><div><br></div><div>Running the promotion in review.rdoproject would not cover #2. The rdoproject jobs</div><div>would be configured in slightly different ways from upstream tripleo. Running the promotion</div><div>in ci.centos has the same issue.</div><div><br></div><div>Using tripleo-testcloud-rh2 I think is fine.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class=""><br>
> This also make sense, as packaging is done in RDO, and you are triggering Centos<br>
> CI things as a result.<br>
<br>
</span>Yes, it would make sense. Right now we have zero TripleO testing when<br>
doing changes in RDO packages (we only run packstack and puppet jobs<br>
which is not enough). Again, I think it's a problem of capacity here.<br></blockquote><div><br></div><div>We made a pass at getting multinode jobs running in RDO with tripleo. That was</div><div>initially not very successful and we chose to instead focus on upstream. We *do*</div><div>have it on our list to gate packages from RDO builds with tripleo. In the short term</div><div>that gate will use rdocloud, in the long term we'd also like to gate w/ multinode nodepool jobs in RDO.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thoughts?<br>
<div class="HOEnZb"><div class="h5"><br>
>> Thanks<br>
>><br>
>> On Wed, Mar 15, 2017 at 11:16 PM, Paul Belanger <<a href="mailto:pabelanger@redhat.com">pabelanger@redhat.com</a>><br>
>> wrote:<br>
>><br>
>> > On Wed, Mar 15, 2017 at 03:42:32PM -0500, Ben Nemec wrote:<br>
>> > ><br>
>> > ><br>
>> > > On 03/13/2017 02:29 PM, Sagi Shnaidman wrote:<br>
>> > > > Hi, all<br>
>> > > ><br>
>> > > > I submitted a change: <a href="https://review.openstack.org/#/c/443964/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/443964/</a><br>
>> > > > but seems like it reached a point which requires an additional<br>
>> > discussion.<br>
>> > > ><br>
>> > > > I had a few proposals, it's increasing period to 12 hours instead of 4<br>
>> > > > for start, and to leave it in regular periodic *low* precedence.<br>
>> > > > I think we can start from 12 hours period to see how it goes, although<br>
>> > I<br>
>> > > > don't think that 4 only jobs will increase load on OVB cloud, it's<br>
>> > > > completely negligible comparing to current OVB capacity and load.<br>
>> > > > But making its precedence as "low" IMHO completely removes any sense<br>
>> > > > from this pipeline to be, because we already run experimental-tripleo<br>
>> > > > pipeline which this priority and it could reach timeouts like 7-14<br>
>> > > > hours. So let's assume we ran periodic job, it's queued to run now 12 +<br>
>> > > > "low queue length" - about 20 and more hours. It's even worse than<br>
>> > usual<br>
>> > > > periodic job and definitely makes this change useless.<br>
>> > > > I'd like to notice as well that those periodic jobs unlike "usual"<br>
>> > > > periodic are used for repository promotion and their value are equal or<br>
>> > > > higher than check jobs, so it needs to run with "normal" or even "high"<br>
>> > > > precedence.<br>
>> > ><br>
>> > > Yeah, it makes no sense from an OVB perspective to add these as low<br>
>> > priority<br>
>> > > jobs. Once in a while we've managed to chew through the entire<br>
>> > experimental<br>
>> > > queue during the day, but with the containers job added it's very<br>
>> > unlikely<br>
>> > > that's going to happen anymore. Right now we have a 4.5 hour wait time<br>
>> > just<br>
>> > > for the check queue, then there's two hours of experimental jobs queued<br>
>> > up<br>
>> > > behind that. All of which means if we started a low priority periodic<br>
>> > job<br>
>> > > right now it probably wouldn't run until about midnight my time, which I<br>
>> > > think is when the regular periodic jobs run now.<br>
>> > ><br>
>> > Lets just give it a try? A 12 hour periodic job with low priority. There is<br>
>> > nothing saying we cannot iterate on this after a few days / weeks / months.<br>
>> ><br>
>> > > ><br>
>> > > > Thanks<br>
>> > > ><br>
>> > > ><br>
>> > > > On Thu, Mar 9, 2017 at 10:06 PM, Wesley Hayutin <<a href="mailto:whayutin@redhat.com">whayutin@redhat.com</a><br>
>> > > > <mailto:<a href="mailto:whayutin@redhat.com">whayutin@redhat.com</a>>> wrote:<br>
>> > > ><br>
>> > > ><br>
>> > > ><br>
>> > > > On Wed, Mar 8, 2017 at 1:29 PM, Jeremy Stanley <<a href="mailto:fungi@yuggoth.org">fungi@yuggoth.org</a><br>
>> > > > <mailto:<a href="mailto:fungi@yuggoth.org">fungi@yuggoth.org</a>>> wrote:<br>
>> > > ><br>
>> > > > On 2017-03-07 10:12:58 -0500 (-0500), Wesley Hayutin wrote:<br>
>> > > > > The TripleO team would like to initiate a conversation about<br>
>> > the<br>
>> > > > > possibility of creating a new pipeline in Openstack Infra to<br>
>> > allow<br>
>> > > > > a set of jobs to run periodically every four hours<br>
>> > > > [...]<br>
>> > > ><br>
>> > > > The request doesn't strike me as contentious/controversial.<br>
>> > Why not<br>
>> > > > just propose your addition to the zuul/layout.yaml file in the<br>
>> > > > openstack-infra/project-config repo and hash out any resulting<br>
>> > > > concerns via code review?<br>
>> > > > --<br>
>> > > > Jeremy Stanley<br>
>> > > ><br>
>> > > ><br>
>> > > > Sounds good to me.<br>
>> > > > We thought it would be nice to walk through it in an email first :)<br>
>> > > ><br>
>> > > > Thanks<br>
>> > > ><br>
>> > > ><br>
>> > > > ______________________________<wbr>______________________________<br>
>> > ______________<br>
>> > > > OpenStack Development Mailing List (not for usage questions)<br>
>> > > > Unsubscribe:<br>
>> > > > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> > > > <<a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject</a>:<br>
>> > unsubscribe><br>
>> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/</a><br>
>> > openstack-dev <<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/</a><br>
>> > openstack-dev><br>
>> > > ><br>
>> > > ><br>
>> > > ><br>
>> > > > ______________________________<wbr>______________________________<br>
>> > ______________<br>
>> > > > OpenStack Development Mailing List (not for usage questions)<br>
>> > > > Unsubscribe:<br>
>> > > > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> > > > <<a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject</a>:<br>
>> > unsubscribe><br>
>> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>> > > > <<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>> > ><br>
>> > > ><br>
>> > > ><br>
>> > > ><br>
>> > > ><br>
>> > > > --<br>
>> > > > Best regards<br>
>> > > > Sagi Shnaidman<br>
>> > > ><br>
>> > > ><br>
>> > > > ______________________________<wbr>______________________________<br>
>> > ______________<br>
>> > > > OpenStack Development Mailing List (not for usage questions)<br>
>> > > > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject</a>:<br>
>> > unsubscribe<br>
>> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>> > > ><br>
>> > ><br>
>> > > ______________________________<wbr>______________________________<br>
>> > ______________<br>
>> > > OpenStack Development Mailing List (not for usage questions)<br>
>> > > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject</a>:<br>
>> > unsubscribe<br>
>> > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>> ><br>
>> > ______________________________<wbr>______________________________<wbr>______________<br>
>> > OpenStack Development Mailing List (not for usage questions)<br>
>> > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
>> ><br>
>><br>
>><br>
>><br>
>> --<br>
>> Best regards<br>
>> Sagi Shnaidman<br>
><br>
>> ______________________________<wbr>______________________________<wbr>______________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
><br>
><br>
> ______________________________<wbr>______________________________<wbr>______________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Emilien Macchi<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>