<div dir="ltr"><div><div>Paul,<br></div>if we run 750 ovb jobs per day, than adding 12 more will be less than 2% increase. I don't believe it will be a serious issue.<br><br></div>Thanks<br><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 21, 2017 at 7:34 PM, Paul Belanger <span dir="ltr"><<a href="mailto:pabelanger@redhat.com" target="_blank">pabelanger@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tue, Mar 21, 2017 at 12:40:39PM -0400, Wesley Hayutin wrote:<br>
> On Tue, Mar 21, 2017 at 12:03 PM, Emilien Macchi <<a href="mailto:emilien@redhat.com">emilien@redhat.com</a>> wrote:<br>
><br>
> > On Mon, Mar 20, 2017 at 3:29 PM, Paul Belanger <<a href="mailto:pabelanger@redhat.com">pabelanger@redhat.com</a>><br>
> > wrote:<br>
> > > On Sun, Mar 19, 2017 at 06:54:27PM +0200, Sagi Shnaidman wrote:<br>
> > >> Hi, Paul<br>
> > >> I would say that real worthwhile try starts from "normal" priority,<br>
> > because<br>
> > >> we want to run promotion jobs more *often*, not more *rarely* which<br>
> > happens<br>
> > >> with low priority.<br>
> > >> In addition the initial idea in the first mail was running them each<br>
> > after<br>
> > >> other almost, not once a day like it happens now or with "low" priority.<br>
> > >><br>
> > > As I've said, my main reluctance is is how the gate will react if we<br>
> > create a<br>
> > > new pipeline with the same priority as our check pipeline. I would much<br>
> > rather<br>
> > > since on caution, default to 'low', see how things react for a day /<br>
> > week /<br>
> > > month, then see what it would like like a normal. I want us to be<br>
> > caution about<br>
> > > adding a new pipeline, as it dynamically changes how our existing<br>
> > pipelines<br>
> > > function.<br>
> > ><br>
> > > Further more, this is actually a capacity issue for<br>
> > tripleo-test-cloud-rh1,<br>
> > > there currently too many jobs running for the amount of hardware. If<br>
> > these jobs<br>
> > > were running on our donated clouds, we could get away with a low priority<br>
> > > periodic pipeline.<br>
> ><br>
> > multinode jobs are running under donated clouds but as you know ovb not.<br>
> > We want to keep ovb jobs in our promotion pipeline because they bring<br>
> > high value to the tests (ironic, ipv6, ssl, probably more).<br>
> ><br>
> > Another alternative would be to reduce it to one ovb job (ironic with<br>
> > introspection + ipv6 + ssl at minimum) and use the 4 multinode jobs<br>
> > into the promotion pipeline -instead of the 3 ovb.<br>
> ><br>
><br>
> I'm +1 on using one ovb jobs + 4 multinode jobs.<br>
><br>
><br>
> ><br>
> > current: 3 ovb jobs running every night<br>
> > proposal: 18 ovb jobs per day<br>
> ><br>
> > The addition will cost us 15 jobs into rh1 load. Would it be acceptable?<br>
> ><br>
> > > Now, allow me to propose another solution.<br>
> > ><br>
> > > RDO project has their own version of zuul, which has the ability to do<br>
> > periodic<br>
> > > pipelines. Since tripleo-test-cloud-rh2 is still around, and has OVB<br>
> > ability, I<br>
> > > would suggest configuring this promoting pipeline within RDO, as to not<br>
> > affect<br>
> > > the capacity of tripleo-test-cloud-rh1. This now means, you can<br>
> > continuously<br>
> > > enqueue jobs at a rate of 4 hours, priority shouldn't matter as you are<br>
> > the only<br>
> > > jobs running on tripleo-test-cloud-rh2, resulting in faster promotions.<br>
> ><br>
> > Using RDO would also be an option. I'm just not sure about our<br>
> > available resources, maybe other can reply on this one.<br>
> ><br>
><br>
> The purpose of the periodic jobs are two fold.<br>
> 1. ensure the latest built packages work<br>
> 2. ensure the tripleo check gates continue to work with out error<br>
><br>
> Running the promotion in review.rdoproject would not cover #2. The<br>
> rdoproject jobs<br>
> would be configured in slightly different ways from upstream tripleo.<br>
> Running the promotion<br>
> in ci.centos has the same issue.<br>
><br>
</div></div>Right, there is some leg work to use the images produced by opentack-infra in<br>
RDO, but that is straightforward. It would be the same build process that a 3rd<br>
party CI system does. It would be a matter of copying nodepool.yaml from<br>
openstack-infra/project-<wbr>config, and (this is harder) using nodepool-builder to<br>
build the images. Today RDO does snapshot images.<br>
<div class="HOEnZb"><div class="h5"><br>
> Using tripleo-testcloud-rh2 I think is fine.<br>
><br>
><br>
> ><br>
> > > This also make sense, as packaging is done in RDO, and you are<br>
> > triggering Centos<br>
> > > CI things as a result.<br>
> ><br>
> > Yes, it would make sense. Right now we have zero TripleO testing when<br>
> > doing changes in RDO packages (we only run packstack and puppet jobs<br>
> > which is not enough). Again, I think it's a problem of capacity here.<br>
> ><br>
><br>
> We made a pass at getting multinode jobs running in RDO with tripleo. That<br>
> was<br>
> initially not very successful and we chose to instead focus on upstream.<br>
> We *do*<br>
> have it on our list to gate packages from RDO builds with tripleo. In the<br>
> short term<br>
> that gate will use rdocloud, in the long term we'd also like to gate w/<br>
> multinode nodepool jobs in RDO.<br>
><br>
><br>
><br>
> ><br>
> > Thoughts?<br>
> ><br>
> > >> Thanks<br>
> > >><br>
> > >> On Wed, Mar 15, 2017 at 11:16 PM, Paul Belanger <<a href="mailto:pabelanger@redhat.com">pabelanger@redhat.com</a>><br>
> > >> wrote:<br>
> > >><br>
> > >> > On Wed, Mar 15, 2017 at 03:42:32PM -0500, Ben Nemec wrote:<br>
> > >> > ><br>
> > >> > ><br>
> > >> > > On 03/13/2017 02:29 PM, Sagi Shnaidman wrote:<br>
> > >> > > > Hi, all<br>
> > >> > > ><br>
> > >> > > > I submitted a change: <a href="https://review.openstack.org/#/c/443964/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/443964/</a><br>
> > >> > > > but seems like it reached a point which requires an additional<br>
> > >> > discussion.<br>
> > >> > > ><br>
> > >> > > > I had a few proposals, it's increasing period to 12 hours instead<br>
> > of 4<br>
> > >> > > > for start, and to leave it in regular periodic *low* precedence.<br>
> > >> > > > I think we can start from 12 hours period to see how it goes,<br>
> > although<br>
> > >> > I<br>
> > >> > > > don't think that 4 only jobs will increase load on OVB cloud, it's<br>
> > >> > > > completely negligible comparing to current OVB capacity and load.<br>
> > >> > > > But making its precedence as "low" IMHO completely removes any<br>
> > sense<br>
> > >> > > > from this pipeline to be, because we already run<br>
> > experimental-tripleo<br>
> > >> > > > pipeline which this priority and it could reach timeouts like 7-14<br>
> > >> > > > hours. So let's assume we ran periodic job, it's queued to run<br>
> > now 12 +<br>
> > >> > > > "low queue length" - about 20 and more hours. It's even worse than<br>
> > >> > usual<br>
> > >> > > > periodic job and definitely makes this change useless.<br>
> > >> > > > I'd like to notice as well that those periodic jobs unlike "usual"<br>
> > >> > > > periodic are used for repository promotion and their value are<br>
> > equal or<br>
> > >> > > > higher than check jobs, so it needs to run with "normal" or even<br>
> > "high"<br>
> > >> > > > precedence.<br>
> > >> > ><br>
> > >> > > Yeah, it makes no sense from an OVB perspective to add these as low<br>
> > >> > priority<br>
> > >> > > jobs. Once in a while we've managed to chew through the entire<br>
> > >> > experimental<br>
> > >> > > queue during the day, but with the containers job added it's very<br>
> > >> > unlikely<br>
> > >> > > that's going to happen anymore. Right now we have a 4.5 hour wait<br>
> > time<br>
> > >> > just<br>
> > >> > > for the check queue, then there's two hours of experimental jobs<br>
> > queued<br>
> > >> > up<br>
> > >> > > behind that. All of which means if we started a low priority<br>
> > periodic<br>
> > >> > job<br>
> > >> > > right now it probably wouldn't run until about midnight my time,<br>
> > which I<br>
> > >> > > think is when the regular periodic jobs run now.<br>
> > >> > ><br>
> > >> > Lets just give it a try? A 12 hour periodic job with low priority.<br>
> > There is<br>
> > >> > nothing saying we cannot iterate on this after a few days / weeks /<br>
> > months.<br>
> > >> ><br>
> > >> > > ><br>
> > >> > > > Thanks<br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > > On Thu, Mar 9, 2017 at 10:06 PM, Wesley Hayutin <<br>
> > <a href="mailto:whayutin@redhat.com">whayutin@redhat.com</a><br>
> > >> > > > <mailto:<a href="mailto:whayutin@redhat.com">whayutin@redhat.com</a>>> wrote:<br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > > On Wed, Mar 8, 2017 at 1:29 PM, Jeremy Stanley <<br>
> > <a href="mailto:fungi@yuggoth.org">fungi@yuggoth.org</a><br>
> > >> > > > <mailto:<a href="mailto:fungi@yuggoth.org">fungi@yuggoth.org</a>>> wrote:<br>
> > >> > > ><br>
> > >> > > > On 2017-03-07 10:12:58 -0500 (-0500), Wesley Hayutin<br>
> > wrote:<br>
> > >> > > > > The TripleO team would like to initiate a conversation<br>
> > about<br>
> > >> > the<br>
> > >> > > > > possibility of creating a new pipeline in Openstack<br>
> > Infra to<br>
> > >> > allow<br>
> > >> > > > > a set of jobs to run periodically every four hours<br>
> > >> > > > [...]<br>
> > >> > > ><br>
> > >> > > > The request doesn't strike me as<br>
> > contentious/controversial.<br>
> > >> > Why not<br>
> > >> > > > just propose your addition to the zuul/layout.yaml file<br>
> > in the<br>
> > >> > > > openstack-infra/project-config repo and hash out any<br>
> > resulting<br>
> > >> > > > concerns via code review?<br>
> > >> > > > --<br>
> > >> > > > Jeremy Stanley<br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > > Sounds good to me.<br>
> > >> > > > We thought it would be nice to walk through it in an email<br>
> > first :)<br>
> > >> > > ><br>
> > >> > > > Thanks<br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > > ______________________________<br>
> > ______________________________<br>
> > >> > ______________<br>
> > >> > > > OpenStack Development Mailing List (not for usage<br>
> > questions)<br>
> > >> > > > Unsubscribe:<br>
> > >> > > > <a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject</a>:<br>
> > unsubscribe<br>
> > >> > > > <<a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject</a><br>
> > :<br>
> > >> > unsubscribe><br>
> > >> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/</a><br>
> > >> > openstack-dev <<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/</a><br>
> > >> > openstack-dev><br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > > ______________________________<wbr>______________________________<br>
> > >> > ______________<br>
> > >> > > > OpenStack Development Mailing List (not for usage questions)<br>
> > >> > > > Unsubscribe:<br>
> > >> > > > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
> > >> > > > <<a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject</a>:<br>
> > >> > unsubscribe><br>
> > >> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/</a><br>
> > openstack-dev<br>
> > >> > > > <<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/</a><br>
> > openstack-dev<br>
> > >> > ><br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > ><br>
</div></div><div class="HOEnZb"><div class="h5">> > >> > > > --<br>
> > >> > > > Best regards<br>
> > >> > > > Sagi Shnaidman<br>
> > >> > > ><br>
> > >> > > ><br>
> > >> > > > ______________________________<wbr>______________________________<br>
> > >> > ______________<br>
> > >> > > > OpenStack Development Mailing List (not for usage questions)<br>
> > >> > > > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject</a>:<br>
> > >> > unsubscribe<br>
> > >> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
> > >> > > ><br>
> > >> > ><br>
> > >> > > ______________________________<wbr>______________________________<br>
> > >> > ______________<br>
> > >> > > OpenStack Development Mailing List (not for usage questions)<br>
> > >> > > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject</a>:<br>
> > >> > unsubscribe<br>
> > >> > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
> > >> ><br>
> > >> > ______________________________<wbr>______________________________<br>
> > ______________<br>
> > >> > OpenStack Development Mailing List (not for usage questions)<br>
> > >> > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject</a>:<br>
> > unsubscribe<br>
> > >> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
> > >> ><br>
> > >><br>
> > >><br>
> > >><br>
> > >> --<br>
> > >> Best regards<br>
> > >> Sagi Shnaidman<br>
> > ><br>
> > >> ______________________________<wbr>______________________________<br>
> > ______________<br>
> > >> OpenStack Development Mailing List (not for usage questions)<br>
> > >> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject</a>:<br>
> > unsubscribe<br>
> > >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
> > ><br>
> > ><br>
> > > ______________________________<wbr>______________________________<br>
> > ______________<br>
> > > OpenStack Development Mailing List (not for usage questions)<br>
> > > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject</a>:<br>
> > unsubscribe<br>
> > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
> ><br>
> ><br>
> ><br>
</div></div><div class="HOEnZb"><div class="h5">> > --<br>
> > Emilien Macchi<br>
> ><br>
> > ______________________________<wbr>______________________________<wbr>______________<br>
> > OpenStack Development Mailing List (not for usage questions)<br>
> > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
> ><br>
<br>
> ______________________________<wbr>______________________________<wbr>______________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Best regards<br></div>Sagi Shnaidman<br></div></div>
</div></div></div></div></div>