<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Sep 28, 2018 at 3:02 PM Matt Riedemann <<a href="mailto:mriedemos@gmail.com">mriedemos@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 9/28/2018 3:12 PM, Clark Boylan wrote:<br>
> I was asked to write a followup to this as the long Zuul queues have persisted through this week. Largely because the situation from last week hasn't changed much. We were down the upgraded cloud region while we worked around a network configuration bug, then once that was addressed we ran into neutron port assignment and deletion issues. We think these are both fixed and we are running in this region again as of today.<br>
> <br>
> Other good news is our classification rate is up significantly. We can use that information to go through the top identified gate bugs:<br>
> <br>
> Network Connectivity issues to test nodes [2]. This is the current top of the list, but I think its impact is relatively small. What is happening here is jobs fail to connect to their test nodes early in the pre-run playbook and then fail. Zuul will rerun these jobs for us because they failed in the pre-run step. Prior to zuulv3 we had nodepool run a ready script before marking test nodes as ready, this script would've caught and filtered out these broken network nodes early. We now notice them late during the pre-run of a job.<br>
> <br>
> Pip fails to find distribution for package [3]. Earlier in the week we had the in region mirror fail in two different regions for unrelated errors. These mirrors were fixed and the only other hits for this bug come from Ara which tried to install the 'black' package on python3.5 but this package requires python>=3.6.<br>
> <br>
> yum, no more mirrors to try [4]. At first glance this appears to be an infrastructure issue because the mirror isn't serving content to yum. On further investigation it turned out to be a DNS resolution issue caused by the installation of designate in the tripleo jobs. Tripleo is aware of this issue and working to correct it.<br>
> <br>
> Stackviz failing on py3 [5]. This is a real bug in stackviz caused by subunit data being binary not utf8 encoded strings. I've written a fix for this problem athttps://<a href="http://review.openstack.org/606184" rel="noreferrer" target="_blank">review.openstack.org/606184</a>, but in doing so found that this was a known issue back in March and there was already a proposed fix,<a href="https://review.openstack.org/#/c/555388/3" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/555388/3</a>. It would be helpful if the QA team could care for this project and get a fix in. Otherwise, we should consider disabling stackviz on our tempest jobs (though the output from stackviz is often useful).<br>
> <br>
> There are other bugs being tracked by e-r. Some are bugs in the openstack software and I'm sure some are also bugs in the infrastructure. I have not yet had the time to work through the others though. It would be helpful if project teams could prioritize the debugging and fixing of these issues though.<br>
> <br>
> [2]<a href="http://status.openstack.org/elastic-recheck/gate.html#1793370" rel="noreferrer" target="_blank">http://status.openstack.org/elastic-recheck/gate.html#1793370</a><br>
> [3]<a href="http://status.openstack.org/elastic-recheck/gate.html#1449136" rel="noreferrer" target="_blank">http://status.openstack.org/elastic-recheck/gate.html#1449136</a><br>
> [4]<a href="http://status.openstack.org/elastic-recheck/gate.html#1708704" rel="noreferrer" target="_blank">http://status.openstack.org/elastic-recheck/gate.html#1708704</a><br>
> [5]<a href="http://status.openstack.org/elastic-recheck/gate.html#1758054" rel="noreferrer" target="_blank">http://status.openstack.org/elastic-recheck/gate.html#1758054</a><br>
<br>
Thanks for the update Clark.<br>
<br>
Another thing this week is the logstash indexing is behind by at least <br>
half a day. That's because workers were hitting OOM errors due to giant <br>
screen log files that aren't formatted properly so that we only index <br>
INFO+ level logs, and were instead trying to index the entire file, <br>
which some of which are 33MB *compressed*. So indexing of those <br>
identified problematic screen logs has been disabled:<br>
<br>
<a href="https://review.openstack.org/#/c/606197/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/606197/</a><br>
<br>
I've reported bugs against each related project.<br>
<br>
-- <br>
<br>
Thanks,<br>
<br>
Matt<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></blockquote><div><br></div><div><br></div><div><div>Greetings Clark and all,</div><div>The TripleO team would like to announce a significant change to the upstream CI the project has in place today. </div><div><br></div><div>TripleO can at times consume a large share of the compute resources [1] provided by the OpenStack upstream infrastructure team and OpenStack providers. The TripleO project has a large code base and high velocity of change which alone can tax the upstream CI system [3]. Additionally like other projects the issue is particularly acute when gate jobs are reset at a high rate. Unlike most other projects in OpenStack, TripleO uses multiple nodepool nodes in each job to more closely emulate customer like deployments. While using multiple nodes per job helps to uncover bugs that are not found in other projects, the resources used, the run time of each job, and usability have proven to be challenging. It has been a challenge to maintain job run times, quality and usability for TripleO and a challenge for the infra team to provide the required compute resources for the project.</div><div><br></div><div>A simplification of our upstream deployments to check and gate changes is in order. </div><div><br></div><div>The TripleO project has created a single node container based composable OpenStack deployment [2]. It is the projects intention to replace most of the TripleO upstream jobs with the Standalone deployment. We would like to reduce our multi-node usage to a total of two or three multinode jobs to handle a basic overcloud deployment, updates and upgrades[a]. Currently in master we are relying on multiple multi-node scenario jobs to test many of the OpenStack services in a single job. Our intention is to move these multinode scenario jobs to single node job(s) that tests a smaller subset of services. The goal of this would be target the specific areas of the TripleO code base that affect these services and only run those there. This would replace the existing 2-3 hour two node job(s) with single node job(s) for specific services that completes in about half the time. This unfortunately will reduce the overall coverage upstream but still allows us a basic smoke test of the supported OpenStack services and their deployment upstream.</div><div><br></div><div>Ideally projects other than TripleO would make use of the Standalone deployment to test their particular service with containers, upgrades or for various other reasons. Additional projects using this deployment would help ensure bugs are found quickly and resolved providing additional resilience to the upstream gate jobs. The TripleO team will begin review to scope out and create estimates for the above work starting on October 18 2018. One should expect to see updates on our progress posted to the list. Below are some details on the proposed changes.</div><div><br></div><div>Thank you all for your time and patience!</div><div><br></div><div>Performance improvements:</div><div> * Standalone jobs use half the nodes of multinode jobs</div><div> * The standalone job has an average run time of 60-80 minutes, about half the run time of our multinode jobs</div><div><br></div><div>Base TripleO Job Definitions (Stein onwards):</div><div>Multi-node jobs</div><div> * containers-multinode</div><div> * containers-multinode-updates</div><div> * containers-multinode-upgrades</div><div>Single node jobs</div><div> * undercloud</div><div> * undercloud-upgrade</div><div> * standalone</div><div><br></div><div>Jobs to be removed (Stein onwards):</div><div>Multi-node jobs[b]</div><div> * scenario001-multinode</div><div> * scenario002-multinode</div><div> * scenario003-multinode</div><div> * scenario004-multinode</div><div> * scenario006-mulitinode</div><div> * scenario007-multinode</div><div> * scenario008-multinode</div><div> * scenario009-multinode</div><div> * scenario010-multinode</div><div> * scenario011-multinode</div><div><br></div><div>Jobs that may need to be created to cover additional services[4] (Stein onwards):</div><div>Single node jobs[c]</div><div> * standalone-barbican</div><div> * standalone-ceph[d]</div><div> * standalone-designate</div><div> * standalone-manila</div><div> * standalone-octavia</div><div> * standalone-openshift</div><div> * standalone-sahara</div><div> * standalone-telemetry</div><div><br></div><div>[1] <a href="https://gist.github.com/notmyname/8bf3dbcb7195250eb76f2a1a8996fb00">https://gist.github.com/notmyname/8bf3dbcb7195250eb76f2a1a8996fb00</a></div><div>[2] <a href="https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/standalone.html">https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/standalone.html</a></div><div>[3] <a href="http://lists.openstack.org/pipermail/openstack-dev/2018-September/134867.html">http://lists.openstack.org/pipermail/openstack-dev/2018-September/134867.html</a></div><div>[4] <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/README.rst#service-testing-matrix">https://github.com/openstack/tripleo-heat-templates/blob/master/README.rst#service-testing-matrix</a> </div></div><div><br></div><div> </div></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="font-family:sans-serif"><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="font-size:14px">Wes Hayutin</span></p><p style="font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;text-transform:uppercase"><span style="color:rgb(0,0,0);font-size:10px">Associate MANAGER</span><br></p><p style="color:rgb(153,153,153);font-family:overpass,sans-serif;margin:0px;font-size:10px"><a href="https://www.redhat.com/" target="_blank" style="color:rgb(0,136,206);margin:0px">Red Hat <br><br></a></p><span style="color:rgb(153,153,153);font-family:overpass,sans-serif;font-size:10px;margin:0px"><p style="margin:0px"><span style="color:rgb(153,153,153);margin:0px;padding:0px"><a href="mailto:whayutin@redhat.com">whayutin@redhat.com</a> </span><span style="color:rgb(153,153,153)"> T: </span><a href="tel:+19197544114" target="_blank" style="color:rgb(0,136,206);margin:0px">+1919</a>4232509<span style="color:rgb(153,153,153)"> IRC: </span>weshay<br></p></span><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img width="90" height="auto" src="https://www.redhat.com/files/brand/email/sig-redhat.png"></a></td></tr></tbody></table></div><br style="font-family:sans-serif"><span style="font-family:sans-serif">View</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> </span><span style="font-family:sans-serif">calendar</span><span style="font-family:sans-serif"> and check </span><span style="font-family:sans-serif">my</span><span style="font-family:sans-serif"> availability for meetings </span><span id="inbox-inbox-inbox-docs-internal-guid-37a711e5-1f4c-b3c1-221b-f589048200c7"><a href="https://calendar.google.com/calendar/b/1/embed?src=whayutin@redhat.com&ctz=America/New_York"><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">HERE</span></a></span><br></div></div>