<div dir="ltr"><div><div><div><div><div>The patch that reduced the number of Tempest Scenarios we run in every job and also reduce the test run concurrency [0] was merged 13 days ago. Since, the situation (i.e the high number of false negative job results) has not improved significantly. We need to keep looking collectively at this.<br><br></div>There seems to be an agreement that we are hitting some memory limit. Several of our most frequent failures are memory related [1]. So we should either reduce our memory usage or ask for bigger VMs, with more than 8GB of RAM.<br><br></div>There was/is several attempts to reduce our memory usage, by reducing the Mysql memory consumption ([2] but quickly reverted [3]), reducing the number of Apache workers ([4], [5]), more apache2 tuning [6]. If you have any crazy idea to help in this regard, please help. This is high priority for the whole openstack project, because it's plaguing many projects.<br><br></div>We have some tools to investigate memory consumption, like some regular "dstat" output [7], a home-made memory tracker [8] and stackviz [9].<br><br></div>Best,<br></div>Jordan<br><div><div><div><div><div><br>[0]: <a href="https://review.openstack.org/#/c/439698/">https://review.openstack.org/#/c/439698/</a><br>[1]: <a href="http://status.openstack.org/elastic-recheck/gate.html">http://status.openstack.org/elastic-recheck/gate.html</a><br>[2] : <a href="https://review.openstack.org/#/c/438668/">https://review.openstack.org/#/c/438668/</a><br>[3]: <a href="https://review.openstack.org/#/c/446196/">https://review.openstack.org/#/c/446196/</a><br>[4]: <a href="https://review.openstack.org/#/c/426264/">https://review.openstack.org/#/c/426264/</a><br>[5]: <a href="https://review.openstack.org/#/c/445910/">https://review.openstack.org/#/c/445910/</a><br>[6]: <a href="https://review.openstack.org/#/c/446741/">https://review.openstack.org/#/c/446741/</a><br>[7]: <a href="http://logs.openstack.org/96/446196/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b5c362f/logs/dstat-csv_log.txt.gz">http://logs.openstack.org/96/446196/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b5c362f/logs/dstat-csv_log.txt.gz</a><br>[8]: <a href="http://logs.openstack.org/96/446196/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b5c362f/logs/screen-peakmem_tracker.txt.gz">http://logs.openstack.org/96/446196/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b5c362f/logs/screen-peakmem_tracker.txt.gz</a><br>[9] : <a href="http://logs.openstack.org/41/446741/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fa4d2e6/logs/stackviz/#/stdin/timeline">http://logs.openstack.org/41/446741/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fa4d2e6/logs/stackviz/#/stdin/timeline</a><br></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Mar 4, 2017 at 4:19 PM, Andrea Frittoli <span dir="ltr"><<a href="mailto:andrea.frittoli@gmail.com" target="_blank">andrea.frittoli@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Quick update on this, the change is now merged, so we now have a smaller number of scenario tests running serially after the api test run.</p>
<p dir="ltr">We'll monitor gate stability for the next week or so and decide whether further actions are required.</p>
<p dir="ltr">Please keep categorizing failures via elastic recheck as usual.</p>
<p dir="ltr">thank you</p><span class="HOEnZb"><font color="#888888">
<p dir="ltr">andrea</p></font></span><div class="HOEnZb"><div class="h5">
<br><div class="gmail_quote"><div dir="ltr">On Fri, 3 Mar 2017, 8:02 a.m. Ghanshyam Mann, <<a href="mailto:ghanshyammann@gmail.com" target="_blank">ghanshyammann@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-7662440994883868273gmail_msg"><div class="gmail_default m_-7662440994883868273gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Thanks. +1. i added my list in ethercalc.</div><div class="gmail_default m_-7662440994883868273gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br class="m_-7662440994883868273gmail_msg"></div><div class="gmail_default m_-7662440994883868273gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Left put scenario tests can be run on periodic and experimental job. IMO on both ( periodic and experimental) to monitor their status periodically as well as on particular patch if we need to. </div><div class="gmail_extra m_-7662440994883868273gmail_msg"><br class="m_-7662440994883868273gmail_msg" clear="all"><div class="m_-7662440994883868273gmail_msg"><div class="m_-7662440994883868273m_7817127192274301852gmail_signature m_-7662440994883868273gmail_msg" data-smartmail="gmail_signature"><div dir="ltr" class="m_-7662440994883868273gmail_msg"><div class="m_-7662440994883868273gmail_msg">-gmann</div></div></div></div>
<br class="m_-7662440994883868273gmail_msg"><div class="gmail_quote m_-7662440994883868273gmail_msg"></div></div></div><div dir="ltr" class="m_-7662440994883868273gmail_msg"><div class="gmail_extra m_-7662440994883868273gmail_msg"><div class="gmail_quote m_-7662440994883868273gmail_msg">On Fri, Mar 3, 2017 at 4:28 PM, Andrea Frittoli <span dir="ltr" class="m_-7662440994883868273gmail_msg"><<a href="mailto:andrea.frittoli@gmail.com" class="m_-7662440994883868273gmail_msg" target="_blank">andrea.frittoli@gmail.com</a>></span> wrote:<br class="m_-7662440994883868273gmail_msg"></div></div></div><div dir="ltr" class="m_-7662440994883868273gmail_msg"><div class="gmail_extra m_-7662440994883868273gmail_msg"><div class="gmail_quote m_-7662440994883868273gmail_msg"><blockquote class="gmail_quote m_-7662440994883868273gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-7662440994883868273gmail_msg">Hello folks,<div class="m_-7662440994883868273gmail_msg"><br class="m_-7662440994883868273gmail_msg"></div><div class="m_-7662440994883868273gmail_msg">we discussed a lot since the PTG about issues with gate stability; we need a stable and reliable gate to ensure smooth progress in Pike.</div><div class="m_-7662440994883868273gmail_msg"><br class="m_-7662440994883868273gmail_msg"></div><div class="m_-7662440994883868273gmail_msg">One of the issues that stands out is that most of the times during test runs our test VMs are under heavy load.</div><div class="m_-7662440994883868273gmail_msg">This can be the common cause behind several failures we've seen in the gate, so we agreed during the QA meeting yesterday [0] that we're going to try reducing the load and see whether that improves stability.<br class="m_-7662440994883868273gmail_msg"></div><div class="m_-7662440994883868273gmail_msg"><br class="m_-7662440994883868273gmail_msg"></div><div class="m_-7662440994883868273gmail_msg">Next steps are:<br class="m_-7662440994883868273gmail_msg"></div><div class="m_-7662440994883868273gmail_msg">- select a subset of scenario tests to be executed in the gate, based on [1], and run them serially only </div><div class="m_-7662440994883868273gmail_msg">- the patch for this is [2] and we will approve this by the end of the day</div><div class="m_-7662440994883868273gmail_msg">- we will monitor stability for a week - if needed we may reduce concurrency a bit on API tests as well, and identify "heavy" tests candidate for removal / refactor</div><div class="m_-7662440994883868273gmail_msg">- the QA team won't approve any new test (scenario or heavy resource consuming api) until gate stability is ensured </div><div class="m_-7662440994883868273gmail_msg"><br class="m_-7662440994883868273gmail_msg"></div><div class="m_-7662440994883868273gmail_msg">Thanks for your patience and collaboration!</div><div class="m_-7662440994883868273gmail_msg"><br class="m_-7662440994883868273gmail_msg"></div><div class="m_-7662440994883868273gmail_msg">Andrea</div><div class="m_-7662440994883868273gmail_msg"><br class="m_-7662440994883868273gmail_msg"></div><div class="m_-7662440994883868273gmail_msg">---</div><div class="m_-7662440994883868273gmail_msg">irc: andreaf</div><div class="m_-7662440994883868273gmail_msg"><br class="m_-7662440994883868273gmail_msg"></div><div class="m_-7662440994883868273gmail_msg">[0] <a href="http://eavesdrop.openstack.org/meetings/qa/2017/qa.2017-03-02-17.00.txt" class="m_-7662440994883868273gmail_msg" target="_blank">http://eavesdrop.<wbr>openstack.org/meetings/qa/<wbr>2017/qa.2017-03-02-17.00.txt</a></div><div class="m_-7662440994883868273gmail_msg">[1] <a href="https://ethercalc.openstack.org/nu56u2wrfb2b" class="m_-7662440994883868273gmail_msg" target="_blank">https://ethercalc.<wbr>openstack.org/nu56u2wrfb2b</a></div><div class="m_-7662440994883868273gmail_msg">[2] <span style="white-space:pre-wrap" class="m_-7662440994883868273gmail_msg"><a href="https://review.openstack.org/#/c/439698/" class="m_-7662440994883868273gmail_msg" target="_blank">https://review.openstack.<wbr>org/#/c/439698/</a> </span></div></div>
<br class="m_-7662440994883868273gmail_msg"></blockquote></div></div></div><div dir="ltr" class="m_-7662440994883868273gmail_msg"><div class="gmail_extra m_-7662440994883868273gmail_msg"><div class="gmail_quote m_-7662440994883868273gmail_msg"><blockquote class="gmail_quote m_-7662440994883868273gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">______________________________<wbr>______________________________<wbr>______________<br class="m_-7662440994883868273gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="m_-7662440994883868273gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="m_-7662440994883868273gmail_msg" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br class="m_-7662440994883868273gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="m_-7662440994883868273gmail_msg" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br class="m_-7662440994883868273gmail_msg">
<br class="m_-7662440994883868273gmail_msg"></blockquote></div><br class="m_-7662440994883868273gmail_msg"></div></div>
______________________________<wbr>______________________________<wbr>______________<br class="m_-7662440994883868273gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="m_-7662440994883868273gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="m_-7662440994883868273gmail_msg" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br class="m_-7662440994883868273gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="m_-7662440994883868273gmail_msg" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br class="m_-7662440994883868273gmail_msg">
</blockquote></div>
</div></div><br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>