<div><div>I had some patches to collect more stats about mlocks here: <a href="https://review.openstack.org/#/q/topic:collect-mlock-stats-in-gate">https://review.openstack.org/#/q/topic:collect-mlock-stats-in-gate</a> but they need reviews.</div></div><div><br></div><div>Ihar</div><div><br><div class="gmail_quote"><div>On Fri, Mar 17, 2017 at 5:28 AM Jordan Pittier <<a href="mailto:jordan.pittier@scality.com">jordan.pittier@scality.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg">The patch that reduced the number of Tempest Scenarios we run in every job and also reduce the test run concurrency [0] was merged 13 days ago. Since, the situation (i.e the high number of false negative job results) has not improved significantly. We need to keep looking collectively at this.<br class="gmail_msg"><br class="gmail_msg"></div>There seems to be an agreement that we are hitting some memory limit. Several of our most frequent failures are memory related [1]. So we should either reduce our memory usage or ask for bigger VMs, with more than 8GB of RAM.<br class="gmail_msg"><br class="gmail_msg"></div>There was/is several attempts to reduce our memory usage, by reducing the Mysql memory consumption ([2] but quickly reverted [3]), reducing the number of Apache workers ([4], [5]), more apache2 tuning [6]. If you have any crazy idea to help in this regard, please help. This is high priority for the whole openstack project, because it's plaguing many projects.<br class="gmail_msg"><br class="gmail_msg"></div>We have some tools to investigate memory consumption, like some regular "dstat" output [7], a home-made memory tracker [8] and stackviz [9].<br class="gmail_msg"><br class="gmail_msg"></div>Best,<br class="gmail_msg"></div>Jordan<br class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg">[0]: <a href="https://review.openstack.org/#/c/439698/" class="gmail_msg" target="_blank">https://review.openstack.org/#/c/439698/</a><br class="gmail_msg">[1]: <a href="http://status.openstack.org/elastic-recheck/gate.html" class="gmail_msg" target="_blank">http://status.openstack.org/elastic-recheck/gate.html</a><br class="gmail_msg">[2] : <a href="https://review.openstack.org/#/c/438668/" class="gmail_msg" target="_blank">https://review.openstack.org/#/c/438668/</a><br class="gmail_msg">[3]: <a href="https://review.openstack.org/#/c/446196/" class="gmail_msg" target="_blank">https://review.openstack.org/#/c/446196/</a><br class="gmail_msg">[4]: <a href="https://review.openstack.org/#/c/426264/" class="gmail_msg" target="_blank">https://review.openstack.org/#/c/426264/</a><br class="gmail_msg">[5]: <a href="https://review.openstack.org/#/c/445910/" class="gmail_msg" target="_blank">https://review.openstack.org/#/c/445910/</a><br class="gmail_msg">[6]: <a href="https://review.openstack.org/#/c/446741/" class="gmail_msg" target="_blank">https://review.openstack.org/#/c/446741/</a><br class="gmail_msg">[7]: <a href="http://logs.openstack.org/96/446196/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b5c362f/logs/dstat-csv_log.txt.gz" class="gmail_msg" target="_blank">http://logs.openstack.org/96/446196/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b5c362f/logs/dstat-csv_log.txt.gz</a><br class="gmail_msg">[8]: <a href="http://logs.openstack.org/96/446196/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b5c362f/logs/screen-peakmem_tracker.txt.gz" class="gmail_msg" target="_blank">http://logs.openstack.org/96/446196/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b5c362f/logs/screen-peakmem_tracker.txt.gz</a><br class="gmail_msg">[9] : <a href="http://logs.openstack.org/41/446741/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fa4d2e6/logs/stackviz/#/stdin/timeline" class="gmail_msg" target="_blank">http://logs.openstack.org/41/446741/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fa4d2e6/logs/stackviz/#/stdin/timeline</a><br class="gmail_msg"></div></div></div></div></div></div><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Sat, Mar 4, 2017 at 4:19 PM, Andrea Frittoli <span class="gmail_msg"><<a href="mailto:andrea.frittoli@gmail.com" class="gmail_msg" target="_blank">andrea.frittoli@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p class="gmail_msg">Quick update on this, the change is now merged, so we now have a smaller number of scenario tests running serially after the api test run.</p>
<p class="gmail_msg">We'll monitor gate stability for the next week or so and decide whether further actions are required.</p>
<p class="gmail_msg">Please keep categorizing failures via elastic recheck as usual.</p>
<p class="gmail_msg">thank you</p><span class="m_6566325548703639844HOEnZb gmail_msg"><font color="#888888" class="gmail_msg">
<p class="gmail_msg">andrea</p></font></span><div class="m_6566325548703639844HOEnZb gmail_msg"><div class="m_6566325548703639844h5 gmail_msg">
<br class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">On Fri, 3 Mar 2017, 8:02 a.m. Ghanshyam Mann, <<a href="mailto:ghanshyammann@gmail.com" class="gmail_msg" target="_blank">ghanshyammann@gmail.com</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="gmail_default m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Thanks. +1. i added my list in ethercalc.</div><div class="gmail_default m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="gmail_default m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Left put scenario tests can be run on periodic and experimental job. IMO on both ( periodic and experimental) to monitor their status periodically as well as on particular patch if we need to. </div><div class="gmail_extra m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" clear="all"><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="m_6566325548703639844m_-7662440994883868273m_7817127192274301852gmail_signature m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" data-smartmail="gmail_signature"><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">-gmann</div></div></div></div>
<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="gmail_quote m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div></div></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="gmail_extra m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="gmail_quote m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">On Fri, Mar 3, 2017 at 4:28 PM, Andrea Frittoli <span class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><<a href="mailto:andrea.frittoli@gmail.com" class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" target="_blank">andrea.frittoli@gmail.com</a>></span> wrote:<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div></div></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="gmail_extra m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="gmail_quote m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><blockquote class="gmail_quote m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">Hello folks,<div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">we discussed a lot since the PTG about issues with gate stability; we need a stable and reliable gate to ensure smooth progress in Pike.</div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">One of the issues that stands out is that most of the times during test runs our test VMs are under heavy load.</div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">This can be the common cause behind several failures we've seen in the gate, so we agreed during the QA meeting yesterday [0] that we're going to try reducing the load and see whether that improves stability.<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">Next steps are:<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">- select a subset of scenario tests to be executed in the gate, based on [1], and run them serially only </div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">- the patch for this is [2] and we will approve this by the end of the day</div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">- we will monitor stability for a week - if needed we may reduce concurrency a bit on API tests as well, and identify "heavy" tests candidate for removal / refactor</div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">- the QA team won't approve any new test (scenario or heavy resource consuming api) until gate stability is ensured </div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">Thanks for your patience and collaboration!</div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">Andrea</div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">---</div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">irc: andreaf</div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">[0] <a href="http://eavesdrop.openstack.org/meetings/qa/2017/qa.2017-03-02-17.00.txt" class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" target="_blank">http://eavesdrop.openstack.org/meetings/qa/2017/qa.2017-03-02-17.00.txt</a></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">[1] <a href="https://ethercalc.openstack.org/nu56u2wrfb2b" class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" target="_blank">https://ethercalc.openstack.org/nu56u2wrfb2b</a></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">[2] <span style="white-space:pre-wrap" class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><a href="https://review.openstack.org/#/c/439698/" class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" target="_blank">https://review.openstack.org/#/c/439698/</a> </span></div></div>
<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></blockquote></div></div></div><div class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="gmail_extra m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><div class="gmail_quote m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"><blockquote class="gmail_quote m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">__________________________________________________________________________<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">
<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></blockquote></div><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg"></div></div>
__________________________________________________________________________<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="m_6566325548703639844m_-7662440994883868273gmail_msg gmail_msg">
</blockquote></div>
</div></div><br class="gmail_msg">__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
<br class="gmail_msg"></blockquote></div><br class="gmail_msg"></div>
__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
</blockquote></div></div>