<div dir="ltr"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">Some (new?) data on the oom kill issue in the gate.</div><div class="gmail_msg"><br></div><div class="gmail_msg">I filed a new bug / E-R query yet for the issue [1][2] since it looks to me like the issue is not specific to mysqld - oom-kill will just pick the best candidate, which in most cases happens to be mysqld. The next most likely candidate to show errors in the logs is keystone, since token requests are rather frequent, more than any other API call probably.</div><div class="gmail_msg"><br></div><div class="gmail_msg">According to logstash [3] all failures identified by [2] happen on RAX nodes [3], which I hadn't realised before.<br></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Comparing dstat data between the failed run and a successful on an OVH node [4], the main difference I can spot is free memory. </div><div class="gmail_msg">For the same test job, the free memory tends to be much lower, quite close to zero for the majority of the time on the RAX node. My guess is that an unlucky scheduling of tests may cause a slightly higher peak in memory usage and trigger the oom-kill.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I find it hard to relate lower free memory to a specific cloud provider / underlying virtualisation technology, but maybe someone has an idea about how that could be?</div><div class="gmail_msg"><br></div><div class="gmail_msg">Andrea</div><div class="gmail_msg"><br></div><div class="gmail_msg"><div class="gmail_msg">[0] <a href="http://logs.openstack.org/93/432793/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/6f31320/logs/syslog.txt.gz#_Feb_14_00_32_28" class="gmail_msg" target="_blank">http://logs.openstack.org/93/432793/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/6f31320/logs/syslog.txt.gz#_Feb_14_00_32_28</a> </div><div class="gmail_msg">[1] <a href="https://bugs.launchpad.net/tempest/+bug/1664953" class="gmail_msg" target="_blank">https://bugs.launchpad.net/tempest/+bug/1664953</a> </div><div class="gmail_msg">[2] <a href="https://review.openstack.org/434238" class="gmail_msg" target="_blank">https://review.openstack.org/434238</a> </div></div><div class="gmail_msg">[3] <a href="http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Out%20of%20memory%3A%20Kill%20process%5C%22%20AND%20tags%3A%5C%22syslog.txt%5C%22" class="gmail_msg" target="_blank">http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Out%20of%20memory%3A%20Kill%20process%5C%22%20AND%20tags%3A%5C%22syslog.txt%5C%22</a> </div><div class="gmail_msg">[4] <a href="http://logs.openstack.org/93/432793/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/1dfb4b7/logs/dstat-csv_log.txt.gz" class="gmail_msg" target="_blank">http://logs.openstack.org/93/432793/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/1dfb4b7/logs/dstat-csv_log.txt.gz</a> </div></div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Mon, Feb 6, 2017 at 10:13 AM Miguel Angel Ajo Pelayo <<a href="mailto:majopela@redhat.com" class="gmail_msg" target="_blank">majopela@redhat.com</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Jeremy Stanley wrote:<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><span class="m_8538578303744445123m_6851083707802139819gmail-im gmail_msg" style="font-size:12.8px"><br class="gmail_msg"></span><span style="font-size:12.8px" class="gmail_msg">> It's an option of last resort, I think. The next consistent flavor</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">> up in most of the providers donating resources is double the one</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">> we're using (which is a fairly typical pattern in public clouds). As</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">> aggregate memory constraints are our primary quota limit, this would</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">> effectively halve our current job capacity.</span><br class="gmail_msg"></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg"><br class="gmail_msg"></span></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">Properly coordinated with all the cloud the providers, they could create flavours which are private but available to our tenants, where a 25-50% more RAM would be just enough.</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">I agree that should probably be a last resort tool, and we should keep looking for proper ways to find where we consume unnecessary RAM and make sure that's properly freed up.</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg"><br class="gmail_msg"></span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg">It could be interesting to coordinate such flavour creation in the mean time, even if we don't use it now, we could eventually test it or put it to work if we find trapped anytime later.</span></div><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg"><br class="gmail_msg"></span></div></div><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Sun, Feb 5, 2017 at 8:37 PM, Matt Riedemann <span dir="ltr" class="gmail_msg"><<a href="mailto:mriedemos@gmail.com" class="gmail_msg" target="_blank">mriedemos@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="gmail_msg">On 2/5/2017 1:19 PM, Clint Byrum wrote:<br class="gmail_msg">
<blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
Also I wonder if there's ever been any serious consideration given to<br class="gmail_msg">
switching to protobuf? Feels like one could make oslo.versionedobjects<br class="gmail_msg">
a wrapper around protobuf relatively easily, but perhaps that's already<br class="gmail_msg">
been explored in a forum that I wasn't paying attention to.<br class="gmail_msg">
</blockquote>
<br class="gmail_msg"></span>
I've never heard of anyone attempting that.<span class="m_8538578303744445123m_6851083707802139819HOEnZb gmail_msg"><font color="#888888" class="gmail_msg"><br class="gmail_msg">
<br class="gmail_msg">
-- <br class="gmail_msg">
<br class="gmail_msg">
Thanks,<br class="gmail_msg">
<br class="gmail_msg">
Matt Riedemann</font></span><div class="m_8538578303744445123m_6851083707802139819HOEnZb gmail_msg"><div class="m_8538578303744445123m_6851083707802139819h5 gmail_msg"><br class="gmail_msg">
<br class="gmail_msg">
__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
</div></div></blockquote></div><br class="gmail_msg"></div>
__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
</blockquote></div></div>