<div dir="ltr">Hey Tom,<div><br></div><div>Where is that script being fired from (a quick grep doesn't find it), or is it a tool people are using?</div><div><br></div><div>If it's a tool we'd need to make sure whoever is using it gets a new version to rule it out.</div><div><br></div><div> - Josh</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 14, 2017 at 7:07 PM, Tom Fifield <span dir="ltr"><<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 14/02/17 16:06, Joshua Hesketh wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hey,<br>
<br>
I've brought the service back up, but have no new clues as to why.<br>
</blockquote>
<br></span>
Cheers.<br>
<br>
Going to try: <a href="https://review.openstack.org/#/c/433478/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/433478/</a><br>
to see if this script is culprit.<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
- Josh<br>
<br>
On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield <<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a><br></span><div><div class="h5">
<mailto:<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a>>> wrote:<br>
<br>
On 10/02/17 22:39, Jeremy Stanley wrote:<br>
<br>
On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote:<br>
[...]<br>
<br>
Down again, this time with "Network is unreachable".<br>
<br>
[...]<br>
<br>
I'm not finding any obvious errors on the server nor relevant<br>
maintenance notices/trouble tickets from the service provider to<br>
explain this. I do see conspicuous gaps in network traffic volume<br>
and system load from ~06:45 to ~08:10 UTC according to cacti:<br>
<br>
<a href="http://cacti.openstack.org/?tree_id=1&leaf_id=156" rel="noreferrer" target="_blank">http://cacti.openstack.org/?tr<wbr>ee_id=1&leaf_id=156</a><br>
<<a href="http://cacti.openstack.org/?tree_id=1&leaf_id=156" rel="noreferrer" target="_blank">http://cacti.openstack.org/?t<wbr>ree_id=1&leaf_id=156</a>><br>
<br>
Skipping back through previous days I find some similar gaps<br>
starting anywhere from 06:30 to 07:00 and ending between 07:00 and<br>
08:00 but they don't seem to occur every day and I'm not having much<br>
luck finding a pattern. It _is_ conspicuously close to when<br>
/etc/cron.daily scripts get fired from the crontab so might coincide<br>
with log rotation/service restarts? The graphs don't show these gaps<br>
correlating with any spikes in CPU, memory or disk activity so it<br>
doesn't seem to be resource starvation (at least not for any common<br>
resources we're tracking).<br>
<br>
<br>
Indeed. It's down again today during the same timeslot.<br>
<br>
Another idea for the cron-based theory:<br>
<br>
<a href="https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py" rel="noreferrer" target="_blank">https://github.com/openstack/u<wbr>c-recognition/blob/master/tool<wbr>s/get_active_moderator.py</a><br>
<<a href="https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>uc-recognition/blob/master/too<wbr>ls/get_active_moderator.py</a>><br>
<br>
loops through the list of Ask OpenStack users via the API on a cron<br></div></div>
running on <a href="http://www.openstack.org" rel="noreferrer" target="_blank">www.openstack.org</a> <<a href="http://www.openstack.org" rel="noreferrer" target="_blank">http://www.openstack.org</a>>. Not sure<span class=""><br>
when that cron runs, but if it's similar, this could potentially be<br>
a high-load generator.<br>
<br>
<br>
<br>
<br>
Regards,<br>
<br>
<br>
Tom<br>
<br>
<br>
______________________________<wbr>_________________<br>
OpenStack-Infra mailing list<br>
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstac<wbr>k.org</a><br></span>
<mailto:<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.<wbr>openstack.org</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-infra</a><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" rel="noreferrer" target="_blank">http://lists.openstack.org/cg<wbr>i-bin/mailman/listinfo/opensta<wbr>ck-infra</a>><br>
<br>
<br>
</blockquote>
<br>
</blockquote></div><br></div>