<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 14, 2017 at 7:15 PM, Tom Fifield <span dir="ltr"><<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 14/02/17 16:11, Joshua Hesketh wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hey Tom,<br>
<br>
Where is that script being fired from (a quick grep doesn't find it), or<br>
is it a tool people are using?<br>
<br>
If it's a tool we'd need to make sure whoever is using it gets a new<br>
version to rule it out.<br>
</blockquote>
<br></span>
Indeed.<br>
<br>
<br>
It's fired from a PHP service on <a href="http://www.openstack.org" rel="noreferrer" target="_blank">www.openstack.org</a> itself, which writes to the Member database:<br>
<br>
<a href="https://github.com/OpenStackweb/openstack-org/blob/master/auc-metrics/code/services/ActiveModeratorService.php" rel="noreferrer" target="_blank">https://github.com/OpenStackwe<wbr>b/openstack-org/blob/master/<wbr>auc-metrics/code/services/Acti<wbr>veModeratorService.php</a></blockquote><div><br></div><div><br></div><div>Right. I wonder if somebody could check the logs to see if the process times out. Sadly looking at that code it looks like any output messages from the script will be discarded.</div><div><br></div><div> - Josh</div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
The next step is to update the copy of the script it references:<br>
<br>
<a href="https://github.com/OpenStackweb/openstack-org/blob/master/auc-metrics/lib/uc-recognition/tools/get_active_moderator.py" rel="noreferrer" target="_blank">https://github.com/OpenStackwe<wbr>b/openstack-org/blob/master/<wbr>auc-metrics/lib/uc-<wbr>recognition/tools/get_active_<wbr>moderator.py</a><br>
<br>
I am not sure if this is in place using git submodules or manually, but will figure it out and get that updated.<br>
<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
- Josh<br>
<br>
On Tue, Feb 14, 2017 at 7:07 PM, Tom Fifield <<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a><br></span><span class="">
<mailto:<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a>>> wrote:<br>
<br>
On 14/02/17 16:06, Joshua Hesketh wrote:<br>
<br>
Hey,<br>
<br>
I've brought the service back up, but have no new clues as to why.<br>
<br>
<br>
Cheers.<br>
<br>
Going to try: <a href="https://review.openstack.org/#/c/433478/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/433478/</a><br>
<<a href="https://review.openstack.org/#/c/433478/" rel="noreferrer" target="_blank">https://review.openstack.org/<wbr>#/c/433478/</a>><br>
to see if this script is culprit.<br>
<br>
<br>
- Josh<br>
<br>
On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield <<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a><br>
<mailto:<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a>><br></span><div><div class="h5">
<mailto:<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a> <mailto:<a href="mailto:tom@openstack.org" target="_blank">tom@openstack.org</a>>>> wrote:<br>
<br>
On 10/02/17 22:39, Jeremy Stanley wrote:<br>
<br>
On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote:<br>
[...]<br>
<br>
Down again, this time with "Network is unreachable".<br>
<br>
[...]<br>
<br>
I'm not finding any obvious errors on the server nor<br>
relevant<br>
maintenance notices/trouble tickets from the service<br>
provider to<br>
explain this. I do see conspicuous gaps in network<br>
traffic volume<br>
and system load from ~06:45 to ~08:10 UTC according to<br>
cacti:<br>
<br>
<a href="http://cacti.openstack.org/?tree_id=1&leaf_id=156" rel="noreferrer" target="_blank">http://cacti.openstack.org/?tr<wbr>ee_id=1&leaf_id=156</a><br>
<<a href="http://cacti.openstack.org/?tree_id=1&leaf_id=156" rel="noreferrer" target="_blank">http://cacti.openstack.org/?t<wbr>ree_id=1&leaf_id=156</a>><br>
<<a href="http://cacti.openstack.org/?tree_id=1&leaf_id=156" rel="noreferrer" target="_blank">http://cacti.openstack.org/?t<wbr>ree_id=1&leaf_id=156</a><br>
<<a href="http://cacti.openstack.org/?tree_id=1&leaf_id=156" rel="noreferrer" target="_blank">http://cacti.openstack.org/?t<wbr>ree_id=1&leaf_id=156</a>>><br>
<br>
Skipping back through previous days I find some similar gaps<br>
starting anywhere from 06:30 to 07:00 and ending between<br>
07:00 and<br>
08:00 but they don't seem to occur every day and I'm not<br>
having much<br>
luck finding a pattern. It _is_ conspicuously close to when<br>
/etc/cron.daily scripts get fired from the crontab so<br>
might coincide<br>
with log rotation/service restarts? The graphs don't<br>
show these gaps<br>
correlating with any spikes in CPU, memory or disk<br>
activity so it<br>
doesn't seem to be resource starvation (at least not for<br>
any common<br>
resources we're tracking).<br>
<br>
<br>
Indeed. It's down again today during the same timeslot.<br>
<br>
Another idea for the cron-based theory:<br>
<br>
<br>
<a href="https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py" rel="noreferrer" target="_blank">https://github.com/openstack/u<wbr>c-recognition/blob/master/tool<wbr>s/get_active_moderator.py</a><br>
<<a href="https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>uc-recognition/blob/master/too<wbr>ls/get_active_moderator.py</a>><br>
<br>
<<a href="https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>uc-recognition/blob/master/too<wbr>ls/get_active_moderator.py</a><br>
<<a href="https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>uc-recognition/blob/master/too<wbr>ls/get_active_moderator.py</a>>><br>
<br>
loops through the list of Ask OpenStack users via the API on<br>
a cron<br>
running on <a href="http://www.openstack.org" rel="noreferrer" target="_blank">www.openstack.org</a> <<a href="http://www.openstack.org" rel="noreferrer" target="_blank">http://www.openstack.org</a>><br>
<<a href="http://www.openstack.org" rel="noreferrer" target="_blank">http://www.openstack.org</a>>. Not sure<br>
when that cron runs, but if it's similar, this could<br>
potentially be<br>
a high-load generator.<br>
<br>
<br>
<br>
<br>
Regards,<br>
<br>
<br>
Tom<br>
<br>
<br>
______________________________<wbr>_________________<br>
OpenStack-Infra mailing list<br>
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstac<wbr>k.org</a><br>
<mailto:<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.<wbr>openstack.org</a>><br></div></div>
<mailto:<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.<wbr>openstack.org</a><span class=""><br>
<mailto:<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.<wbr>openstack.org</a>>><br>
<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-infra</a><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" rel="noreferrer" target="_blank">http://lists.openstack.org/cg<wbr>i-bin/mailman/listinfo/opensta<wbr>ck-infra</a>><br>
<br></span>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" rel="noreferrer" target="_blank">http://lists.openstack.org/cg<wbr>i-bin/mailman/listinfo/opensta<wbr>ck-infra</a><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" rel="noreferrer" target="_blank">http://lists.openstack.org/cg<wbr>i-bin/mailman/listinfo/opensta<wbr>ck-infra</a>>><br>
<br>
<br>
<br>
<br>
</blockquote>
<br>
</blockquote></div><br></div></div>