Open Stack

On 14/02/17 16:11, Joshua Hesketh wrote:
> Hey Tom,
>
> Where is that script being fired from (a quick grep doesn't find it), or
> is it a tool people are using?
>
> If it's a tool we'd need to make sure whoever is using it gets a new
> version to rule it out.

Indeed.


It's fired from a PHP service on www.openstack.org itself, which writes 
to the Member database:

https://github.com/OpenStackweb/openstack-org/blob/master/auc-metrics/code/services/ActiveModeratorService.php

The next step is to update the copy of the script it references:

https://github.com/OpenStackweb/openstack-org/blob/master/auc-metrics/lib/uc-recognition/tools/get_active_moderator.py

I am not sure if this is in place using git submodules or manually, but 
will figure it out and get that updated.




>  - Josh
>
> On Tue, Feb 14, 2017 at 7:07 PM, Tom Fifield <tom at openstack.org
> <mailto:tom at openstack.org>> wrote:
>
>     On 14/02/17 16:06, Joshua Hesketh wrote:
>
>         Hey,
>
>         I've brought the service back up, but have no new clues as to why.
>
>
>     Cheers.
>
>     Going to try: https://review.openstack.org/#/c/433478/
>     <https://review.openstack.org/#/c/433478/>
>     to see if this script is culprit.
>
>
>         - Josh
>
>         On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield <tom at openstack.org
>         <mailto:tom at openstack.org>
>         <mailto:tom at openstack.org <mailto:tom at openstack.org>>> wrote:
>
>             On 10/02/17 22:39, Jeremy Stanley wrote:
>
>                 On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote:
>                 [...]
>
>                     Down again, this time with "Network is unreachable".
>
>                 [...]
>
>                 I'm not finding any obvious errors on the server nor
>         relevant
>                 maintenance notices/trouble tickets from the service
>         provider to
>                 explain this. I do see conspicuous gaps in network
>         traffic volume
>                 and system load from ~06:45 to ~08:10 UTC according to
>         cacti:
>
>                     http://cacti.openstack.org/?tree_id=1&leaf_id=156
>         <http://cacti.openstack.org/?tree_id=1&leaf_id=156>
>                 <http://cacti.openstack.org/?tree_id=1&leaf_id=156
>         <http://cacti.openstack.org/?tree_id=1&leaf_id=156>>
>
>                 Skipping back through previous days I find some similar gaps
>                 starting anywhere from 06:30 to 07:00 and ending between
>         07:00 and
>                 08:00 but they don't seem to occur every day and I'm not
>         having much
>                 luck finding a pattern. It _is_ conspicuously close to when
>                 /etc/cron.daily scripts get fired from the crontab so
>         might coincide
>                 with log rotation/service restarts? The graphs don't
>         show these gaps
>                 correlating with any spikes in CPU, memory or disk
>         activity so it
>                 doesn't seem to be resource starvation (at least not for
>         any common
>                 resources we're tracking).
>
>
>             Indeed. It's down again today during the same timeslot.
>
>             Another idea for the cron-based theory:
>
>
>         https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py
>         <https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py>
>
>         <https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py
>         <https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py>>
>
>             loops through the list of Ask OpenStack users via the API on
>         a cron
>             running on www.openstack.org <http://www.openstack.org>
>         <http://www.openstack.org>. Not sure
>             when that cron runs, but if it's similar, this could
>         potentially be
>             a high-load generator.
>
>
>
>
>             Regards,
>
>
>             Tom
>
>
>             _______________________________________________
>             OpenStack-Infra mailing list
>             OpenStack-Infra at lists.openstack.org
>         <mailto:OpenStack-Infra at lists.openstack.org>
>             <mailto:OpenStack-Infra at lists.openstack.org
>         <mailto:OpenStack-Infra at lists.openstack.org>>
>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra>
>
>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra>>
>
>
>
>