Open Stack

Hey Tom,

Where is that script being fired from (a quick grep doesn't find it), or is
it a tool people are using?

If it's a tool we'd need to make sure whoever is using it gets a new
version to rule it out.

 - Josh

On Tue, Feb 14, 2017 at 7:07 PM, Tom Fifield <tom at openstack.org> wrote:

> On 14/02/17 16:06, Joshua Hesketh wrote:
>
>> Hey,
>>
>> I've brought the service back up, but have no new clues as to why.
>>
>
> Cheers.
>
> Going to try: https://review.openstack.org/#/c/433478/
> to see if this script is culprit.
>
>
> - Josh
>>
>> On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield <tom at openstack.org
>> <mailto:tom at openstack.org>> wrote:
>>
>>     On 10/02/17 22:39, Jeremy Stanley wrote:
>>
>>         On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote:
>>         [...]
>>
>>             Down again, this time with "Network is unreachable".
>>
>>         [...]
>>
>>         I'm not finding any obvious errors on the server nor relevant
>>         maintenance notices/trouble tickets from the service provider to
>>         explain this. I do see conspicuous gaps in network traffic volume
>>         and system load from ~06:45 to ~08:10 UTC according to cacti:
>>
>>             http://cacti.openstack.org/?tree_id=1&leaf_id=156
>>         <http://cacti.openstack.org/?tree_id=1&leaf_id=156>
>>
>>         Skipping back through previous days I find some similar gaps
>>         starting anywhere from 06:30 to 07:00 and ending between 07:00 and
>>         08:00 but they don't seem to occur every day and I'm not having
>> much
>>         luck finding a pattern. It _is_ conspicuously close to when
>>         /etc/cron.daily scripts get fired from the crontab so might
>> coincide
>>         with log rotation/service restarts? The graphs don't show these
>> gaps
>>         correlating with any spikes in CPU, memory or disk activity so it
>>         doesn't seem to be resource starvation (at least not for any
>> common
>>         resources we're tracking).
>>
>>
>>     Indeed. It's down again today during the same timeslot.
>>
>>     Another idea for the cron-based theory:
>>
>>     https://github.com/openstack/uc-recognition/blob/master/tool
>> s/get_active_moderator.py
>>     <https://github.com/openstack/uc-recognition/blob/master/too
>> ls/get_active_moderator.py>
>>
>>     loops through the list of Ask OpenStack users via the API on a cron
>>     running on www.openstack.org <http://www.openstack.org>. Not sure
>>     when that cron runs, but if it's similar, this could potentially be
>>     a high-load generator.
>>
>>
>>
>>
>>     Regards,
>>
>>
>>     Tom
>>
>>
>>     _______________________________________________
>>     OpenStack-Infra mailing list
>>     OpenStack-Infra at lists.openstack.org
>>     <mailto:OpenStack-Infra at lists.openstack.org>
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20170214/faa3f1fd/attachment.html>