Open Stack

Hey,

I've brought the service back up, but have no new clues as to why.

- Josh

On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield <tom at openstack.org> wrote:

> On 10/02/17 22:39, Jeremy Stanley wrote:
>
>> On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote:
>> [...]
>>
>>> Down again, this time with "Network is unreachable".
>>>
>> [...]
>>
>> I'm not finding any obvious errors on the server nor relevant
>> maintenance notices/trouble tickets from the service provider to
>> explain this. I do see conspicuous gaps in network traffic volume
>> and system load from ~06:45 to ~08:10 UTC according to cacti:
>>
>>     http://cacti.openstack.org/?tree_id=1&leaf_id=156
>>
>> Skipping back through previous days I find some similar gaps
>> starting anywhere from 06:30 to 07:00 and ending between 07:00 and
>> 08:00 but they don't seem to occur every day and I'm not having much
>> luck finding a pattern. It _is_ conspicuously close to when
>> /etc/cron.daily scripts get fired from the crontab so might coincide
>> with log rotation/service restarts? The graphs don't show these gaps
>> correlating with any spikes in CPU, memory or disk activity so it
>> doesn't seem to be resource starvation (at least not for any common
>> resources we're tracking).
>>
>>
> Indeed. It's down again today during the same timeslot.
>
> Another idea for the cron-based theory:
>
> https://github.com/openstack/uc-recognition/blob/master/tool
> s/get_active_moderator.py
>
> loops through the list of Ask OpenStack users via the API on a cron
> running on www.openstack.org. Not sure when that cron runs, but if it's
> similar, this could potentially be a high-load generator.
>
>
>
>
> Regards,
>
>
> Tom
>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20170214/ae6ea894/attachment.html>