Open Stack

On 14/02/17 16:06, Joshua Hesketh wrote:
> Hey,
>
> I've brought the service back up, but have no new clues as to why.

Cheers.

Going to try: https://review.openstack.org/#/c/433478/
to see if this script is culprit.


> - Josh
>
> On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield <tom at openstack.org
> <mailto:tom at openstack.org>> wrote:
>
>     On 10/02/17 22:39, Jeremy Stanley wrote:
>
>         On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote:
>         [...]
>
>             Down again, this time with "Network is unreachable".
>
>         [...]
>
>         I'm not finding any obvious errors on the server nor relevant
>         maintenance notices/trouble tickets from the service provider to
>         explain this. I do see conspicuous gaps in network traffic volume
>         and system load from ~06:45 to ~08:10 UTC according to cacti:
>
>             http://cacti.openstack.org/?tree_id=1&leaf_id=156
>         <http://cacti.openstack.org/?tree_id=1&leaf_id=156>
>
>         Skipping back through previous days I find some similar gaps
>         starting anywhere from 06:30 to 07:00 and ending between 07:00 and
>         08:00 but they don't seem to occur every day and I'm not having much
>         luck finding a pattern. It _is_ conspicuously close to when
>         /etc/cron.daily scripts get fired from the crontab so might coincide
>         with log rotation/service restarts? The graphs don't show these gaps
>         correlating with any spikes in CPU, memory or disk activity so it
>         doesn't seem to be resource starvation (at least not for any common
>         resources we're tracking).
>
>
>     Indeed. It's down again today during the same timeslot.
>
>     Another idea for the cron-based theory:
>
>     https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py
>     <https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py>
>
>     loops through the list of Ask OpenStack users via the API on a cron
>     running on www.openstack.org <http://www.openstack.org>. Not sure
>     when that cron runs, but if it's similar, this could potentially be
>     a high-load generator.
>
>
>
>
>     Regards,
>
>
>     Tom
>
>
>     _______________________________________________
>     OpenStack-Infra mailing list
>     OpenStack-Infra at lists.openstack.org
>     <mailto:OpenStack-Infra at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra>
>
>