Open Stack

On 16 December 2016 at 21:40, Ben Nemec <openstack at nemebean.com> wrote:
> Just a heads up for everyone, I've added some DB cleanup jobs to rh1 which
> will hopefully prevent the performance degradations over time that we've
> been seeing in that environment.  Specifically, the crontab now looks like
> this:
>
> # Clean up heat db
> 0 5 * * * heat-manage purge_deleted 7
> # Archive nova db entries 5 times so we get everything
> 0,10,20,30,40 6 * * * nova-manage db archive_deleted_rows --max_rows 100000
>

lgtm, We did a number of things last week in order to deal with the
performance problems, these db archive commands cover part of it and
needed to be added.

The other thing we need to keep an eye on is the number of OVS ports
on each compute node, on some compute nodes we had over 2000 ovs ports
and ovs couldn't be restarted(and some other services). Ultimately to
deal with this we rebooted each compute node and allowed
neutron-ovs-cleanup to delete the unused ports. This wasn't ideal
because
1) neutron-ovs-cleanup in some cases took a long time to delete the
ports, and delayed nova-compute from starting up, so compute nodes
couldn't be used for a extended period of time.
2) This also caused a reboot of the infrastructure nodes we run on rh1
(e.g. proxy, te-broker etc...) some of these didn't come back as
expected, where needed patches have been submitted[1][2]

I haven't done anything to prevent the ports building up again, so
this is probably still a ongoing issue we need to look into

[1] - https://review.openstack.org/#/c/409930/
[2] - https://review.openstack.org/#/c/406927/

> I picked 5 and 6 AM UTC because I think that's before most people in the EU
> are starting and well after the US is done so the cloud should be pretty
> quiet at that time.
>
> I think it's worth noting that we should probably be setting up this sort of
> thing on initial deployment by default.  Maybe we are now (rh1 is still back
> on Mitaka), but if not we should figure out some appropriate defaults.
>
> -Ben
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev