Open Stack

Currently manage-projects is frequently failing on a mysql error:
http://paste.openstack.org/show/134438/

Our operating theory is that if new project creation takes a long time
(definition to be determined) the open mysql port goes away and then it
isn't open when we need it, so the update to the groups file for the new
project fails and then so do all subsequent actions.

This creates a repo in gerrit without a default branch which then breaks
any image creation actions for us and for many third parties
implementing their own CI as well. This is not directly related to
manage-projects but is one of the effects we are seeing when we have a
manage-projects failure.

The frequency of the failures on the mysql error is quite high, most
recent projects have hit it and have had to be respun manually, which is
less than efficient since manage-projects is triggered on average about
3 times a day currently.

The story for this can be found here:
https://storyboard.openstack.org/#!/story/2000017

Course of action:
    - cease approving new project repos until further notice
    - help us to remedy the situation (calling any mysql folks with
advice on how to keep the port open, or re-open it when we need it)
    - share this information with anyone who you feel needs to know

Thank you,
Anita.