Open Stack

Hi Mohammed,

Have you opened stories for these issues? I haven't seen any bug
reports about this. If not, could you capture your information in
stories for us to work against?

I am not sure I follow the issue fully, so hopefully we can clarify.

Housekeeping, when spares pool is enabled, boot spare amphora VMs. I'm
not sure how that could inhibit load balancers from provisioning.
Sure, some periodic jobs in the housekeeping process may deadlock and
not complete booting spare VMs, but this will not block any load
balancer provisioning. If there are no spares available the worker
will simply boot a VM as it would normally do without spares enabled
(This was functionality we added to Taskflow from the beginning to
make sure we didn't have issues blocking load balancers from
provisioning if the spares pool was depleted). This lock was added at
an operators request as they did not want any "extra" amphora booted
beyond the configured spares pool limit.

The quota management does lock the project during the critical phase
of managing the quota for the project, just like every OpenStack
project. If that is not completing the quota update in a timely
manner, please open a story with the logs so we can investigate. I
assume your application is correctly designed to handle an
asynchronous API (such as neutron, Octavia, etc.) and handle any
responses that indicate the object is currently immutable and will
retry the request.

Michael

On Sun, Aug 30, 2020 at 8:47 AM Mohammed Naser <mnaser at vexxhost.com> wrote:
>
> Hi everyone,
>
> We're being particularly hit hard across different deployments where
> Octavia has several SELECT .. FOR UPDATE queries which are causing
> load balancers to fail to provision properly.
>
> - spare_pools: This usually hits on rolling restarts of o-housekeeping
> as they all seem to try to capture a lock --
> https://github.com/openstack/octavia/blob/73fbc05386b512aa1dd86a0ed6e8455cc6b8dc7f/octavia/controller/housekeeping/house_keeping.py#L54
>
> - quota: This hits when provisioning a lot of load balancers in
> parallel.  For example in cases when using Heat --
> https://github.com/openstack/octavia/blob/bf3d5372b9fc670ecd08339fa989c9b738ad8d69/octavia/db/repositories.py#L565-L566
>
> These hurt quite a lot in a busy deployment and result in a poor user
> experience unfortunately.  We're trying to off-load Octavia to it's
> own database server but that is more of a "throw power at the problem"
> solution.  I can imagine that we can probably likely look into a
> better/cleaner alternative that avoids this entirely?
>
> I'm happy to try and push for some of this work on our side.
>
> Thanks,
> Mohammed
>
> --
> Mohammed Naser
> VEXXHOST, Inc.
>