In the case where it's stuck in PENDING_UPDATE for a long time, we force the loadbalancer to ERROR status and then perform a failover to let Octavia re-create the amphorae.


SQL> UPDATE load_balancer SET provisioning_status = 'ERROR' WHERE id = '<LOADBALANCER ID>' LIMIT 1;


$ openstack loadbalancer failover <LOADBALANCER ID>



That *may* also help the PENDING_DELETE case, if it can recover and then allow a subsequent delete.



regards,

Dale Smith



On 14/02/24 01:59, Paul Browne wrote:
Hello Octavia users,

For a while I've been trying to find a systematic way to remove Octavia LBs+Amphora that have become stuck in ERROR or transitional PENDING_* states ;

$ openstack loadbalancer amphora list -f value  | grep ERROR
a2323f14-3aaa-418a-8249-111aaa9c21fe 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR MASTER 10.8.0.242 192.168.3.247
e5b236ba-e7ee-4ed7-9f58-57ce7a408489 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR BACKUP 10.8.0.190 192.168.3.247
6b556f28-93c9-49dd-b6ee-4379288e7957 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR MASTER 10.8.0.39 192.168.3.126
c669db5d-8686-4d5c-9e95-e02030b34301 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR BACKUP 10.8.0.174 192.168.3.126

$ openstack loadbalancer list -f value | grep -vi active
1fa7bd54-f60c-420c-94f3-d4c02f03d4fe k8s-clusterapi-cluster-default-ci-6386871107-kube-upgrade-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.247 PENDING_UPDATE amphora
d5e402fe-2c4b-49af-a700-532cb408cee5 k8s-clusterapi-cluster-default-ci-6386871107-latest-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.126 PENDING_DELETE amphora

These resources are marked immutable and so cannot be failed over or deleted;

$ openstack loadbalancer amphora failover a2323f14-3aaa-418a-8249-111aaa9c21fe
Load Balancer 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe is immutable and cannot be updated. (HTTP 409) (Request-ID: req-6e66c4e8-c3d3-4549-a03c-367017c8c8b3)

$ openstack loadbalancer  failover 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe
Invalid state PENDING_UPDATE of loadbalancer resource 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe (HTTP 409) (Request-ID: req-84b44212-e7a8-4101-a16f-18c774c0577e)

The backing Nova instances for these Amphora do seem to exist and be in good working order.

Is there any API way to purge these out of Octavia's service state, or would (very careful) DB hackery be required here?

Many thanks,
Paul Browne

*******************
Paul Browne
Research Computing Platforms
University Information Services
Roger Needham Building
JJ Thompson Avenue
University of Cambridge
Cambridge
United Kingdom
E-Mail: pfb29@cam.ac.uk
Tel: 0044-1223-746548
*******************