Hi,

I've never seen such an issue.
Do you know why you had LBs with 0 amphorae? what was their provisioning_status before performing a failover?
Did you notice any suspicious logs in the octavia worker service during the failover (ERROR-level messages)?
Can you check if the 2 MASTER amphorae of each LB were created by the same octavia-worker? ("Created Amphora xxx in DB for load balancer xxx" in the logs) I want to ensure that no tasks has been executed on 2 different workers
If you have the logs of one failover (with DEBUG or not), it would be great if you could open a bug on launchpad.

Thanks,

On Sun, Sep 21, 2025 at 7:29 PM <yardalgedal@gmail.com> wrote:
Hi all, we're using octavia of version zed and caught a case when it regularly creating 3 amphorae instead of 2.

We have not had any amphorae for which of these LBs and I did failover for each of them:

openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf

openstack loadbalancer failover 70f98bfb-8045-4d4a-9729-d60b6e141dbf

openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739

openstack loadbalancer failover 96223d34-6941-4bc5-881a-ddfd83070739

openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4

openstack loadbalancer failover f9f859d0-2b31-449b-8723-700bb7f3f3f4

then first backup amphorae was created pretty fast

openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf
openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739
openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| 3dfa43cd-2719-487e-8ba2-8a4aa0fe7301 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | BACKUP | 10.94.64.112  | 10.210.73.197 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip        |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| b39472aa-04a0-42b8-8c09-4617ea1a0f33 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | BACKUP | 10.94.64.232  | 10.210.64.32 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+
| 60af8e11-c6ad-45f0-8e07-1bcc4a477444 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | BACKUP | 10.94.64.74   | None  |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+

but then it started to create 2x Master's for each (!) of them (we can talk about lags, but not in case of 3x):

openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf
openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739
openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| 3dfa43cd-2719-487e-8ba2-8a4aa0fe7301 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | BACKUP | 10.94.64.112  | 10.210.73.197 |
| 3dd9bdb4-0dfe-4d77-b08f-02fbbffcd892 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | BOOTING   | None   | None          | None          |
| ff6ec3de-6942-4978-9ea5-d793ff1fdbf8 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | BOOTING   | None   | None          | None          |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip        |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| b39472aa-04a0-42b8-8c09-4617ea1a0f33 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | BACKUP | 10.94.64.232  | 10.210.64.32 |
| 9909b300-d266-4344-b6fc-edc4af651485 | 96223d34-6941-4bc5-881a-ddfd83070739 | BOOTING   | None   | None          | None         |
| cc123c55-2953-425e-867e-f4057483bb09 | 96223d34-6941-4bc5-881a-ddfd83070739 | BOOTING   | None   | None          | None         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+
| 60af8e11-c6ad-45f0-8e07-1bcc4a477444 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | BACKUP | 10.94.64.74   | None  |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+

as a result we started to have 3x Master's for them

openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf
openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739
openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| 3dfa43cd-2719-487e-8ba2-8a4aa0fe7301 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | BACKUP | 10.94.64.112  | 10.210.73.197 |
| 3dd9bdb4-0dfe-4d77-b08f-02fbbffcd892 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | MASTER | 10.94.64.56   | 10.210.73.197 |
| 14e8acfe-8510-4629-a583-53ba88f91eca | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | MASTER | 10.94.64.30   | 10.210.73.197 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip        |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| b39472aa-04a0-42b8-8c09-4617ea1a0f33 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | BACKUP | 10.94.64.232  | 10.210.64.32 |
| 9909b300-d266-4344-b6fc-edc4af651485 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | MASTER | 10.94.64.151  | 10.210.64.32 |
| 9d197234-6411-4069-8be3-a399cc1fbc3a | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | MASTER | 10.94.64.212  | 10.210.64.32 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| 60af8e11-c6ad-45f0-8e07-1bcc4a477444 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | BACKUP | 10.94.64.74   | 10.211.193.63 |
| d086f283-b9d8-4b23-a782-e0ace769e893 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | MASTER | 10.94.64.225  | 10.211.193.63 |
| 8a90fc7c-eb0a-4635-8ece-7c4704c60f7a | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | MASTER | 10.94.64.50   | 10.211.193.63 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+

taskflow is enabled in our case, backend is redis

can you suggest smth to check or maybe some updates were already done?