On Mon, Sep 22, 2025 at 12:47 PM Max Payne <yardalgedal@gmail.com> wrote:
Hi, Gregory, 

Thanks for your answer! 

Answering on you questions … first of all, we have pretty overloaded Octavia environment in this region when the problem happened, e.g. we have around 650 LBs and 1300 amphorae respectively, which in general causes a problems like mentioned one. 

Why we had LBs with 0 amphorae – I’m seeing it as a separate problem and want to treat them separately, let’s think that they were just deleted manually (in fact it will be related to the huge load on octavia in this region). 

From suspicious, look at attached screenshot, you can see that it was two parallel master creation flows on two different controllers. 


Yeah there are 2 tasks, with the same name and same uuid, but with different amphora IDs, that looks highly suspicious.
Taskflow uses Redis to lock the job that is being executed, so it's not supposed to happen. IMHO it could be an issue with redis/taskflow configuration or a bug in taskflow.
Can you open a bug in launchpad in the octavia project? https://bugs.launchpad.net/octavia
it would be great to include the full logs of the failover of one of the loadbalancers (and maybe split the logs if they come from different workers/servers)
please include your settings for taskflow/jobboard from octavia.conf, and maybe provide some detail on how you deployed and use redis (use of sentinel, etc..)
 
Thanks!

Yes they were took by different workers (we have task flow >= 4.4.0).

От: Gregory Thiemonge <gthiemonge@redhat.com>
Дата: понедельник, 22 сентября 2025 г. в 08:05
Кому: yardalgedal@gmail.com <yardalgedal@gmail.com>
Копия: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>
Тема: Re: [octavia] Creation of several master amps

Hi,

I've never seen such an issue.
Do you know why you had LBs with 0 amphorae? what was their provisioning_status before performing a failover?
Did you notice any suspicious logs in the octavia worker service during the failover (ERROR-level messages)?
Can you check if the 2 MASTER amphorae of each LB were created by the same octavia-worker? ("Created Amphora xxx in DB for load balancer xxx" in the logs) I want to ensure that no tasks has been executed on 2 different workers
If you have the logs of one failover (with DEBUG or not), it would be great if you could open a bug on launchpad.

Thanks,

On Sun, Sep 21, 2025 at 7:29 PM <yardalgedal@gmail.com> wrote:
Hi all, we're using octavia of version zed and caught a case when it regularly creating 3 amphorae instead of 2.

We have not had any amphorae for which of these LBs and I did failover for each of them:

openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf

openstack loadbalancer failover 70f98bfb-8045-4d4a-9729-d60b6e141dbf

openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739

openstack loadbalancer failover 96223d34-6941-4bc5-881a-ddfd83070739

openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4

openstack loadbalancer failover f9f859d0-2b31-449b-8723-700bb7f3f3f4

then first backup amphorae was created pretty fast

openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf
openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739
openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| 3dfa43cd-2719-487e-8ba2-8a4aa0fe7301 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | BACKUP | 10.94.64.112  | 10.210.73.197 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip        |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| b39472aa-04a0-42b8-8c09-4617ea1a0f33 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | BACKUP | 10.94.64.232  | 10.210.64.32 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+
| 60af8e11-c6ad-45f0-8e07-1bcc4a477444 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | BACKUP | 10.94.64.74   | None  |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+

but then it started to create 2x Master's for each (!) of them (we can talk about lags, but not in case of 3x):

openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf
openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739
openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| 3dfa43cd-2719-487e-8ba2-8a4aa0fe7301 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | BACKUP | 10.94.64.112  | 10.210.73.197 |
| 3dd9bdb4-0dfe-4d77-b08f-02fbbffcd892 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | BOOTING   | None   | None          | None          |
| ff6ec3de-6942-4978-9ea5-d793ff1fdbf8 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | BOOTING   | None   | None          | None          |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip        |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| b39472aa-04a0-42b8-8c09-4617ea1a0f33 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | BACKUP | 10.94.64.232  | 10.210.64.32 |
| 9909b300-d266-4344-b6fc-edc4af651485 | 96223d34-6941-4bc5-881a-ddfd83070739 | BOOTING   | None   | None          | None         |
| cc123c55-2953-425e-867e-f4057483bb09 | 96223d34-6941-4bc5-881a-ddfd83070739 | BOOTING   | None   | None          | None         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+
| 60af8e11-c6ad-45f0-8e07-1bcc4a477444 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | BACKUP | 10.94.64.74   | None  |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+

as a result we started to have 3x Master's for them

openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf
openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739
openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| 3dfa43cd-2719-487e-8ba2-8a4aa0fe7301 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | BACKUP | 10.94.64.112  | 10.210.73.197 |
| 3dd9bdb4-0dfe-4d77-b08f-02fbbffcd892 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | MASTER | 10.94.64.56   | 10.210.73.197 |
| 14e8acfe-8510-4629-a583-53ba88f91eca | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | MASTER | 10.94.64.30   | 10.210.73.197 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip        |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
| b39472aa-04a0-42b8-8c09-4617ea1a0f33 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | BACKUP | 10.94.64.232  | 10.210.64.32 |
| 9909b300-d266-4344-b6fc-edc4af651485 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | MASTER | 10.94.64.151  | 10.210.64.32 |
| 9d197234-6411-4069-8be3-a399cc1fbc3a | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | MASTER | 10.94.64.212  | 10.210.64.32 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip         |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+
| 60af8e11-c6ad-45f0-8e07-1bcc4a477444 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | BACKUP | 10.94.64.74   | 10.211.193.63 |
| d086f283-b9d8-4b23-a782-e0ace769e893 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | MASTER | 10.94.64.225  | 10.211.193.63 |
| 8a90fc7c-eb0a-4635-8ece-7c4704c60f7a | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | MASTER | 10.94.64.50   | 10.211.193.63 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+

taskflow is enabled in our case, backend is redis

can you suggest smth to check or maybe some updates were already done?