Hi, Gregory, Thanks for your answer! Answering on you questions … first of all, we have pretty overloaded Octavia environment in this region when the problem happened, e.g. we have around 650 LBs and 1300 amphorae respectively, which in general causes a problems like mentioned one. Why we had LBs with 0 amphorae – I’m seeing it as a separate problem and want to treat them separately, let’s think that they were just deleted manually (in fact it will be related to the huge load on octavia in this region). From suspicious, look at attached screenshot, you can see that it was two parallel master creation flows on two different controllers. Yes they were took by different workers (we have task flow >= 4.4.0). От: Gregory Thiemonge <gthiemonge@redhat.com> Дата: понедельник, 22 сентября 2025 г. в 08:05 Кому: yardalgedal@gmail.com <yardalgedal@gmail.com> Копия: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Тема: Re: [octavia] Creation of several master amps Hi, I've never seen such an issue. Do you know why you had LBs with 0 amphorae? what was their provisioning_status before performing a failover? Did you notice any suspicious logs in the octavia worker service during the failover (ERROR-level messages)? Can you check if the 2 MASTER amphorae of each LB were created by the same octavia-worker? ("Created Amphora xxx in DB for load balancer xxx" in the logs) I want to ensure that no tasks has been executed on 2 different workers If you have the logs of one failover (with DEBUG or not), it would be great if you could open a bug on launchpad. Thanks, On Sun, Sep 21, 2025 at 7:29 PM <yardalgedal@gmail.com<mailto:yardalgedal@gmail.com>> wrote: Hi all, we're using octavia of version zed and caught a case when it regularly creating 3 amphorae instead of 2. We have not had any amphorae for which of these LBs and I did failover for each of them: openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf openstack loadbalancer failover 70f98bfb-8045-4d4a-9729-d60b6e141dbf openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739 openstack loadbalancer failover 96223d34-6941-4bc5-881a-ddfd83070739 openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4 openstack loadbalancer failover f9f859d0-2b31-449b-8723-700bb7f3f3f4 then first backup amphorae was created pretty fast openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739 openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4 +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ | 3dfa43cd-2719-487e-8ba2-8a4aa0fe7301 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | BACKUP | 10.94.64.112 | 10.210.73.197 | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ +--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+ | b39472aa-04a0-42b8-8c09-4617ea1a0f33 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | BACKUP | 10.94.64.232 | 10.210.64.32 | +--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+ +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+ | 60af8e11-c6ad-45f0-8e07-1bcc4a477444 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | BACKUP | 10.94.64.74 | None | +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+ but then it started to create 2x Master's for each (!) of them (we can talk about lags, but not in case of 3x): openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739 openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4 +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ | 3dfa43cd-2719-487e-8ba2-8a4aa0fe7301 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | BACKUP | 10.94.64.112 | 10.210.73.197 | | 3dd9bdb4-0dfe-4d77-b08f-02fbbffcd892 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | BOOTING | None | None | None | | ff6ec3de-6942-4978-9ea5-d793ff1fdbf8 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | BOOTING | None | None | None | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ +--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+ | b39472aa-04a0-42b8-8c09-4617ea1a0f33 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | BACKUP | 10.94.64.232 | 10.210.64.32 | | 9909b300-d266-4344-b6fc-edc4af651485 | 96223d34-6941-4bc5-881a-ddfd83070739 | BOOTING | None | None | None | | cc123c55-2953-425e-867e-f4057483bb09 | 96223d34-6941-4bc5-881a-ddfd83070739 | BOOTING | None | None | None | +--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+ +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+ | 60af8e11-c6ad-45f0-8e07-1bcc4a477444 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | BACKUP | 10.94.64.74 | None | +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------+ as a result we started to have 3x Master's for them openstack loadbalancer amphora list --loadbalancer 70f98bfb-8045-4d4a-9729-d60b6e141dbf openstack loadbalancer amphora list --loadbalancer 96223d34-6941-4bc5-881a-ddfd83070739 openstack loadbalancer amphora list --loadbalancer f9f859d0-2b31-449b-8723-700bb7f3f3f4 +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ | 3dfa43cd-2719-487e-8ba2-8a4aa0fe7301 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | BACKUP | 10.94.64.112 | 10.210.73.197 | | 3dd9bdb4-0dfe-4d77-b08f-02fbbffcd892 | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | MASTER | 10.94.64.56 | 10.210.73.197 | | 14e8acfe-8510-4629-a583-53ba88f91eca | 70f98bfb-8045-4d4a-9729-d60b6e141dbf | ALLOCATED | MASTER | 10.94.64.30 | 10.210.73.197 | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ +--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+ | b39472aa-04a0-42b8-8c09-4617ea1a0f33 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | BACKUP | 10.94.64.232 | 10.210.64.32 | | 9909b300-d266-4344-b6fc-edc4af651485 | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | MASTER | 10.94.64.151 | 10.210.64.32 | | 9d197234-6411-4069-8be3-a399cc1fbc3a | 96223d34-6941-4bc5-881a-ddfd83070739 | ALLOCATED | MASTER | 10.94.64.212 | 10.210.64.32 | +--------------------------------------+--------------------------------------+-----------+--------+---------------+--------------+ +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ | 60af8e11-c6ad-45f0-8e07-1bcc4a477444 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | BACKUP | 10.94.64.74 | 10.211.193.63 | | d086f283-b9d8-4b23-a782-e0ace769e893 | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | MASTER | 10.94.64.225 | 10.211.193.63 | | 8a90fc7c-eb0a-4635-8ece-7c4704c60f7a | f9f859d0-2b31-449b-8723-700bb7f3f3f4 | ALLOCATED | MASTER | 10.94.64.50 | 10.211.193.63 | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------+ taskflow is enabled in our case, backend is redis can you suggest smth to check or maybe some updates were already done?
