[octavia] Replace broken amphoras
Hi, we had some network issues and now have amphoras which are marked in ERROR state. What we already tried: - failover the amphora - failover the loadbalancer both did not work, got "unable to attach port to (new) amphora". Then we removed the vrrp_port, set the vrrp_port_id to NULL and repeated the amphora failover Reverting Err: "PortID: Null" Then we created a new vrrp_port as described [1] and added the port-id to the vrrp_port_id and the a suitable vrrp_ip field to our ERRORed amphora entry. Restarted failover -> without luck. Currently we have an single STANDALONE amphora configured. Is there a way to trigger octavia to create new "clean" amphoras for MASTER/BACKUP? Thanks, Fabian [1] http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas...
Hi Fabian, Sorry you have run into trouble and we have missed you in the IRC channel. Yeah, that transcript from three years ago isn't going to be much help. A few things we will want to know are: 1. What version of Octavia are you using? 2. Do you have the DNS extension to neutron enabled? 3. When it said "unable to attach port to amphora", can you provide the full error? Was it due to a hostname mismatch error from nova? My guess is you ran into the issue where a port will not attach if the DNS name doesn't match. Our workaround for that accidentally got removed and re-added in https://review.opendev.org/#/c/663277/. Replacing a vrrp_port is tricky, so I'm not surprised you ran into some trouble. Can you please provide the controller worker log output when doing a load balancer failover (let's not use amphora failover here) on paste.openstack.org? You can mark it private and directly reply to me if you have concerns about the log content. All this said, I have recently completely refactored the failover flows recently. This has already merged on the master branch and backports are in process. Michael On Fri, Jul 10, 2020 at 7:07 AM Fabian Zimmermann <dev.faz@gmail.com> wrote:
Hi,
we had some network issues and now have amphoras which are marked in ERROR state.
What we already tried:
- failover the amphora - failover the loadbalancer
both did not work, got "unable to attach port to (new) amphora".
Then we removed the vrrp_port, set the vrrp_port_id to NULL and repeated the amphora failover
Reverting Err: "PortID: Null"
Then we created a new vrrp_port as described [1] and added the port-id to the vrrp_port_id and the a suitable vrrp_ip field to our ERRORed amphora entry.
Restarted failover -> without luck.
Currently we have an single STANDALONE amphora configured.
Is there a way to trigger octavia to create new "clean" amphoras for MASTER/BACKUP?
Thanks,
Fabian
[1]http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas...
Hi, Am Di., 14. Juli 2020 um 02:04 Uhr schrieb Michael Johnson < johnsomor@gmail.com>:
Sorry you have run into trouble and we have missed you in the IRC channel.
Thanks for your great work and support!
Yeah, that transcript from three years ago isn't going to be much help.
Arg.
A few things we will want to know are: 1. What version of Octavia are you using?
3.1.0
2. Do you have the DNS extension to neutron enabled?
yes
3. When it said "unable to attach port to amphora", can you provide the full error? Was it due to a hostname mismatch error from nova?
arg, debug logs got already rotated. I will repeat my debug-session and paste the output. Any suggestions what I should do? Maybe I can already try something different? My guess is you ran into the issue where a port will not attach if the
DNS name doesn't match. Our workaround for that accidentally got removed and re-added in https://review.opendev.org/#/c/663277/.
So, this should already be fixed in stable/rocky. Should upgrading octavia to latest stable/rocky be enough to get my amphoras working again? Replacing a vrrp_port is tricky, so I'm not surprised you ran into
some trouble. Can you please provide the controller worker log output when doing a load balancer failover (let's not use amphora failover here) on paste.openstack.org? You can mark it private and directly reply to me if you have concerns about the log content.
Will provide this asap.
All this said, I have recently completely refactored the failover flows recently. This has already merged on the master branch and backports are in process.
Thanks a lot, Fabian
Hi again, So looking at the patch in question, yes, upgrading to the latest version of Octavia for Rocky, 3.2.2 will resolve the DNS issue going forward. It was originally included in the 3.2.0 release for Rocky, but we would recommend updating to the latest Rocky release, 3.2.2. Michael On Tue, Jul 14, 2020 at 9:44 AM Fabian Zimmermann <dev.faz@gmail.com> wrote:
Hi,
Am Di., 14. Juli 2020 um 02:04 Uhr schrieb Michael Johnson <johnsomor@gmail.com>:
Sorry you have run into trouble and we have missed you in the IRC channel.
Thanks for your great work and support!
Yeah, that transcript from three years ago isn't going to be much help.
Arg.
A few things we will want to know are: 1. What version of Octavia are you using?
3.1.0
2. Do you have the DNS extension to neutron enabled?
yes
3. When it said "unable to attach port to amphora", can you provide the full error? Was it due to a hostname mismatch error from nova?
arg, debug logs got already rotated. I will repeat my debug-session and paste the output.
Any suggestions what I should do? Maybe I can already try something different?
My guess is you ran into the issue where a port will not attach if the DNS name doesn't match. Our workaround for that accidentally got removed and re-added in https://review.opendev.org/#/c/663277/.
So, this should already be fixed in stable/rocky. Should upgrading octavia to latest stable/rocky be enough to get my amphoras working again?
Replacing a vrrp_port is tricky, so I'm not surprised you ran into some trouble. Can you please provide the controller worker log output when doing a load balancer failover (let's not use amphora failover here) on paste.openstack.org? You can mark it private and directly reply to me if you have concerns about the log content.
Will provide this asap.
All this said, I have recently completely refactored the failover flows recently. This has already merged on the master branch and backports are in process.
Thanks a lot,
Fabian
participants (2)
-
Fabian Zimmermann
-
Michael Johnson