Open Stack

Hi Michael,

Thank you for the detailed explanation. I was in the worst scenario where the database entries were purged and I had to manually re-create the DB entries and the ports. I successfully managed to insert the lines in the database and the amphoras were created.

Thanks a lot for the hints. I also increased the DB cleanup interval to 1 week.

Mihaela

-----Original Message-----
From: Michael Johnson [mailto:johnsomor at gmail.com] 
Sent: Friday, November 10, 2017 3:31 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [octavia] how to recreate amphora instances

I can give it a go, there are also logs of our conversation on evesdrop here: http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2017-11-02.log.html#t2017-11-02T11:07:45

Short background, they had an infrastructure issue where networking was out.  This caused the load balancer amphora to be detected as failed and started the failover process which rebuilds the amphora, however this process also failed due to the wider infrastructure issues (Note, this is what I remember from the conversation. Correct the background if I am wrong). At this point the load balancers would be in a provisioning status of "ERROR", the amphora instances were likely deleted (depending on where the infrastructure issue impacted the failover process), and the amphora db records would be in the "DELETED" state.  To make their situation worse, the DB cleanup cycle of the housekeeping process was set low (default is a week) and the amphora records were purged. This meant the customer data for the VIP address/ports was also purged.

Recovery:
If the database records were not purged, after the infrastructure issues are resolved, you can simply go into the octavia database and
issue: update load_balancer set provisioning_status="ACTIVE" where provisioning_status = "ERROR"; This will restart the health manager expecting the amphora to report health heartbeats and the failover process with start again rebuilding the amphora.  We have also recently added an admin API to trigger a failover manually (https://developer.openstack.org/api-ref/load-balancer/v2/index.html#failover-a-load-balancer).

If your amphora records have been purged, you are in some pain (don't do this).  This means that the VIP address and neutron port information for that VIP address are not available for the failover process to rebuild.  In this case, before you start the above process you will need to recreate the amphora records from your logs, either adding the port information if the ports are still live in neutron, or creating replacement ports.

Michael

On Wed, Nov 8, 2017 at 2:07 AM,  <mihaela.balas at orange.com> wrote:
> I am also interested how to fix this. If you can describe shortly the procedure.
>
> Thanks,
> Mihaela
>
> -----Original Message-----
> From: Michael Johnson [mailto:johnsomor at gmail.com]
> Sent: Monday, November 06, 2017 6:54 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [octavia] how to recreate amphora 
> instances
>
> I think we helped you get going again in the IRC channel.  Please ping us again in the IRC channel if you need more assistance.
>
> Michael
>
> On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm <kim-norman.sahm at noris.de> wrote:
>> Hi,
>>
>> after a rabbitmq problem octavia has removed all amphora instances.
>> the loadbalancers are in provisioning_status "ACTIVE"
>>
>> ~$ neutron lbaas-loadbalancer-list
>> neutron CLI is deprecated and will be removed in the future. Use 
>> openstack CLI instead.
>> | 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
>> 4                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.18   | ACTIVE              | octavia  |
>> | 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
>> 2                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.7    | ACTIVE              | octavia  |
>> | 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
>> 6                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.17   | ACTIVE              | octavia  |
>> | b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
>> 5                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.24   | ACTIVE              | octavia  |
>> | d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
>> 1                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.5    | ACTIVE              | octavia  |
>> | dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
>> 3                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.23   | ACTIVE              | octavia  |
>>
>> How can we trigger octavia to rebuild the amphore instances?
>> I've tried to restart the octavia services but it didn't solved the 
>> problem.
>>
>> Best regards
>> Kim
>>
>>
>> Kim-Norman Sahm
>> Cloud & Infrastructure(OCI)
>> noris network AG
>> Thomas-Mann-Straße 16-20
>> 90471 Nürnberg
>> Deutschland
>> Tel +49 911 9352 1433
>> Fax +49 911 9352 100
>>
>> kim-norman.sahm at noris.de
>> https://www.noris.de - Mehr Leistung als Standard
>> Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel Vorsitzender des
>> Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689
>>
>>
>>
>>
>>
>> _____________________________________________________________________
>> _ ____ OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> ______________________________________________________________________
> ____ OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
> ______________________________________________________________________
> ____ OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.