Open Stack

I can give it a go, there are also logs of our conversation on
evesdrop here: http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2017-11-02.log.html#t2017-11-02T11:07:45

Short background, they had an infrastructure issue where networking
was out.  This caused the load balancer amphora to be detected as
failed and started the failover process which rebuilds the amphora,
however this process also failed due to the wider infrastructure
issues (Note, this is what I remember from the conversation. Correct
the background if I am wrong). At this point the load balancers would
be in a provisioning status of "ERROR", the amphora instances were
likely deleted (depending on where the infrastructure issue impacted
the failover process), and the amphora db records would be in the
"DELETED" state.  To make their situation worse, the DB cleanup cycle
of the housekeeping process was set low (default is a week) and the
amphora records were purged. This meant the customer data for the VIP
address/ports was also purged.

Recovery:
If the database records were not purged, after the infrastructure
issues are resolved, you can simply go into the octavia database and
issue: update load_balancer set provisioning_status="ACTIVE" where
provisioning_status = "ERROR";
This will restart the health manager expecting the amphora to report
health heartbeats and the failover process with start again rebuilding
the amphora.  We have also recently added an admin API to trigger a
failover manually
(https://developer.openstack.org/api-ref/load-balancer/v2/index.html#failover-a-load-balancer).

If your amphora records have been purged, you are in some pain (don't
do this).  This means that the VIP address and neutron port
information for that VIP address are not available for the failover
process to rebuild.  In this case, before you start the above process
you will need to recreate the amphora records from your logs, either
adding the port information if the ports are still live in neutron, or
creating replacement ports.

Michael

On Wed, Nov 8, 2017 at 2:07 AM,  <mihaela.balas at orange.com> wrote:
> I am also interested how to fix this. If you can describe shortly the procedure.
>
> Thanks,
> Mihaela
>
> -----Original Message-----
> From: Michael Johnson [mailto:johnsomor at gmail.com]
> Sent: Monday, November 06, 2017 6:54 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [octavia] how to recreate amphora instances
>
> I think we helped you get going again in the IRC channel.  Please ping us again in the IRC channel if you need more assistance.
>
> Michael
>
> On Thu, Nov 2, 2017 at 4:42 AM, Kim-Norman Sahm <kim-norman.sahm at noris.de> wrote:
>> Hi,
>>
>> after a rabbitmq problem octavia has removed all amphora instances.
>> the loadbalancers are in provisioning_status "ACTIVE"
>>
>> ~$ neutron lbaas-loadbalancer-list
>> neutron CLI is deprecated and will be removed in the future. Use
>> openstack CLI instead.
>> | 07b41df6-bb75-4502-975a-20140b0832dd | Load Balancer
>> 4                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.18   | ACTIVE              | octavia  |
>> | 25664be7-15cb-426b-ad09-6102afb62b14 | Load Balancer
>> 2                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.7    | ACTIVE              | octavia  |
>> | 927eb754-7c52-4060-b130-1f5e82d92555 | Load Balancer
>> 6                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.17   | ACTIVE              | octavia  |
>> | b4d93c68-89d6-4e4f-b06c-117d4ea933fa | Load Balancer
>> 5                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.24   | ACTIVE              | octavia  |
>> | d7699f8d-2106-42d6-8797-5feb72de6e2e | Load Balancer
>> 1                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.5    | ACTIVE              | octavia  |
>> | dd6114ae-21e9-41bd-b155-325287aed420 | Load Balancer
>> 3                   | 260b0c452e214accaf6cc0e98fb10fc0 |
>> 192.168.1.23   | ACTIVE              | octavia  |
>>
>> How can we trigger octavia to rebuild the amphore instances?
>> I've tried to restart the octavia services but it didn't solved the
>> problem.
>>
>> Best regards
>> Kim
>>
>>
>> Kim-Norman Sahm
>> Cloud & Infrastructure(OCI)
>> noris network AG
>> Thomas-Mann-Straße 16-20
>> 90471 Nürnberg
>> Deutschland
>> Tel +49 911 9352 1433
>> Fax +49 911 9352 100
>>
>> kim-norman.sahm at noris.de
>> https://www.noris.de - Mehr Leistung als Standard
>> Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel Vorsitzender des
>> Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689
>>
>>
>>
>>
>>
>> ______________________________________________________________________
>> ____ OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev