Octavia group, I created this story to see if I could get some assistance on an Octavia issue I am having with my infra. I did a vanilla install of Openstack Wallaby using the Kolla project with Octavia enabled and auto-configure set. Everything appeared to install fine. When I try to create a load balancer using the horizon console or the cli or through further installs such as cloudfoundry that require a loadbalancer, I get an error from glance about “No image found with tag amphora” even though the image does exist. I hope it is something simple or an oversight on my part. Could I get some ideas or places to look? Story: https://storyboard.openstack.org/#!/story/2009103 Kolla install followed: https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia... cli output : [root@kolla ~]# openstack image list --tag amphora +--------------------------------------+---------------------+--------+ | ID | Name | Status | +--------------------------------------+---------------------+--------+ | 8f7398e7-7912-43b8-8131-e90f21c91ab4 | amphora | active | | 9bf14389-8521-4fb7-a7db-925f4dea1bf3 | amphora-x64-haproxy | active | +--------------------------------------+---------------------+--------+ [root@kolla ~]# My Infra scripts (relevant): https://github.com/mephmanx/openstack-scripts/blob/32363ef5753fdec381f713c74... I am admin of said infra so if anyone has time or availability (or interest) to log in and take a look I would be happy to provide creds. Thank you for any time that you could provide or offer!
This is a kolla bug most likely. This could be caused by a few possible issues in kolla: 1. The images are owned by the wrong project. Kolla may deploy Octavia under a special service project. The image kolla uploads to glance must be uploaded under the service project as owner. A simple test to see if this is the issue would be to set the image as "public" such that all users of the cloud can see it. If a load balancer can be created after that change, the images are loaded under a project that Octavia cannot access. 2. The other possibility is kolla configured an alternate tag name. Check the [controller_worker] amp_image_tag setting [1] in octavia.conf. The images must be tagged with the same name as is configured in the octavia.conf for the controllers. Let us know either way (and ideally open a bug for kolla) what you find out. Michael [1] https://docs.openstack.org/octavia/latest/configuration/configref.html#contr... On Fri, Aug 6, 2021 at 5:52 AM Chris Lyons <chris@lyonsgroup.family> wrote:
Octavia group,
I created this story to see if I could get some assistance on an Octavia issue I am having with my infra. I did a vanilla install of Openstack Wallaby using the Kolla project with Octavia enabled and auto-configure set. Everything appeared to install fine. When I try to create a load balancer using the horizon console or the cli or through further installs such as cloudfoundry that require a loadbalancer, I get an error from glance about “No image found with tag amphora” even though the image does exist. I hope it is something simple or an oversight on my part. Could I get some ideas or places to look?
Story:
https://storyboard.openstack.org/#!/story/2009103
Kolla install followed:
https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia...
cli output :
[root@kolla ~]# openstack image list --tag amphora
+--------------------------------------+---------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------+--------+
| 8f7398e7-7912-43b8-8131-e90f21c91ab4 | amphora | active |
| 9bf14389-8521-4fb7-a7db-925f4dea1bf3 | amphora-x64-haproxy | active |
+--------------------------------------+---------------------+--------+
[root@kolla ~]#
My Infra scripts (relevant):
https://github.com/mephmanx/openstack-scripts/blob/32363ef5753fdec381f713c74...
I am admin of said infra so if anyone has time or availability (or interest) to log in and take a look I would be happy to provide creds.
Thank you for any time that you could provide or offer!
Both of those appear ok…. I do have the amphora image set to public and I see this in the Octavia.conf file: [controller_worker] amp_ssh_key_name = octavia_ssh_key amp_image_tag = amphora amp_image_owner_id = 4269b3cd5452416e8153f5b4f5adcf0c amp_boot_network_list = adfa01a9-55bd-42ea-b849-81060d5d7c09 amp_secgroup_list = ecf269e2-42fc-477b-9ce1-38d60c1d8d5d amp_flavor_id = c854d1e7-885f-4f7a-88a3-79728b561830 client_ca = /etc/octavia/certs/client_ca.cert.pem network_driver = allowed_address_pairs_driver compute_driver = compute_nova_driver amphora_driver = amphora_haproxy_rest_driver amp_active_retries = 100 amp_active_wait_sec = 2 loadbalancer_topology = SINGLE From: Michael Johnson <johnsomor@gmail.com> Date: Friday, August 6, 2021 at 10:04 AM To: Chris Lyons <chris@lyonsgroup.family> Cc: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: Re: Octavia This is a kolla bug most likely. This could be caused by a few possible issues in kolla: 1. The images are owned by the wrong project. Kolla may deploy Octavia under a special service project. The image kolla uploads to glance must be uploaded under the service project as owner. A simple test to see if this is the issue would be to set the image as "public" such that all users of the cloud can see it. If a load balancer can be created after that change, the images are loaded under a project that Octavia cannot access. 2. The other possibility is kolla configured an alternate tag name. Check the [controller_worker] amp_image_tag setting [1] in octavia.conf. The images must be tagged with the same name as is configured in the octavia.conf for the controllers. Let us know either way (and ideally open a bug for kolla) what you find out. Michael [1] https://docs.openstack.org/octavia/latest/configuration/configref.html#contr... On Fri, Aug 6, 2021 at 5:52 AM Chris Lyons <chris@lyonsgroup.family> wrote:
Octavia group,
I created this story to see if I could get some assistance on an Octavia issue I am having with my infra. I did a vanilla install of Openstack Wallaby using the Kolla project with Octavia enabled and auto-configure set. Everything appeared to install fine. When I try to create a load balancer using the horizon console or the cli or through further installs such as cloudfoundry that require a loadbalancer, I get an error from glance about “No image found with tag amphora” even though the image does exist. I hope it is something simple or an oversight on my part. Could I get some ideas or places to look?
Story:
https://storyboard.openstack.org/#!/story/2009103
Kolla install followed:
https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia...
cli output :
[root@kolla ~]# openstack image list --tag amphora
+--------------------------------------+---------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------+--------+
| 8f7398e7-7912-43b8-8131-e90f21c91ab4 | amphora | active |
| 9bf14389-8521-4fb7-a7db-925f4dea1bf3 | amphora-x64-haproxy | active |
+--------------------------------------+---------------------+--------+
[root@kolla ~]#
My Infra scripts (relevant):
https://github.com/mephmanx/openstack-scripts/blob/32363ef5753fdec381f713c74...
I am admin of said infra so if anyone has time or availability (or interest) to log in and take a look I would be happy to provide creds.
Thank you for any time that you could provide or offer!
This email has been scanned by Inbound Shield™.
Hmm, so not the usual suspects. Can you check the glance logs and see what error it is reporting when Octavia attempts to access the image? Also, confirm that the settings kolla provided for the glance endpoint are valid in the octavia.conf. [glance] section documented here: https://docs.openstack.org/octavia/latest/configuration/configref.html#glanc... They may not be configured, which is ok as it will look them up in the catalog if they are not specified. Also, double check that the image driver is set to glance in [controller_worker] image_driver. https://docs.openstack.org/octavia/latest/configuration/configref.html#contr... Though from the logs this setting looks correct. The default setting if not specified is the glance driver. If you want to chat in more real-time, we are on IRC OFTC in channel #openstack-lbaas. Michael On Fri, Aug 6, 2021 at 7:07 AM Chris Lyons <chris@lyonsgroup.family> wrote:
Both of those appear ok…. I do have the amphora image set to public and I see this in the Octavia.conf file:
[controller_worker]
amp_ssh_key_name = octavia_ssh_key
amp_image_tag = amphora
amp_image_owner_id = 4269b3cd5452416e8153f5b4f5adcf0c
amp_boot_network_list = adfa01a9-55bd-42ea-b849-81060d5d7c09
amp_secgroup_list = ecf269e2-42fc-477b-9ce1-38d60c1d8d5d
amp_flavor_id = c854d1e7-885f-4f7a-88a3-79728b561830
client_ca = /etc/octavia/certs/client_ca.cert.pem
network_driver = allowed_address_pairs_driver
compute_driver = compute_nova_driver
amphora_driver = amphora_haproxy_rest_driver
amp_active_retries = 100
amp_active_wait_sec = 2
loadbalancer_topology = SINGLE
From: Michael Johnson <johnsomor@gmail.com> Date: Friday, August 6, 2021 at 10:04 AM To: Chris Lyons <chris@lyonsgroup.family> Cc: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: Re: Octavia
This is a kolla bug most likely.
This could be caused by a few possible issues in kolla:
1. The images are owned by the wrong project. Kolla may deploy Octavia under a special service project. The image kolla uploads to glance must be uploaded under the service project as owner. A simple test to see if this is the issue would be to set the image as "public" such that all users of the cloud can see it. If a load balancer can be created after that change, the images are loaded under a project that Octavia cannot access. 2. The other possibility is kolla configured an alternate tag name. Check the [controller_worker] amp_image_tag setting [1] in octavia.conf. The images must be tagged with the same name as is configured in the octavia.conf for the controllers.
Let us know either way (and ideally open a bug for kolla) what you find out.
Michael
[1] https://docs.openstack.org/octavia/latest/configuration/configref.html#contr...
On Fri, Aug 6, 2021 at 5:52 AM Chris Lyons <chris@lyonsgroup.family> wrote:
Octavia group,
I created this story to see if I could get some assistance on an Octavia issue I am having with my infra. I did a vanilla install of Openstack Wallaby using the Kolla project with Octavia enabled and auto-configure set. Everything appeared to install fine. When I try to create a load balancer using the horizon console or the cli or through further installs such as cloudfoundry that require a loadbalancer, I get an error from glance about “No image found with tag amphora” even though the image does exist. I hope it is something simple or an oversight on my part. Could I get some ideas or places to look?
Story:
https://storyboard.openstack.org/#!/story/2009103
Kolla install followed:
https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia...
cli output :
[root@kolla ~]# openstack image list --tag amphora
+--------------------------------------+---------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------+--------+
| 8f7398e7-7912-43b8-8131-e90f21c91ab4 | amphora | active |
| 9bf14389-8521-4fb7-a7db-925f4dea1bf3 | amphora-x64-haproxy | active |
+--------------------------------------+---------------------+--------+
[root@kolla ~]#
My Infra scripts (relevant):
https://github.com/mephmanx/openstack-scripts/blob/32363ef5753fdec381f713c74...
I am admin of said infra so if anyone has time or availability (or interest) to log in and take a look I would be happy to provide creds.
Thank you for any time that you could provide or offer!
This email has been scanned by Inbound Shield™.
I had that problem when setting up a Victoria Kolla cluster, and it turned out that the image was owned by the wrong project, as suspected by Michael. I changed the owner in Glance, and it worked. The owner must be 4269b3cd5452416e8153f5b4f5adcf0c in Chris' case (openstack image set --project 4269b3cd5452416e8153f5b4f5adcf0c amphora-image-name). I don't think this is a Kolla problem, as Kolla leaves the responsibility of building and uploading the image to the administrator (or it did so at Victoria). Bernd Bausch On 2021/08/06 11:07 PM, Chris Lyons wrote:
Both of those appear ok…. I do have the amphora image set to public and I see this in the Octavia.conf file:
[controller_worker]
amp_ssh_key_name = octavia_ssh_key
amp_image_tag = amphora
amp_image_owner_id = 4269b3cd5452416e8153f5b4f5adcf0c
amp_boot_network_list = adfa01a9-55bd-42ea-b849-81060d5d7c09
amp_secgroup_list = ecf269e2-42fc-477b-9ce1-38d60c1d8d5d
amp_flavor_id = c854d1e7-885f-4f7a-88a3-79728b561830
client_ca = /etc/octavia/certs/client_ca.cert.pem
network_driver = allowed_address_pairs_driver
compute_driver = compute_nova_driver
amphora_driver = amphora_haproxy_rest_driver
amp_active_retries = 100
amp_active_wait_sec = 2
loadbalancer_topology = SINGLE
*From: *Michael Johnson <johnsomor@gmail.com> *Date: *Friday, August 6, 2021 at 10:04 AM *To: *Chris Lyons <chris@lyonsgroup.family> *Cc: *openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> *Subject: *Re: Octavia
This is a kolla bug most likely.
This could be caused by a few possible issues in kolla:
1. The images are owned by the wrong project. Kolla may deploy Octavia under a special service project. The image kolla uploads to glance must be uploaded under the service project as owner. A simple test to see if this is the issue would be to set the image as "public" such that all users of the cloud can see it. If a load balancer can be created after that change, the images are loaded under a project that Octavia cannot access. 2. The other possibility is kolla configured an alternate tag name. Check the [controller_worker] amp_image_tag setting [1] in octavia.conf. The images must be tagged with the same name as is configured in the octavia.conf for the controllers.
Let us know either way (and ideally open a bug for kolla) what you find out.
Michael
[1] https://docs.openstack.org/octavia/latest/configuration/configref.html#contr... <https://docs.openstack.org/octavia/latest/configuration/configref.html#controller_worker.amp_image_tag>
On Fri, Aug 6, 2021 at 5:52 AM Chris Lyons <chris@lyonsgroup.family> wrote:
Octavia group,
I created this story to see if I could get some assistance on an
Octavia issue I am having with my infra. I did a vanilla install of Openstack Wallaby using the Kolla project with Octavia enabled and auto-configure set. Everything appeared to install fine. When I try to create a load balancer using the horizon console or the cli or through further installs such as cloudfoundry that require a loadbalancer, I get an error from glance about “No image found with tag amphora” even though the image does exist. I hope it is something simple or an oversight on my part. Could I get some ideas or places to look?
Story:
<https://storyboard.openstack.org/#!/story/2009103>
Kolla install followed:
https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia... <https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia.html>
cli output :
[root@kolla ~]# openstack image list --tag amphora
+--------------------------------------+---------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------+--------+
| 8f7398e7-7912-43b8-8131-e90f21c91ab4 | amphora | active |
| 9bf14389-8521-4fb7-a7db-925f4dea1bf3 | amphora-x64-haproxy | active |
+--------------------------------------+---------------------+--------+
[root@kolla ~]#
My Infra scripts (relevant):
https://github.com/mephmanx/openstack-scripts/blob/32363ef5753fdec381f713c74... <https://github.com/mephmanx/openstack-scripts/blob/32363ef5753fdec381f713c747c90a5c14b3ae72/kolla.sh#L340>
I am admin of said infra so if anyone has time or availability (or
interest) to log in and take a look I would be happy to provide creds.
Thank you for any time that you could provide or offer!
This email has been scanned by Inbound Shield™.
I think I might have solved it. It looks like during the kolla install a “Octavia-openrc.sh” script is created which sets up the env to use an account called “service”. After running that and re-uploading the amphora image I am now able to see amphora instances get created. Thank you Bernd for the suggestion that you gve! It caused me to look around in the permissions area and found the solution! From: Bernd Bausch <berndbausch@gmail.com> Date: Friday, August 6, 2021 at 8:21 PM To: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>, Chris Lyons <chris@lyonsgroup.family> Subject: Re: Octavia I had that problem when setting up a Victoria Kolla cluster, and it turned out that the image was owned by the wrong project, as suspected by Michael. I changed the owner in Glance, and it worked. The owner must be 4269b3cd5452416e8153f5b4f5adcf0c in Chris' case (openstack image set --project 4269b3cd5452416e8153f5b4f5adcf0c amphora-image-name). I don't think this is a Kolla problem, as Kolla leaves the responsibility of building and uploading the image to the administrator (or it did so at Victoria). Bernd Bausch On 2021/08/06 11:07 PM, Chris Lyons wrote: Both of those appear ok…. I do have the amphora image set to public and I see this in the Octavia.conf file: [controller_worker] amp_ssh_key_name = octavia_ssh_key amp_image_tag = amphora amp_image_owner_id = 4269b3cd5452416e8153f5b4f5adcf0c amp_boot_network_list = adfa01a9-55bd-42ea-b849-81060d5d7c09 amp_secgroup_list = ecf269e2-42fc-477b-9ce1-38d60c1d8d5d amp_flavor_id = c854d1e7-885f-4f7a-88a3-79728b561830 client_ca = /etc/octavia/certs/client_ca.cert.pem network_driver = allowed_address_pairs_driver compute_driver = compute_nova_driver amphora_driver = amphora_haproxy_rest_driver amp_active_retries = 100 amp_active_wait_sec = 2 loadbalancer_topology = SINGLE From: Michael Johnson <johnsomor@gmail.com><mailto:johnsomor@gmail.com> Date: Friday, August 6, 2021 at 10:04 AM To: Chris Lyons <chris@lyonsgroup.family><mailto:chris@lyonsgroup.family> Cc: openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> <openstack-discuss@lists.openstack.org><mailto:openstack-discuss@lists.openstack.org> Subject: Re: Octavia This is a kolla bug most likely. This could be caused by a few possible issues in kolla: 1. The images are owned by the wrong project. Kolla may deploy Octavia under a special service project. The image kolla uploads to glance must be uploaded under the service project as owner. A simple test to see if this is the issue would be to set the image as "public" such that all users of the cloud can see it. If a load balancer can be created after that change, the images are loaded under a project that Octavia cannot access. 2. The other possibility is kolla configured an alternate tag name. Check the [controller_worker] amp_image_tag setting [1] in octavia.conf. The images must be tagged with the same name as is configured in the octavia.conf for the controllers. Let us know either way (and ideally open a bug for kolla) what you find out. Michael [1] https://docs.openstack.org/octavia/latest/configuration/configref.html#contr... On Fri, Aug 6, 2021 at 5:52 AM Chris Lyons <chris@lyonsgroup.family><mailto:chris@lyonsgroup.family> wrote:
Octavia group,
I created this story to see if I could get some assistance on an Octavia issue I am having with my infra. I did a vanilla install of Openstack Wallaby using the Kolla project with Octavia enabled and auto-configure set. Everything appeared to install fine. When I try to create a load balancer using the horizon console or the cli or through further installs such as cloudfoundry that require a loadbalancer, I get an error from glance about “No image found with tag amphora” even though the image does exist. I hope it is something simple or an oversight on my part. Could I get some ideas or places to look?
Story:
https://storyboard.openstack.org/#!/story/2009103
Kolla install followed:
https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia...
cli output :
[root@kolla ~]# openstack image list --tag amphora
+--------------------------------------+---------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------+--------+
| 8f7398e7-7912-43b8-8131-e90f21c91ab4 | amphora | active |
| 9bf14389-8521-4fb7-a7db-925f4dea1bf3 | amphora-x64-haproxy | active |
+--------------------------------------+---------------------+--------+
[root@kolla ~]#
My Infra scripts (relevant):
https://github.com/mephmanx/openstack-scripts/blob/32363ef5753fdec381f713c74...
I am admin of said infra so if anyone has time or availability (or interest) to log in and take a look I would be happy to provide creds.
Thank you for any time that you could provide or offer!
This email has been scanned by Inbound Shield™. This email has been scanned by Inbound Shield™.
Well…it gets. A lot further…. I see this error now…. Im looking around to see if it’s a security group missing or if there is some other setting I missed. Im not seeing any scripts to prep the env…usually there is something like that if it’s a security group…anyone know? |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e'}, 'provides': 'b6a7018d-d436-477b-8aad-b1b53b1e6cde'} |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow' |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e', 'flavor': {}, 'availability_zone': {}, 'server_group_id': None}, 'provides': None} |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow' |__Atom 'octavia.controller.worker.v1.tasks.network_tasks.GetSubnetFromVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f10d00d48d0>}, 'provides': <octavia.network.data_models.Subnet object at 0x7f10d0077a58>} |__Atom 'octavia.controller.worker.v1.tasks.network_tasks.UpdateVIPSecurityGroup' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e'}, 'provides': '3512c022-9189-4709-be49-46555d7ce8c6'} |__Atom 'octavia.controller.worker.v1.tasks.database_tasks.UpdateVIPAfterAllocation' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e', 'vip': <octavia.common.data_models.Vip object at 0x7f10d0118d68>}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f10d00d48d0>} |__Atom 'octavia.controller.worker.v1.tasks.network_tasks.AllocateVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f10d1966b38>}, 'provides': <octavia.common.data_models.Vip object at 0x7f10d0118d68>} |__Atom 'reload-lb-before-allocate-vip' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f10d1966b38>} |__Atom 'octavia.controller.worker.v1.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e'}, 'provides': None} |__Flow 'octavia-create-loadbalancer-flow': octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last): 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 424, in execute 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amp_info = self.amphora_driver.get_info(amphora) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 373, in get_info 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amphora, raise_retry_exception=raise_retry_exception) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 106, in _populate_amphora_api_version 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception)['api_version'] 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 744, in get_api_version 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 738, in request 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise driver_except.TimeOutException() 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker 2021-08-08 21:22:35.995 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amp-compute-connectivity-wait' (7f8c1119-9020-4fb8-ae98-1495875a3dc2) transitioned into state 'REVERTED' from state 'REVERTING' 2021-08-08 21:22:36.003 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-info' (ca7b4681-3fe6-4834-b7ee-2fb2dbe01c3c) transitioned into state 'REVERTED' from state 'REVERTING' 2021-08-08 21:22:36.011 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait' (2ef36bd4-15be-468f-9a8c-f6b60728da6c) transitioned into state 'REVERTED' from state 'REVERTING' 2021-08-08 21:22:36.017 27 WARNING octavia.controller.worker.v1.tasks.database_tasks [-] Reverting mark amphora booting in DB for amp id b6a7018d-d436-477b-8aad-b1b53b1e6cde and compute id c93393df-a39a-486e-812d-73652209fcff 2021-08-08 21:22:36.049 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb' (6aac42e0-fe77-43b8-9323-e0f79304d9d6) transitioned into state 'REVERTED' from state 'REVERTING' 2021-08-08 21:22:36.056 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid' (592da2b6-ab90-4292-a854-c6e2a679dabf) transitioned into state 'REVERTED' from state 'REVERTING' 2021-08-08 21:22:36.061 27 WARNING octavia.controller.worker.v1.tasks.compute_tasks [-] Reverting compute create for amphora with id b6a7018d-d436-477b-8aad-b1b53b1e6cde and compute id: c93393df-a39a-486e-812d-73652209fcff 2021-08-08 21:22:36.576 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create' (59401bf8-9191-4ac1-b4f3-b551a139df9d) transitioned into state 'REVERTED' from state 'REVERTING' 2021-08-08 21:22:36.582 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration' (9e731d93-8cb8-46cb-9be4-073e1579a5f8) transitioned into state 'REVERTED' from state 'REVERTING' From: Chris Lyons <chris@lyonsgroup.family> Date: Sunday, August 8, 2021 at 8:56 PM To: Bernd Bausch <berndbausch@gmail.com>, openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: Re: Octavia I think I might have solved it. It looks like during the kolla install a “Octavia-openrc.sh” script is created which sets up the env to use an account called “service”. After running that and re-uploading the amphora image I am now able to see amphora instances get created. Thank you Bernd for the suggestion that you gve! It caused me to look around in the permissions area and found the solution! From: Bernd Bausch <berndbausch@gmail.com> Date: Friday, August 6, 2021 at 8:21 PM To: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>, Chris Lyons <chris@lyonsgroup.family> Subject: Re: Octavia I had that problem when setting up a Victoria Kolla cluster, and it turned out that the image was owned by the wrong project, as suspected by Michael. I changed the owner in Glance, and it worked. The owner must be 4269b3cd5452416e8153f5b4f5adcf0c in Chris' case (openstack image set --project 4269b3cd5452416e8153f5b4f5adcf0c amphora-image-name). I don't think this is a Kolla problem, as Kolla leaves the responsibility of building and uploading the image to the administrator (or it did so at Victoria). Bernd Bausch On 2021/08/06 11:07 PM, Chris Lyons wrote: Both of those appear ok…. I do have the amphora image set to public and I see this in the Octavia.conf file: [controller_worker] amp_ssh_key_name = octavia_ssh_key amp_image_tag = amphora amp_image_owner_id = 4269b3cd5452416e8153f5b4f5adcf0c amp_boot_network_list = adfa01a9-55bd-42ea-b849-81060d5d7c09 amp_secgroup_list = ecf269e2-42fc-477b-9ce1-38d60c1d8d5d amp_flavor_id = c854d1e7-885f-4f7a-88a3-79728b561830 client_ca = /etc/octavia/certs/client_ca.cert.pem network_driver = allowed_address_pairs_driver compute_driver = compute_nova_driver amphora_driver = amphora_haproxy_rest_driver amp_active_retries = 100 amp_active_wait_sec = 2 loadbalancer_topology = SINGLE From: Michael Johnson <johnsomor@gmail.com><mailto:johnsomor@gmail.com> Date: Friday, August 6, 2021 at 10:04 AM To: Chris Lyons <chris@lyonsgroup.family><mailto:chris@lyonsgroup.family> Cc: openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> <openstack-discuss@lists.openstack.org><mailto:openstack-discuss@lists.openstack.org> Subject: Re: Octavia This is a kolla bug most likely. This could be caused by a few possible issues in kolla: 1. The images are owned by the wrong project. Kolla may deploy Octavia under a special service project. The image kolla uploads to glance must be uploaded under the service project as owner. A simple test to see if this is the issue would be to set the image as "public" such that all users of the cloud can see it. If a load balancer can be created after that change, the images are loaded under a project that Octavia cannot access. 2. The other possibility is kolla configured an alternate tag name. Check the [controller_worker] amp_image_tag setting [1] in octavia.conf. The images must be tagged with the same name as is configured in the octavia.conf for the controllers. Let us know either way (and ideally open a bug for kolla) what you find out. Michael [1] https://docs.openstack.org/octavia/latest/configuration/configref.html#contr... On Fri, Aug 6, 2021 at 5:52 AM Chris Lyons <chris@lyonsgroup.family><mailto:chris@lyonsgroup.family> wrote:
Octavia group,
I created this story to see if I could get some assistance on an Octavia issue I am having with my infra. I did a vanilla install of Openstack Wallaby using the Kolla project with Octavia enabled and auto-configure set. Everything appeared to install fine. When I try to create a load balancer using the horizon console or the cli or through further installs such as cloudfoundry that require a loadbalancer, I get an error from glance about “No image found with tag amphora” even though the image does exist. I hope it is something simple or an oversight on my part. Could I get some ideas or places to look?
Story:
https://storyboard.openstack.org/#!/story/2009103
Kolla install followed:
https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia...
cli output :
[root@kolla ~]# openstack image list --tag amphora
+--------------------------------------+---------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------+--------+
| 8f7398e7-7912-43b8-8131-e90f21c91ab4 | amphora | active |
| 9bf14389-8521-4fb7-a7db-925f4dea1bf3 | amphora-x64-haproxy | active |
+--------------------------------------+---------------------+--------+
[root@kolla ~]#
My Infra scripts (relevant):
https://github.com/mephmanx/openstack-scripts/blob/32363ef5753fdec381f713c74...
I am admin of said infra so if anyone has time or availability (or interest) to log in and take a look I would be happy to provide creds.
Thank you for any time that you could provide or offer!
This email has been scanned by Inbound Shield™. This email has been scanned by Inbound Shield™.
The stacktrace shows that Octavia can't reach an Amphora. I suppose you get this log when trying to create a loadbalancer? If so, most likely the Amphora management network is not set up correctly. The difficulty is that Octavia workers, which are /processes /running on the controllers, need to have access to the same management network as the Amphora /instances/. If you implement the management network as a normal tenant network, some non-trivial manual Openvswitch configuration is required. See https://docs.openstack.org/octavia/latest/install/install-ubuntu.html#instal... for instructions. In production settings, usually a VLAN is used, which is easy to access from controller processes. I succeeded running Octavia on a Kolla cluster with three-way controllers, with a tenant network (VXLAN-based) for amphora management. My notes are attached (they are tailored to my configuration, of course). Bernd. On 2021/08/09 10:57 AM, Chris Lyons wrote: Well…it gets. A lot further…. I see this error now…. Im looking around to see if it’s a security group missing or if there is some other setting I missed. Im not seeing any scripts to prep the env…usually there is something like that if it’s a security group…anyone know? ... 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last): 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 424, in execute 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amp_info = self.amphora_driver.get_info(amphora) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 373, in get_info 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amphora, raise_retry_exception=raise_retry_exception) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 106, in _populate_amphora_api_version 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception)['api_version'] 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 744, in get_api_version 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 738, in request 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise driver_except.TimeOutException() 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out ...
Thank you! I am trying the path of adding a separate physical (although its really virtual..) network that will be exclusively for lb-mgmt….that would work also, correct? I added a port to my virtual router and the added a corresponding port to each openstack vm. It sounds like only control and compute nodes need it but I was going to add it to everything to start and then eliminate one by one once it worked. Would that approach also work? From: Bernd Bausch <berndbausch@gmail.com> Date: Tuesday, August 10, 2021 at 2:39 AM To: Chris Lyons <chris@lyonsgroup.family>, openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: Re: Octavia The stacktrace shows that Octavia can't reach an Amphora. I suppose you get this log when trying to create a loadbalancer? If so, most likely the Amphora management network is not set up correctly. The difficulty is that Octavia workers, which are processes running on the controllers, need to have access to the same management network as the Amphora instances. If you implement the management network as a normal tenant network, some non-trivial manual Openvswitch configuration is required. See https://docs.openstack.org/octavia/latest/install/install-ubuntu.html#instal... for instructions. In production settings, usually a VLAN is used, which is easy to access from controller processes. I succeeded running Octavia on a Kolla cluster with three-way controllers, with a tenant network (VXLAN-based) for amphora management. My notes are attached (they are tailored to my configuration, of course). Bernd. On 2021/08/09 10:57 AM, Chris Lyons wrote: Well…it gets. A lot further…. I see this error now…. Im looking around to see if it’s a security group missing or if there is some other setting I missed. Im not seeing any scripts to prep the env…usually there is something like that if it’s a security group…anyone know? ... 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last): 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 424, in execute 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amp_info = self.amphora_driver.get_info(amphora) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 373, in get_info 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amphora, raise_retry_exception=raise_retry_exception) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 106, in _populate_amphora_api_version 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception)['api_version'] 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 744, in get_api_version 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 738, in request 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise driver_except.TimeOutException() 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out ... This email has been scanned by Inbound Shield™.
A flat network for LB management should work as well. However, the LB management network has nothing to do with your VMs (except the amphorae), and it's unclear to me what ports you are adding to them and the Neutron router. There is no router for a flat network. You seem to imply that your machines (controllers, computes) are virtual. My experience with Kolla is also in a virtual environment. I had to abandon VLAN or flat LB management networks because I was not competent enough to solve the networking problems outside of Kolla and went for the tenant network solution that I shared. Bernd On 2021/08/10 9:31 PM, Chris Lyons wrote:
Thank you! I am trying the path of adding a separate physical (although its really virtual..) network that will be exclusively for lb-mgmt….that would work also, correct? I added a port to my virtual router and the added a corresponding port to each openstack vm. It sounds like only control and compute nodes need it but I was going to add it to everything to start and then eliminate one by one once it worked.
Would that approach also work?
*From: *Bernd Bausch <berndbausch@gmail.com> *Date: *Tuesday, August 10, 2021 at 2:39 AM *To: *Chris Lyons <chris@lyonsgroup.family>, openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> *Subject: *Re: Octavia
The stacktrace shows that Octavia can't reach an Amphora. I suppose you get this log when trying to create a loadbalancer? If so, most likely the Amphora management network is not set up correctly.
The difficulty is that Octavia workers, which are /processes /running on the controllers, need to have access to the same management network as the Amphora /instances/. If you implement the management network as a normal tenant network, some non-trivial manual Openvswitch configuration is required. See https://docs.openstack.org/octavia/latest/install/install-ubuntu.html#instal... <https://docs.openstack.org/octavia/latest/install/install-ubuntu.html#install-and-configure-components> for instructions. In production settings, usually a VLAN is used, which is easy to access from controller processes.
I succeeded running Octavia on a Kolla cluster with three-way controllers, with a tenant network (VXLAN-based) for amphora management. My notes are attached (they are tailored to my configuration, of course).
Bernd.
On 2021/08/09 10:57 AM, Chris Lyons wrote:
Well…it gets. A lot further…. I see this error now…. Im looking around to see if it’s a security group missing or if there is some other setting I missed. Im not seeing any scripts to prep the env…usually there is something like that if it’s a security group…anyone know?
...
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last):
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments)
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 424, in execute
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amp_info = self.amphora_driver.get_info(amphora)
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 373, in get_info
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amphora, raise_retry_exception=raise_retry_exception)
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 106, in _populate_amphora_api_version
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception)['api_version']
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 744, in get_api_version
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception)
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 738, in request
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise driver_except.TimeOutException()
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out
...
/This email has been scanned by Inbound Shield™./
I got close with my approach but I wasn’t sure how to get the network interface I created for Octavia to map to physnet2 to tell the kolla Octavia LB mgmt. subnet to use it. So….Ill go with your approach. It sounds like you have an all-in-one setup….Im going through the steps with a multinode setup. How do I find the DHCP servers port? This is my output from this command. I used network01 host as that has the OVS stuff on it. Network02 also has OVS… [root@kolla ~]# ssh network01 sudo docker exec openvswitch_vswitchd ovsdb-client dump unix:/var/run/openvswitch/db.sock Open_vSwitch Port name tag Port table name tag -------------- --- br-ex [] br-int [] br-tun [] eth1 [] int-br-ex [] patch-int [] patch-tun [] phy-br-ex [] qg-dd91a0d9-e7 2 qr-1fb72495-df 3 qr-a7bcbd91-14 4 qr-cd05b21c-b7 6 qr-e9938869-17 5 tap12d3571c-7f 1 tapd6c55c73-05 4 tape62e9ed6-a7 3 vxlan-0a00c817 [] vxlan-0a00c81b [] [root@kolla ~]# From: Bernd Bausch <berndbausch@gmail.com> Date: Tuesday, August 10, 2021 at 9:38 PM To: Chris Lyons <chris@lyonsgroup.family>, openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: Octavia A flat network for LB management should work as well. However, the LB management network has nothing to do with your VMs (except the amphorae), and it's unclear to me what ports you are adding to them and the Neutron router. There is no router for a flat network. You seem to imply that your machines (controllers, computes) are virtual. My experience with Kolla is also in a virtual environment. I had to abandon VLAN or flat LB management networks because I was not competent enough to solve the networking problems outside of Kolla and went for the tenant network solution that I shared. Bernd On 2021/08/10 9:31 PM, Chris Lyons wrote: Thank you! I am trying the path of adding a separate physical (although its really virtual..) network that will be exclusively for lb-mgmt….that would work also, correct? I added a port to my virtual router and the added a corresponding port to each openstack vm. It sounds like only control and compute nodes need it but I was going to add it to everything to start and then eliminate one by one once it worked. Would that approach also work? From: Bernd Bausch <berndbausch@gmail.com><mailto:berndbausch@gmail.com> Date: Tuesday, August 10, 2021 at 2:39 AM To: Chris Lyons <chris@lyonsgroup.family><mailto:chris@lyonsgroup.family>, openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> <openstack-discuss@lists.openstack.org><mailto:openstack-discuss@lists.openstack.org> Subject: Re: Octavia The stacktrace shows that Octavia can't reach an Amphora. I suppose you get this log when trying to create a loadbalancer? If so, most likely the Amphora management network is not set up correctly. The difficulty is that Octavia workers, which are processes running on the controllers, need to have access to the same management network as the Amphora instances. If you implement the management network as a normal tenant network, some non-trivial manual Openvswitch configuration is required. See https://docs.openstack.org/octavia/latest/install/install-ubuntu.html#instal... for instructions. In production settings, usually a VLAN is used, which is easy to access from controller processes. I succeeded running Octavia on a Kolla cluster with three-way controllers, with a tenant network (VXLAN-based) for amphora management. My notes are attached (they are tailored to my configuration, of course). Bernd. On 2021/08/09 10:57 AM, Chris Lyons wrote: Well…it gets. A lot further…. I see this error now…. Im looking around to see if it’s a security group missing or if there is some other setting I missed. Im not seeing any scripts to prep the env…usually there is something like that if it’s a security group…anyone know? ... 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last): 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 424, in execute 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amp_info = self.amphora_driver.get_info(amphora) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 373, in get_info 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amphora, raise_retry_exception=raise_retry_exception) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 106, in _populate_amphora_api_version 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception)['api_version'] 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 744, in get_api_version 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception) 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 738, in request 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise driver_except.TimeOutException() 2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out ... This email has been scanned by Inbound Shield™.
On Mon, Aug 9, 2021, 11:27 AM Chris Lyons <chris@lyonsgroup.family> wrote:
Well…it gets. A lot further…. I see this error now…. Im looking around to see if it’s a security group missing or if there is some other setting I missed. Im not seeing any scripts to prep the env…usually there is something like that if it’s a security group…anyone know?
|__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e'}, 'provides': 'b6a7018d-d436-477b-8aad-b1b53b1e6cde'}
|__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow'
|__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e', 'flavor': {}, 'availability_zone': {}, 'server_group_id': None}, 'provides': None}
|__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
|__Atom 'octavia.controller.worker.v1.tasks.network_tasks.GetSubnetFromVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f10d00d48d0>}, 'provides': <octavia.network.data_models.Subnet object at 0x7f10d0077a58>}
|__Atom 'octavia.controller.worker.v1.tasks.network_tasks.UpdateVIPSecurityGroup' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e'}, 'provides': '3512c022-9189-4709-be49-46555d7ce8c6'}
|__Atom 'octavia.controller.worker.v1.tasks.database_tasks.UpdateVIPAfterAllocation' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e', 'vip': <octavia.common.data_models.Vip object at 0x7f10d0118d68>}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f10d00d48d0>}
|__Atom 'octavia.controller.worker.v1.tasks.network_tasks.AllocateVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f10d1966b38>}, 'provides': <octavia.common.data_models.Vip object at 0x7f10d0118d68>}
|__Atom 'reload-lb-before-allocate-vip' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f10d1966b38>}
|__Atom 'octavia.controller.worker.v1.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '5b4598ee-d055-40c4-997d-ea41009d819e'}, 'provides': None}
|__Flow 'octavia-create-loadbalancer-flow': octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last):
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments)
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 424, in execute
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amp_info = self.amphora_driver.get_info(amphora)
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 373, in get_info
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker amphora, raise_retry_exception=raise_retry_exception)
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 106, in _populate_amphora_api_version
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception)['api_version']
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 744, in get_api_version
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise_retry_exception=raise_retry_exception)
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 738, in request
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker raise driver_except.TimeOutException()
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out
2021-08-08 21:22:35.965 27 ERROR octavia.controller.worker.v1.controller_worker
2021-08-08 21:22:35.995 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-amp-compute-connectivity-wait' (7f8c1119-9020-4fb8-ae98-1495875a3dc2) transitioned into state 'REVERTED' from state 'REVERTING'
2021-08-08 21:22:36.003 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-info' (ca7b4681-3fe6-4834-b7ee-2fb2dbe01c3c) transitioned into state 'REVERTED' from state 'REVERTING'
2021-08-08 21:22:36.011 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait' (2ef36bd4-15be-468f-9a8c-f6b60728da6c) transitioned into state 'REVERTED' from state 'REVERTING'
2021-08-08 21:22:36.017 27 WARNING octavia.controller.worker.v1.tasks.database_tasks [-] Reverting mark amphora booting in DB for amp id b6a7018d-d436-477b-8aad-b1b53b1e6cde and compute id c93393df-a39a-486e-812d-73652209fcff
2021-08-08 21:22:36.049 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb' (6aac42e0-fe77-43b8-9323-e0f79304d9d6) transitioned into state 'REVERTED' from state 'REVERTING'
2021-08-08 21:22:36.056 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid' (592da2b6-ab90-4292-a854-c6e2a679dabf) transitioned into state 'REVERTED' from state 'REVERTING'
2021-08-08 21:22:36.061 27 WARNING octavia.controller.worker.v1.tasks.compute_tasks [-] Reverting compute create for amphora with id b6a7018d-d436-477b-8aad-b1b53b1e6cde and compute id: c93393df-a39a-486e-812d-73652209fcff
2021-08-08 21:22:36.576 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create' (59401bf8-9191-4ac1-b4f3-b551a139df9d) transitioned into state 'REVERTED' from state 'REVERTING'
2021-08-08 21:22:36.582 27 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration' (9e731d93-8cb8-46cb-9be4-073e1579a5f8) transitioned into state 'REVERTED' from state 'REVERTING'
*From: *Chris Lyons <chris@lyonsgroup.family> *Date: *Sunday, August 8, 2021 at 8:56 PM *To: *Bernd Bausch <berndbausch@gmail.com>, openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject: *Re: Octavia
I think I might have solved it. It looks like during the kolla install a “Octavia-openrc.sh” script is created which sets up the env to use an account called “service”. After running that and re-uploading the amphora image I am now able to see amphora instances get created. Thank you Bernd for the suggestion that you gve! It caused me to look around in the permissions area and found the solution!
*From: *Bernd Bausch <berndbausch@gmail.com> *Date: *Friday, August 6, 2021 at 8:21 PM *To: *openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org>, Chris Lyons <chris@lyonsgroup.family> *Subject: *Re: Octavia
I had that problem when setting up a Victoria Kolla cluster, and it turned out that the image was owned by the wrong project, as suspected by Michael. I changed the owner in Glance, and it worked. The owner must be 4269b3cd5452416e8153f5b4f5adcf0c in Chris' case (openstack image set --project 4269b3cd5452416e8153f5b4f5adcf0c amphora-image-name).
I don't think this is a Kolla problem, as Kolla leaves the responsibility of building and uploading the image to the administrator (or it did so at Victoria).
Bernd Bausch
On 2021/08/06 11:07 PM, Chris Lyons wrote:
Both of those appear ok…. I do have the amphora image set to public and I see this in the Octavia.conf file:
[controller_worker]
amp_ssh_key_name = octavia_ssh_key
amp_image_tag = amphora
amp_image_owner_id = 4269b3cd5452416e8153f5b4f5adcf0c
amp_boot_network_list = adfa01a9-55bd-42ea-b849-81060d5d7c09
amp_secgroup_list = ecf269e2-42fc-477b-9ce1-38d60c1d8d5d
amp_flavor_id = c854d1e7-885f-4f7a-88a3-79728b561830
client_ca = /etc/octavia/certs/client_ca.cert.pem
network_driver = allowed_address_pairs_driver
compute_driver = compute_nova_driver
amphora_driver = amphora_haproxy_rest_driver
amp_active_retries = 100
amp_active_wait_sec = 2
loadbalancer_topology = SINGLE
*From: *Michael Johnson <johnsomor@gmail.com> <johnsomor@gmail.com> *Date: *Friday, August 6, 2021 at 10:04 AM *To: *Chris Lyons <chris@lyonsgroup.family> <chris@lyonsgroup.family> *Cc: *openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> <openstack-discuss@lists.openstack.org> *Subject: *Re: Octavia
This is a kolla bug most likely.
This could be caused by a few possible issues in kolla:
1. The images are owned by the wrong project. Kolla may deploy Octavia under a special service project. The image kolla uploads to glance must be uploaded under the service project as owner. A simple test to see if this is the issue would be to set the image as "public" such that all users of the cloud can see it. If a load balancer can be created after that change, the images are loaded under a project that Octavia cannot access. 2. The other possibility is kolla configured an alternate tag name. Check the [controller_worker] amp_image_tag setting [1] in octavia.conf. The images must be tagged with the same name as is configured in the octavia.conf for the controllers.
Let us know either way (and ideally open a bug for kolla) what you find out.
Michael
[1] https://docs.openstack.org/octavia/latest/configuration/configref.html#contr...
On Fri, Aug 6, 2021 at 5:52 AM Chris Lyons <chris@lyonsgroup.family> <chris@lyonsgroup.family> wrote:
Octavia group,
I created this story to see if I could get some assistance on an Octavia
issue I am having with my infra. I did a vanilla install of Openstack Wallaby using the Kolla project with Octavia enabled and auto-configure set. Everything appeared to install fine. When I try to create a load balancer using the horizon console or the cli or through further installs such as cloudfoundry that require a loadbalancer, I get an error from glance about “No image found with tag amphora” even though the image does exist. I hope it is something simple or an oversight on my part. Could I get some ideas or places to look?
Story:
https://storyboard.openstack.org/#!/story/2009103
Kolla install followed:
https://docs.openstack.org/kolla-ansible/latest/reference/networking/octavia...
cli output :
[root@kolla ~]# openstack image list --tag amphora
+--------------------------------------+---------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------+--------+
| 8f7398e7-7912-43b8-8131-e90f21c91ab4 | amphora | active |
| 9bf14389-8521-4fb7-a7db-925f4dea1bf3 | amphora-x64-haproxy | active |
+--------------------------------------+---------------------+--------+
[root@kolla ~]#
My Infra scripts (relevant):
https://github.com/mephmanx/openstack-scripts/blob/32363ef5753fdec381f713c74...
I am admin of said infra so if anyone has time or availability (or
interest) to log in and take a look I would be happy to provide creds.
Thank you for any time that you could provide or offer!
This email has been scanned by Inbound Shield™.
*This email has been scanned by Inbound Shield™.*
participants (4)
-
Bernd Bausch
-
Chris Lyons
-
Michael Johnson
-
Shane Miner