Hello All Any help is highly appreciated. On Thu, 15 Dec 2022 at 01:41, pradeep <pradeep8985@gmail.com> wrote:

Hi Danny,

Yes, i did, All of them are mounted in all the 3 controllers as well.

# cat /etc/kolla/globals.yml | egrep "glance_backend_file|glance_file_datadir_volume" #glance_backend_file: "yes" glance_file_datadir_volume: "/glanceimages"

root@ctrl-1:~# df -h /glanceimages/ Filesystem Size Used Avail Use% Mounted on <IPaddress>:/Openstack_fileshare1 150G 888M 150G 1% /glanceimages

root@ctrl-2:~# df -h /glanceimages/ Filesystem Size Used Avail Use% Mounted on <IPaddress>:/Openstack_fileshare1 150G 888M 150G 1% /glanceimages

root@ctrl-3:~# df -h /glanceimages/ Filesystem Size Used Avail Use% Mounted on <IPaddress>:/Openstack_fileshare1 150G 888M 150G 1% /glanceimages

On Wed, 14 Dec 2022 at 17:54, Danny Webb <Danny.Webb@thehutgroup.com> wrote:

...

did you change the glance_file_datadir_volume variable? that is required for kolla to deploy multiples:

https://github.com/openstack/kolla-ansible/blob/ae3de342e48bc4293564a3b532a6...

------------------------------ *From:* pradeep <pradeep8985@gmail.com> *Sent:* 14 December 2022 10:19 *To:* Danny Webb <Danny.Webb@thehutgroup.com> *Subject:* Re: auto switch of glance-api containers to other controllers

* CAUTION: This email originates from outside THG * ------------------------------ Thanks Danny, All

I have found the glance-api containers not deployed in control2 and control3 hosts although I have used NFS as my backend and share is enabled in all my controllers.

I have tried to deploy the containers in control 2 and 3 as below but see some failures. I hope I am not using the wrong command. Am i missing something? Please let me know.

*Command used* kolla-ansible -i multinode deploy --tags "glance" --limit control02.lab,control03.lab -vv

*Errors in deployment*

RUNNING HANDLER [glance : Restart glance-api container] ********************************************************************************* task path: /root/kolla-ansible/venv/share/kolla-ansible/ansible/roles/glance/handlers/main.yml:2

fatal: [control02.lab]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 268, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=quay.io%2Fopenstack.kolla%2Fubuntu-source-glance-api\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_iltg71bi/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 381, in main\\n File \"/tmp/ansible_kolla_docker_payload_iltg71bi/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 651, in recreate_or_restart_container\\n self.start_container()\\n File \"/tmp/ansible_kolla_docker_payload_iltg71bi/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 669, in start_container\\n self.pull_image()\\n File \"/tmp/ansible_kolla_docker_payload_iltg71bi/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 450, in pull_image\\n json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n File \"/usr/local/lib/python3.8/dist-packages/docker/api/image.py\", line 430, in pull\\n self._raise_for_status(response)\\n File \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 270, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/local/lib/python3.8/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=quay.io%2Fopenstack.kolla%2Fubuntu-source-glance-api: Internal Server Error (\"Get \"https://quay.io/v2/\ <https://quay.io/v2//>": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\")\\n'"} fatal: [control03.lab]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 268, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=quay.io%2Fopenstack.kolla%2Fubuntu-source-glance-api\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_kkez0k31/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 381, in main\\n File \"/tmp/ansible_kolla_docker_payload_kkez0k31/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 651, in recreate_or_restart_container\\n self.start_container()\\n File \"/tmp/ansible_kolla_docker_payload_kkez0k31/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 669, in start_container\\n self.pull_image()\\n File \"/tmp/ansible_kolla_docker_payload_kkez0k31/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 450, in pull_image\\n json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n File \"/usr/local/lib/python3.8/dist-packages/docker/api/image.py\", line 430, in pull\\n self._raise_for_status(response)\\n File \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 270, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/local/lib/python3.8/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=quay.io%2Fopenstack.kolla%2Fubuntu-source-glance-api: Internal Server Error (\"Get \"https://quay.io/v2/\ <https://quay.io/v2//>": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\")\\n'"} META: ran handlers

Regards Pradeep

On Mon, 12 Dec 2022 at 14:54, Danny Webb <Danny.Webb@thehutgroup.com> wrote:

Kolla only uses a single controller if you use file based backend by default. If you want to continue using the file backend, you can use something like an NFS share to share images between controllers and then change the glance_file_datadir_volume parameter which will result in the api being deployed on all controllers ------------------------------ *From:* pradeep <pradeep8985@gmail.com> *Sent:* 12 December 2022 04:53 *To:* Dmitriy Rabotyagov <noonedeadpunk@gmail.com> *Cc:* openstack-discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: auto switch of glance-api containers to other controllers

* CAUTION: This email originates from outside THG * ------------------------------ Any help is highly appreciated.

Thank you!

On Fri, 9 Dec 2022 at 21:46, pradeep <pradeep8985@gmail.com> wrote:

Hi Dmitriy, Thanks for the response. Sorry, I forgot to mention that i used kolla ansible for deployment. As per kolla docs, glance-api container runs only in one controller. So i am looking from kolla perspective if it can automatically switch it over other available controllers.

Regards Pradeep

On Fri, 9 Dec 2022 at 14:51, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:

Hi,

I'm not sure if you used any deployment tool for your environment, but usually for such failover there's a load balancer in conjunction with VRRP or Anycast, that able to detect when controller1 is down and forward traffic to another controller.

As example, OpenStack-Ansible as default option installs haproxy to each controller and keepalived to implement VRRP and Virtual IPs. So when glance is down on controller1, haproxy will detect that and forward traffic to other controllers. If controller1 is down as a whole, keepalived will detect that and raise VIP on controller2, so all client traffic will go there. Since controller2 also have haproxy, it will pass traffic to available backends based on the source IP.

пт, 9 дек. 2022 г., 09:06 pradeep <pradeep8985@gmail.com>:

Hello All,

I understand that glance-api container run only one controller always. I have tried to check if the glance-api container to switch over to the next available controller by means of rebooting and stopping the containers, but nothing happened. Is there a way to make sure that glance-api container switches to other controllers if controller 1 is not available.

Thanks for you help

Regards Pradeep

-- ----------------------- Regards Pradeep Kumar

-- ----------------------- Regards Pradeep Kumar

*Danny Webb* Principal OpenStack Engineer Danny.Webb@thehutgroup.com [image: THG Ingenuity Logo] www.thg.com <https://www.linkedin.com/company/thg-ingenuity/?originalSubdomain=uk> <https://twitter.com/thgingenuity?lang=en>

Danny Webb Principal OpenStack Engineer The Hut Group <http://www.thehutgroup.com/>

Tel: Email: Danny.Webb@thehutgroup.com

For the purposes of this email, the "company" means The Hut Group Limited, a company registered in England and Wales (company number 6539496) whose registered office is at Fifth Floor, Voyager House, Chicago Avenue, Manchester Airport, M90 3DQ and/or any of its respective subsidiaries.

*Confidentiality Notice* This e-mail is confidential and intended for the use of the named recipient only. If you are not the intended recipient please notify us by telephone immediately on +44(0)1606 811888 or return it to us by e-mail. Please then delete it from your system and note that any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views or opinions are solely those of the author and do not necessarily represent those of the company.

*Encryptions and Viruses* Please note that this e-mail and any attachments have not been encrypted. They may therefore be liable to be compromised. Please also note that it is your responsibility to scan this e-mail and any attachments for viruses. We do not, to the extent permitted by law, accept any liability (whether in contract, negligence or otherwise) for any virus infection and/or external compromise of security and/or confidentiality in relation to transmissions sent by e-mail.

*Monitoring* Activity and use of the company's systems is monitored to secure its effective use and operation and for other lawful business purposes. Communications using these systems will also be monitored and may be recorded to secure effective use and operation and for other lawful business purposes. hgvyjuv

-- ----------------------- Regards Pradeep Kumar

*Danny Webb* Principal OpenStack Engineer Danny.Webb@thehutgroup.com [image: THG Ingenuity Logo] www.thg.com <https://www.linkedin.com/company/thg-ingenuity/?originalSubdomain=uk> <https://twitter.com/thgingenuity?lang=en>

-- ----------------------- Regards Pradeep Kumar

-- ----------------------- Regards Pradeep Kumar