auto switch of glance-api containers to other controllers

pradeep pradeep8985 at gmail.com
Thu Dec 15 16:35:54 UTC 2022 

Hello All

Any help is highly appreciated.

On Thu, 15 Dec 2022 at 01:41, pradeep <pradeep8985 at gmail.com> wrote:

> Hi Danny,
>
> Yes, i did, All of them are mounted in all the 3 controllers as well.
>
> # cat /etc/kolla/globals.yml | egrep
> "glance_backend_file|glance_file_datadir_volume"
> #glance_backend_file: "yes"
> glance_file_datadir_volume: "/glanceimages"
>
> root at ctrl-1:~# df -h /glanceimages/
> Filesystem                            Size  Used Avail Use% Mounted on
> <IPaddress>:/Openstack_fileshare1  150G  888M  150G   1% /glanceimages
>
>  root at ctrl-2:~# df -h /glanceimages/
> Filesystem                            Size  Used Avail Use% Mounted on
> <IPaddress>:/Openstack_fileshare1  150G  888M  150G   1% /glanceimages
>
>   root at ctrl-3:~# df -h /glanceimages/
> Filesystem                            Size  Used Avail Use% Mounted on
> <IPaddress>:/Openstack_fileshare1  150G  888M  150G   1% /glanceimages
>
>
> On Wed, 14 Dec 2022 at 17:54, Danny Webb <Danny.Webb at thehutgroup.com>
> wrote:
>
>> did you change the glance_file_datadir_volume variable?  that is
>> required for kolla to deploy multiples:
>>
>>
>> https://github.com/openstack/kolla-ansible/blob/ae3de342e48bc4293564a3b532a66bfbf1326c0d/ansible/group_vars/all.yml#L903
>>
>> ------------------------------
>> *From:* pradeep <pradeep8985 at gmail.com>
>> *Sent:* 14 December 2022 10:19
>> *To:* Danny Webb <Danny.Webb at thehutgroup.com>
>> *Subject:* Re: auto switch of glance-api containers to other controllers
>>
>>
>> * CAUTION: This email originates from outside THG *
>> ------------------------------
>> Thanks Danny, All
>>
>> I have found the glance-api containers not deployed in control2 and
>> control3 hosts although I have used NFS as my backend and share is enabled
>> in all my controllers.
>>
>> I have tried to deploy the containers in control 2 and 3 as below but see
>> some failures. I hope I am not using the wrong command. Am i missing
>> something? Please let me know.
>>
>> *Command used*
>> kolla-ansible -i multinode deploy --tags "glance" --limit
>>  control02.lab,control03.lab -vv
>>
>> *Errors in deployment*
>>
>> RUNNING HANDLER [glance : Restart glance-api container]
>> *********************************************************************************
>> task path:
>> /root/kolla-ansible/venv/share/kolla-ansible/ansible/roles/glance/handlers/main.yml:2
>>
>> fatal: [control02.lab]: FAILED! => {"changed": true, "msg": "'Traceback
>> (most recent call last):\\n  File
>> \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 268,
>> in _raise_for_status\\n    response.raise_for_status()\\n  File
>> \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in
>> raise_for_status\\n raise HTTPError(http_error_msg,
>> response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal
>> Server Error for url:
>> http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=quay.io%2Fopenstack.kolla%2Fubuntu-source-glance-api\\n\\nDuring
>> handling of the above exception, another exception occurred:\\n\\nTraceback
>> (most recent call last):\\n  File
>> \"/tmp/ansible_kolla_docker_payload_iltg71bi/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
>> line 381, in main\\n  File
>> \"/tmp/ansible_kolla_docker_payload_iltg71bi/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
>> line 651, in recreate_or_restart_container\\n    self.start_container()\\n
>>  File
>> \"/tmp/ansible_kolla_docker_payload_iltg71bi/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
>> line 669, in start_container\\n    self.pull_image()\\n  File
>> \"/tmp/ansible_kolla_docker_payload_iltg71bi/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
>> line 450, in pull_image\\n    json.loads(line.strip().decode(\\'utf-8\\'))
>> for line in self.dc.pull(\\n  File
>> \"/usr/local/lib/python3.8/dist-packages/docker/api/image.py\", line 430,
>> in pull\\n    self._raise_for_status(response)\\n  File
>> \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 270,
>> in _raise_for_status\\n    raise create_api_error_from_http_exception(e)\\n
>>  File \"/usr/local/lib/python3.8/dist-packages/docker/errors.py\", line 31,
>> in create_api_error_from_http_exception\\n raise cls(e, response=response,
>> explanation=explanation)\\ndocker.errors.APIError: 500 Server Error for
>> http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=quay.io%2Fopenstack.kolla%2Fubuntu-source-glance-api:
>> Internal Server Error (\"Get \"https://quay.io/v2/\
>> <https://quay.io/v2//>": net/http: request canceled while waiting for
>> connection (Client.Timeout exceeded while awaiting headers)\")\\n'"}
>> fatal: [control03.lab]: FAILED! => {"changed": true, "msg": "'Traceback
>> (most recent call last):\\n  File
>> \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 268,
>> in _raise_for_status\\n    response.raise_for_status()\\n  File
>> \"/usr/lib/python3/dist-packages/requests/models.py\", line 940, in
>> raise_for_status\\n raise HTTPError(http_error_msg,
>> response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal
>> Server Error for url:
>> http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=quay.io%2Fopenstack.kolla%2Fubuntu-source-glance-api\\n\\nDuring
>> handling of the above exception, another exception occurred:\\n\\nTraceback
>> (most recent call last):\\n  File
>> \"/tmp/ansible_kolla_docker_payload_kkez0k31/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\",
>> line 381, in main\\n  File
>> \"/tmp/ansible_kolla_docker_payload_kkez0k31/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
>> line 651, in recreate_or_restart_container\\n    self.start_container()\\n
>>  File
>> \"/tmp/ansible_kolla_docker_payload_kkez0k31/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
>> line 669, in start_container\\n    self.pull_image()\\n  File
>> \"/tmp/ansible_kolla_docker_payload_kkez0k31/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\",
>> line 450, in pull_image\\n    json.loads(line.strip().decode(\\'utf-8\\'))
>> for line in self.dc.pull(\\n  File
>> \"/usr/local/lib/python3.8/dist-packages/docker/api/image.py\", line 430,
>> in pull\\n    self._raise_for_status(response)\\n  File
>> \"/usr/local/lib/python3.8/dist-packages/docker/api/client.py\", line 270,
>> in _raise_for_status\\n    raise create_api_error_from_http_exception(e)\\n
>>  File \"/usr/local/lib/python3.8/dist-packages/docker/errors.py\", line 31,
>> in create_api_error_from_http_exception\\n raise cls(e, response=response,
>> explanation=explanation)\\ndocker.errors.APIError: 500 Server Error for
>> http+docker://localhost/v1.41/images/create?tag=yoga&fromImage=quay.io%2Fopenstack.kolla%2Fubuntu-source-glance-api:
>> Internal Server Error (\"Get \"https://quay.io/v2/\
>> <https://quay.io/v2//>": net/http: request canceled while waiting for
>> connection (Client.Timeout exceeded while awaiting headers)\")\\n'"}
>> META: ran handlers
>>
>> Regards
>> Pradeep
>>
>>
>> On Mon, 12 Dec 2022 at 14:54, Danny Webb <Danny.Webb at thehutgroup.com>
>> wrote:
>>
>> Kolla only uses a single controller if you use file based backend by
>> default.  If you want to continue using the file backend, you can use
>> something like an NFS share to share images between controllers and then
>> change the glance_file_datadir_volume parameter which will result in the
>> api being deployed on all controllers
>> ------------------------------
>> *From:* pradeep <pradeep8985 at gmail.com>
>> *Sent:* 12 December 2022 04:53
>> *To:* Dmitriy Rabotyagov <noonedeadpunk at gmail.com>
>> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
>> *Subject:* Re: auto switch of glance-api containers to other controllers
>>
>>
>> * CAUTION: This email originates from outside THG *
>> ------------------------------
>> Any help is highly appreciated.
>>
>> Thank you!
>>
>> On Fri, 9 Dec 2022 at 21:46, pradeep <pradeep8985 at gmail.com> wrote:
>>
>> Hi Dmitriy, Thanks for the response.  Sorry, I forgot to mention that i
>> used kolla ansible for deployment. As per kolla docs, glance-api container
>> runs only in one controller. So i am looking from kolla perspective if it
>> can automatically switch it over other available controllers.
>>
>> Regards
>> Pradeep
>>
>> On Fri, 9 Dec 2022 at 14:51, Dmitriy Rabotyagov <noonedeadpunk at gmail.com>
>> wrote:
>>
>> Hi,
>>
>> I'm not sure if you used any deployment tool for your environment, but
>> usually for such failover there's a load balancer in conjunction with VRRP
>> or Anycast, that able to detect when controller1 is down and forward
>> traffic to another controller.
>>
>> As example, OpenStack-Ansible as default option installs haproxy to each
>> controller and keepalived to implement VRRP and Virtual IPs. So when glance
>> is down on controller1, haproxy will detect that and forward traffic to
>> other controllers. If controller1 is down as a whole, keepalived will
>> detect that and raise VIP on controller2, so all client traffic will go
>> there. Since controller2 also have haproxy, it will pass traffic to
>> available backends based on the source IP.
>>
>>
>> пт, 9 дек. 2022 г., 09:06 pradeep <pradeep8985 at gmail.com>:
>>
>> Hello All,
>>
>> I understand that glance-api container run only one controller always. I
>> have tried to check if the glance-api container to switch over to the next
>> available controller by means of rebooting and stopping the containers, but
>> nothing happened. Is there a way to make sure that glance-api container
>> switches to other controllers if controller 1 is not available.
>>
>> Thanks for you help
>>
>> Regards
>> Pradeep
>>
>>
>>
>>
>> --
>> -----------------------
>> Regards
>> Pradeep Kumar
>>
>>
>>
>> --
>> -----------------------
>> Regards
>> Pradeep Kumar
>>
>> *Danny Webb*
>> Principal OpenStack Engineer
>> Danny.Webb at thehutgroup.com
>> [image: THG Ingenuity Logo]
>> www.thg.com
>> <https://www.linkedin.com/company/thg-ingenuity/?originalSubdomain=uk>
>> <https://twitter.com/thgingenuity?lang=en>
>>
>> Danny Webb
>> Principal OpenStack Engineer
>> The Hut Group <http://www.thehutgroup.com/>
>>
>> Tel:
>> Email: Danny.Webb at thehutgroup.com
>>
>>
>> For the purposes of this email, the "company" means The Hut Group
>> Limited, a company registered in England and Wales (company number 6539496)
>> whose registered office is at Fifth Floor, Voyager House, Chicago Avenue,
>> Manchester Airport, M90 3DQ and/or any of its respective subsidiaries.
>>
>> *Confidentiality Notice*
>> This e-mail is confidential and intended for the use of the named
>> recipient only. If you are not the intended recipient please notify us by
>> telephone immediately on +44(0)1606 811888 or return it to us by e-mail.
>> Please then delete it from your system and note that any use,
>> dissemination, forwarding, printing or copying is strictly prohibited. Any
>> views or opinions are solely those of the author and do not necessarily
>> represent those of the company.
>>
>> *Encryptions and Viruses*
>> Please note that this e-mail and any attachments have not been encrypted.
>> They may therefore be liable to be compromised. Please also note that it is
>> your responsibility to scan this e-mail and any attachments for viruses. We
>> do not, to the extent permitted by law, accept any liability (whether in
>> contract, negligence or otherwise) for any virus infection and/or external
>> compromise of security and/or confidentiality in relation to transmissions
>> sent by e-mail.
>>
>> *Monitoring*
>> Activity and use of the company's systems is monitored to secure its
>> effective use and operation and for other lawful business purposes.
>> Communications using these systems will also be monitored and may be
>> recorded to secure effective use and operation and for other lawful
>> business purposes.
>> hgvyjuv
>>
>>
>>
>> --
>> -----------------------
>> Regards
>> Pradeep Kumar
>>
>> *Danny Webb*
>> Principal OpenStack Engineer
>> Danny.Webb at thehutgroup.com
>> [image: THG Ingenuity Logo]
>> www.thg.com
>> <https://www.linkedin.com/company/thg-ingenuity/?originalSubdomain=uk>
>> <https://twitter.com/thgingenuity?lang=en>
>>
>
>
> --
> -----------------------
> Regards
> Pradeep Kumar
>


-- 
-----------------------
Regards
Pradeep Kumar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221215/9f9c9f17/attachment-0001.htm>

More information about the openstack-discuss mailing list