[glance] Does glance not support using local filesystem storage in a cluster
Deal all, Sorry that maybe I ask a stupid question. But I'm really confused with it and didn't find discuss in glance document(https://docs.openstack.org/glance/latest/). I have a OpenStack Victoria cluster with three all-in-one node in centos8. I implemented it with reference to https://docs.openstack.org/ha-guide/. So this cluster use Pacemaker, HAproxy and Galera. "To implement high availability, run an instance of the database on each controller node and use Galera Cluster to provide replication between them." I found that I will encounter an error If I configure Glance backend to use local storage driver to store image files on the local disk. If I upload a image, this image only will be storaged in one node. But the database only storage the file path of image such as "/v2/images/aa3cbee0-717f-4699-8cca-61243302d693/file", don't have the host information. The database data is same in three node. If I upload a image in node1, image only is storaged in node1. The database of three node stores the local filesystem path of image. And If The create Instance task is assigned to node2, It will find image in node2, but image can't be found in node2. So we get the "Image has no associated data" error. So I want to ask: 1. Wheter glance does not support using local filesystem storage in a cluster? 2. If 1 was right, why do we do this design instead of storing information about the host on which images is located, as nova does with instance. I would appreciate any kind of guidance or help. Thank you, Han Guangyu
Hi, if you really aim towards a highly available cluster you'll also need a ha storage solution like ceph. Having glance images or VMs on local storage can make it easier to deploy, maybe for testing and getting involved with openstack, but it's not really recommended for production use. You'll probably have the same issue with cinder volumes, I believe. Or do you have a different backend for cinder? Regards, Eugen Zitat von 韩光宇 <hanguangyu2@gmail.com>:
Deal all,
Sorry that maybe I ask a stupid question. But I'm really confused with it and didn't find discuss in glance document(https://docs.openstack.org/glance/latest/).
I have a OpenStack Victoria cluster with three all-in-one node in centos8. I implemented it with reference to https://docs.openstack.org/ha-guide/. So this cluster use Pacemaker, HAproxy and Galera. "To implement high availability, run an instance of the database on each controller node and use Galera Cluster to provide replication between them."
I found that I will encounter an error If I configure Glance backend to use local storage driver to store image files on the local disk. If I upload a image, this image only will be storaged in one node. But the database only storage the file path of image such as "/v2/images/aa3cbee0-717f-4699-8cca-61243302d693/file", don't have the host information. The database data is same in three node.
If I upload a image in node1, image only is storaged in node1. The database of three node stores the local filesystem path of image. And If The create Instance task is assigned to node2, It will find image in node2, but image can't be found in node2. So we get the "Image has no associated data" error.
So I want to ask: 1. Wheter glance does not support using local filesystem storage in a cluster? 2. If 1 was right, why do we do this design instead of storing information about the host on which images is located, as nova does with instance.
I would appreciate any kind of guidance or help.
Thank you, Han Guangyu
Hi, Yes, you are right. I have deploied a ceph as the backend storage of glance, cinder and nova. And it can resolve this question. But I wonder why it's designed this way. It doesn't fit my perception of OpenStack. As currently designed, local storage of glance must not be used in the cluster. Why not record the host where the image resides? Just like the local storage of the nova-compute node, if a Glance node breaks down, the image on the host cannot be accessed. Sorry that maybe this idea is unreasonable and stupid. Could anyone tell me the reason or what's the problem with that best wishes to you, love you. Thank you, Han Guangyu Eugen Block <eblock@nde.ag> 于2022年1月6日周四 17:03写道：
Hi,
if you really aim towards a highly available cluster you'll also need a ha storage solution like ceph. Having glance images or VMs on local storage can make it easier to deploy, maybe for testing and getting involved with openstack, but it's not really recommended for production use. You'll probably have the same issue with cinder volumes, I believe. Or do you have a different backend for cinder?
Regards, Eugen
Zitat von 韩光宇 <hanguangyu2@gmail.com>:
Deal all,
Sorry that maybe I ask a stupid question. But I'm really confused with it and didn't find discuss in glance document(https://docs.openstack.org/glance/latest/).
I have a OpenStack Victoria cluster with three all-in-one node in centos8. I implemented it with reference to https://docs.openstack.org/ha-guide/. So this cluster use Pacemaker, HAproxy and Galera. "To implement high availability, run an instance of the database on each controller node and use Galera Cluster to provide replication between them."
I found that I will encounter an error If I configure Glance backend to use local storage driver to store image files on the local disk. If I upload a image, this image only will be storaged in one node. But the database only storage the file path of image such as "/v2/images/aa3cbee0-717f-4699-8cca-61243302d693/file", don't have the host information. The database data is same in three node.
If I upload a image in node1, image only is storaged in node1. The database of three node stores the local filesystem path of image. And If The create Instance task is assigned to node2, It will find image in node2, but image can't be found in node2. So we get the "Image has no associated data" error.
So I want to ask: 1. Wheter glance does not support using local filesystem storage in a cluster? 2. If 1 was right, why do we do this design instead of storing information about the host on which images is located, as nova does with instance.
I would appreciate any kind of guidance or help.
Thank you, Han Guangyu
Hi everyone, we are actually using glance with filesystem storage in our clusters. To do this we use an NFS share / k8s PVC as the backing storage for the images on all nodes. However we only go this route since we are using NFS for our cinder storage as well. So it probably only makes sense if you have a shared Filesystem available to your setup anyway. Best Regards Felix -----Original Message----- From: 韩光宇 <hanguangyu2@gmail.com> Sent: Thursday, January 6, 2022 10:42 AM To: openstack-discuss <openstack-discuss@lists.openstack.org> Cc: wangleic@uniontech.com; hanguangyu@uniontech.com Subject: Re: [glance] Does glance not support using local filesystem storage in a cluster Hi, Yes, you are right. I have deploied a ceph as the backend storage of glance, cinder and nova. And it can resolve this question. But I wonder why it's designed this way. It doesn't fit my perception of OpenStack. As currently designed, local storage of glance must not be used in the cluster. Why not record the host where the image resides? Just like the local storage of the nova-compute node, if a Glance node breaks down, the image on the host cannot be accessed. Sorry that maybe this idea is unreasonable and stupid. Could anyone tell me the reason or what's the problem with that best wishes to you, love you. Thank you, Han Guangyu Eugen Block <eblock@nde.ag> 于2022年1月6日周四 17:03写道：
Hi,
if you really aim towards a highly available cluster you'll also need a ha storage solution like ceph. Having glance images or VMs on local storage can make it easier to deploy, maybe for testing and getting involved with openstack, but it's not really recommended for production use. You'll probably have the same issue with cinder volumes, I believe. Or do you have a different backend for cinder?
Regards, Eugen
Zitat von 韩光宇 <hanguangyu2@gmail.com>:
Deal all,
Sorry that maybe I ask a stupid question. But I'm really confused with it and didn't find discuss in glance document(https://docs.openstack.org/glance/latest/).
I have a OpenStack Victoria cluster with three all-in-one node in centos8. I implemented it with reference to https://docs.openstack.org/ha-guide/. So this cluster use Pacemaker, HAproxy and Galera. "To implement high availability, run an instance of the database on each controller node and use Galera Cluster to provide replication between them."
I found that I will encounter an error If I configure Glance backend to use local storage driver to store image files on the local disk. If I upload a image, this image only will be storaged in one node. But the database only storage the file path of image such as "/v2/images/aa3cbee0-717f-4699-8cca-61243302d693/file", don't have the host information. The database data is same in three node.
If I upload a image in node1, image only is storaged in node1. The database of three node stores the local filesystem path of image. And If The create Instance task is assigned to node2, It will find image in node2, but image can't be found in node2. So we get the "Image has no associated data" error.
So I want to ask: 1. Wheter glance does not support using local filesystem storage in a cluster? 2. If 1 was right, why do we do this design instead of storing information about the host on which images is located, as nova does with instance.
I would appreciate any kind of guidance or help.
Thank you, Han Guangyu
Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz>.
easier to deploy, maybe for testing and getting involved with openstack, but it's not really recommended for
Hi, Sorry to bother. I continued to think about it. If the the original purpose of "local storage of glance" is just to make openstack production use. I can accept this setting. I used to think of it as an equal way to use shared storage. Thank you. Han 韩光宇 <hanguangyu2@gmail.com> 于2022年1月6日周四 10:33写道：
Deal all,
Sorry that maybe I ask a stupid question. But I'm really confused with it and didn't find discuss in glance document(https://docs.openstack.org/glance/latest/).
I have a OpenStack Victoria cluster with three all-in-one node in centos8. I implemented it with reference to https://docs.openstack.org/ha-guide/. So this cluster use Pacemaker, HAproxy and Galera. "To implement high availability, run an instance of the database on each controller node and use Galera Cluster to provide replication between them."
I found that I will encounter an error If I configure Glance backend to use local storage driver to store image files on the local disk. If I upload a image, this image only will be storaged in one node. But the database only storage the file path of image such as "/v2/images/aa3cbee0-717f-4699-8cca-61243302d693/file", don't have the host information. The database data is same in three node.
If I upload a image in node1, image only is storaged in node1. The database of three node stores the local filesystem path of image. And If The create Instance task is assigned to node2, It will find image in node2, but image can't be found in node2. So we get the "Image has no associated data" error.
So I want to ask: 1. Wheter glance does not support using local filesystem storage in a cluster? 2. If 1 was right, why do we do this design instead of storing information about the host on which images is located, as nova does with instance.
I would appreciate any kind of guidance or help.
Thank you, Han Guangyu
On 1/6/22 03:33, 韩光宇 wrote:
Deal all,
Sorry that maybe I ask a stupid question. But I'm really confused with it and didn't find discuss in glance document(https://docs.openstack.org/glance/latest/).
I have a OpenStack Victoria cluster with three all-in-one node in centos8. I implemented it with reference to https://docs.openstack.org/ha-guide/. So this cluster use Pacemaker, HAproxy and Galera. "To implement high availability, run an instance of the database on each controller node and use Galera Cluster to provide replication between them."
I found that I will encounter an error If I configure Glance backend to use local storage driver to store image files on the local disk. If I upload a image, this image only will be storaged in one node. But the database only storage the file path of image such as "/v2/images/aa3cbee0-717f-4699-8cca-61243302d693/file", don't have the host information. The database data is same in three node.
If I upload a image in node1, image only is storaged in node1. The database of three node stores the local filesystem path of image. And If The create Instance task is assigned to node2, It will find image in node2, but image can't be found in node2. So we get the "Image has no associated data" error.
So I want to ask: 1. Wheter glance does not support using local filesystem storage in a cluster? 2. If 1 was right, why do we do this design instead of storing information about the host on which images is located, as nova does with instance.
I would appreciate any kind of guidance or help.
Thank you, Han Guangyu
Hi 光宇, It is possible to setup Glance with a local storage in a HA way. The way to do this, is simply to get your HAproxy to use one node, always, and the other as backups. Then have a cron job that does the rsync from the first node to the other 2. A simple command as Glance user like this is enough (to be run as Glance user, and having the ssh host keys thingy fixed (we sign host keys, so we don't have this problem)): rsync -e ssh -avz --delete /var/lib/glance/images/ \ <dest-host>:/var/lib/glance/images/ >/dev/null 2>&1 We have some internal logic to iterate through all the backup nodes and replace dest-host accordingly... This way, if the first node fails, yes, you do have a problem because there wont be the primary node that is up, so saving new Glance image will be a problem as it wont be replicated to other nodes. But existing image will be there already, so it ok until you repair the first node. I hope this helps, Cheers, Thomas Goirand (zigo)
We've done something similar insofar as we've installed gluster with replica X on our controllers to share out the glace images volume between nodes. Saves you having to rsync between nodes and supports proper HA. ________________________________ From: Thomas Goirand <zigo@debian.org> Sent: 06 January 2022 12:44 To: 韩光宇 <hanguangyu2@gmail.com>; openstack-discuss <openstack-discuss@lists.openstack.org> Cc: hanguangyu@uniontech.com <hanguangyu@uniontech.com>; wangleic@uniontech.com <wangleic@uniontech.com> Subject: Re: [glance] Does glance not support using local filesystem storage in a cluster CAUTION: This email originates from outside THG On 1/6/22 03:33, 韩光宇 wrote:
Deal all,
Sorry that maybe I ask a stupid question. But I'm really confused with it and didn't find discuss in glance document(https://docs.openstack.org/glance/latest/<https://docs.openstack.org/glance/latest>).
I have a OpenStack Victoria cluster with three all-in-one node in centos8. I implemented it with reference to https://docs.openstack.org/ha-guide/<https://docs.openstack.org/ha-guide>. So this cluster use Pacemaker, HAproxy and Galera. "To implement high availability, run an instance of the database on each controller node and use Galera Cluster to provide replication between them."
I found that I will encounter an error If I configure Glance backend to use local storage driver to store image files on the local disk. If I upload a image, this image only will be storaged in one node. But the database only storage the file path of image such as "/v2/images/aa3cbee0-717f-4699-8cca-61243302d693/file", don't have the host information. The database data is same in three node.
If I upload a image in node1, image only is storaged in node1. The database of three node stores the local filesystem path of image. And If The create Instance task is assigned to node2, It will find image in node2, but image can't be found in node2. So we get the "Image has no associated data" error.
So I want to ask: 1. Wheter glance does not support using local filesystem storage in a cluster? 2. If 1 was right, why do we do this design instead of storing information about the host on which images is located, as nova does with instance.
I would appreciate any kind of guidance or help.
Thank you, Han Guangyu
Hi 光宇, It is possible to setup Glance with a local storage in a HA way. The way to do this, is simply to get your HAproxy to use one node, always, and the other as backups. Then have a cron job that does the rsync from the first node to the other 2. A simple command as Glance user like this is enough (to be run as Glance user, and having the ssh host keys thingy fixed (we sign host keys, so we don't have this problem)): rsync -e ssh -avz --delete /var/lib/glance/images/ \ <dest-host>:/var/lib/glance/images/ >/dev/null 2>&1 We have some internal logic to iterate through all the backup nodes and replace dest-host accordingly... This way, if the first node fails, yes, you do have a problem because there wont be the primary node that is up, so saving new Glance image will be a problem as it wont be replicated to other nodes. But existing image will be there already, so it ok until you repair the first node. I hope this helps, Cheers, Thomas Goirand (zigo) Danny Webb Senior Linux Systems Administrator The Hut Group<http://www.thehutgroup.com/> Tel: Email: Danny.Webb@thehutgroup.com<mailto:Danny.Webb@thehutgroup.com> For the purposes of this email, the "company" means The Hut Group Limited, a company registered in England and Wales (company number 6539496) whose registered office is at Fifth Floor, Voyager House, Chicago Avenue, Manchester Airport, M90 3DQ and/or any of its respective subsidiaries. Confidentiality Notice This e-mail is confidential and intended for the use of the named recipient only. If you are not the intended recipient please notify us by telephone immediately on +44(0)1606 811888 or return it to us by e-mail. Please then delete it from your system and note that any use, dissemination, forwarding, printing or copying is strictly prohibited. Any views or opinions are solely those of the author and do not necessarily represent those of the company. Encryptions and Viruses Please note that this e-mail and any attachments have not been encrypted. They may therefore be liable to be compromised. Please also note that it is your responsibility to scan this e-mail and any attachments for viruses. We do not, to the extent permitted by law, accept any liability (whether in contract, negligence or otherwise) for any virus infection and/or external compromise of security and/or confidentiality in relation to transmissions sent by e-mail. Monitoring Activity and use of the company's systems is monitored to secure its effective use and operation and for other lawful business purposes. Communications using these systems will also be monitored and may be recorded to secure effective use and operation and for other lawful business purposes. hgvyjuv
On 1/6/22 14:05, Danny Webb wrote:
We've done something similar insofar as we've installed gluster with replica X on our controllers to share out the glace images volume between nodes. Saves you having to rsync between nodes and supports proper HA.
One needs to like Gluster though ... :) Thomas
participants (5)
-
Danny Webb
-
Eugen Block
-
Felix Hüttner
-
Thomas Goirand
-
韩光宇