Re: [ops] [glance] Need to move images to a different backend
Hi Abhishek Yes, before trying the: glance location-add --url rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap 6bcc4eab-ed35-42dc-88bd-1d45de73b628 command, I uploaded the image to ceph. And indeed I can see it: [root@ceph-mon-01 ~]# rbd info images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 rbd image '6bcc4eab-ed35-42dc-88bd-1d45de73b628': size 10GiB in 1280 objects order 23 (8MiB objects) block_name_prefix: rbd_data.b7b56aa1e4f0bc format: 2 features: layering flags: create_timestamp: Tue Aug 27 11:42:32 2019 [root@ceph-mon-01 ~]# rbd snap ls images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 SNAPID NAME SIZE TIMESTAMP 455500 snap 10GiB Tue Aug 27 11:46:37 2019 [root@ceph-mon-01 ~]# Cheers, Massimo On Tue, Aug 27, 2019 at 2:41 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Massimo,
I need to reproduce this first, but pretty much sure this issue is not because you configured multiple backends. What you are doing is downloading existing image to your local storage and then uploading it to glance using add-location operation.
So before adding this using add-location operation have you manually uploaded this image to ceph/rbd? if not then how you are building your location url which you are mentioning in the add-location command? If this location is not existing then it might be a problem.
Thank you,
Abhishek
On Tue, 27 Aug 2019 at 5:09 PM, Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I have a Rocky installation where glance is configured with multiple backends. This [*] is the relevant part of the glance configuration.
I now need to dismiss the file backend and move the images-snapshots stored there to rbd.
I had in mind to follow this procedure (already successfully tested with an older version of OpenStack):
1) download the image from the file backend (glance image-download --file <file> <imageid> 2) upload the file <file> to rbd, relying on this function: https://github.com/openstack/glance_store/blob/stable/rocky/glance_store/_dr... 3) glance location-add --url rbd://xyz <imageid> 4) glance location-delete --url file://abc <imageid>
I have problems with 3:
# glance --debug location-add --url rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap 6bcc4eab-ed35-42dc-88bd-1d45de73b628 ... ... DEBUG:keystoneauth.session:PATCH call to image for https://cloud-areapd.pd.infn.it:9292/v2/images/6bcc4eab-ed35-42dc-88bd-1d45d... used request id req-6c0598cc-582c-4ce7-a14c-5d1bb6ec4f14 Request returned failure status 400. DEBUG:glanceclient.common.http:Request returned failure status 400. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 687, in main OpenStackImagesShell().main(argv) File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 591, in main args.func(client, args) File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 749, in do_location_add image = gc.images.add_location(args.id, args.url, metadata) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 448, in add_location response = self._send_image_update_request(image_id, add_patch) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner return RequestIdProxy(wrapped(*args, **kwargs)) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 432, in _send_image_update_request data=json.dumps(patch_body)) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 340, in patch return self.request(url, 'PATCH', **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request return self._handle_response(resp) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response raise exc.from_response(resp, resp.content) HTTPBadRequest: 400 Bad Request: Invalid location (HTTP 400) 400 Bad Request: Invalid location (HTTP 400)
As far as I can see with the "openstack" client there is not something to add/delete a location. So I guess it is necessary to change the 'direct_url' and 'locations' properties.
If I try to change the direct_url property:
# openstack image set --property direct_url='rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap' 6bcc4eab-ed35-42dc-88bd-1d45de73b628 403 Forbidden: Attribute 'direct_url' is read-only. (HTTP 403)
Any hints ? Thanks, Massimo
[*]
[default] ... enabled_backends = file:file,http:http,rbd:rbd show_image_direct_url = true show_multiple_locations = true
[glance_store] default_backend = rbd
[file] filesystem_store_datadir = /var/lib/glance/images/
[rbd] rbd_store_chunk_size = 8 rbd_store_ceph_conf = /etc/ceph/ceph.conf rbd_store_user = glance-prod rbd_store_pool = images-prod
-- Thanks & Best Regards,
Abhishek Kekane
Hi Massimo, Cool, thank you for confirmation, I will try to reproduce this in my environment. It will take some time, will revert back to you once done. Meanwhile could you please try the same with single store (disabling the multiple stores)? Thank you, Abhishek On Tue, 27 Aug 2019 at 6:33 PM, Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
Hi Abhishek
Yes, before trying the:
glance location-add --url rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap 6bcc4eab-ed35-42dc-88bd-1d45de73b628
command, I uploaded the image to ceph. And indeed I can see it:
[root@ceph-mon-01 ~]# rbd info images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 rbd image '6bcc4eab-ed35-42dc-88bd-1d45de73b628': size 10GiB in 1280 objects order 23 (8MiB objects) block_name_prefix: rbd_data.b7b56aa1e4f0bc format: 2 features: layering flags: create_timestamp: Tue Aug 27 11:42:32 2019
[root@ceph-mon-01 ~]# rbd snap ls images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 SNAPID NAME SIZE TIMESTAMP 455500 snap 10GiB Tue Aug 27 11:46:37 2019 [root@ceph-mon-01 ~]#
Cheers, Massimo
On Tue, Aug 27, 2019 at 2:41 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Massimo,
I need to reproduce this first, but pretty much sure this issue is not because you configured multiple backends. What you are doing is downloading existing image to your local storage and then uploading it to glance using add-location operation.
So before adding this using add-location operation have you manually uploaded this image to ceph/rbd? if not then how you are building your location url which you are mentioning in the add-location command? If this location is not existing then it might be a problem.
Thank you,
Abhishek
On Tue, 27 Aug 2019 at 5:09 PM, Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I have a Rocky installation where glance is configured with multiple backends. This [*] is the relevant part of the glance configuration.
I now need to dismiss the file backend and move the images-snapshots stored there to rbd.
I had in mind to follow this procedure (already successfully tested with an older version of OpenStack):
1) download the image from the file backend (glance image-download --file <file> <imageid> 2) upload the file <file> to rbd, relying on this function: https://github.com/openstack/glance_store/blob/stable/rocky/glance_store/_dr... 3) glance location-add --url rbd://xyz <imageid> 4) glance location-delete --url file://abc <imageid>
I have problems with 3:
# glance --debug location-add --url rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap 6bcc4eab-ed35-42dc-88bd-1d45de73b628 ... ... DEBUG:keystoneauth.session:PATCH call to image for https://cloud-areapd.pd.infn.it:9292/v2/images/6bcc4eab-ed35-42dc-88bd-1d45d... used request id req-6c0598cc-582c-4ce7-a14c-5d1bb6ec4f14 Request returned failure status 400. DEBUG:glanceclient.common.http:Request returned failure status 400. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 687, in main OpenStackImagesShell().main(argv) File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 591, in main args.func(client, args) File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 749, in do_location_add image = gc.images.add_location(args.id, args.url, metadata) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 448, in add_location response = self._send_image_update_request(image_id, add_patch) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner return RequestIdProxy(wrapped(*args, **kwargs)) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 432, in _send_image_update_request data=json.dumps(patch_body)) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 340, in patch return self.request(url, 'PATCH', **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request return self._handle_response(resp) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response raise exc.from_response(resp, resp.content) HTTPBadRequest: 400 Bad Request: Invalid location (HTTP 400) 400 Bad Request: Invalid location (HTTP 400)
As far as I can see with the "openstack" client there is not something to add/delete a location. So I guess it is necessary to change the 'direct_url' and 'locations' properties.
If I try to change the direct_url property:
# openstack image set --property direct_url='rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap' 6bcc4eab-ed35-42dc-88bd-1d45de73b628 403 Forbidden: Attribute 'direct_url' is read-only. (HTTP 403)
Any hints ? Thanks, Massimo
[*]
[default] ... enabled_backends = file:file,http:http,rbd:rbd show_image_direct_url = true show_multiple_locations = true
[glance_store] default_backend = rbd
[file] filesystem_store_datadir = /var/lib/glance/images/
[rbd] rbd_store_chunk_size = 8 rbd_store_ceph_conf = /etc/ceph/ceph.conf rbd_store_user = glance-prod rbd_store_pool = images-prod
-- Thanks & Best Regards,
Abhishek Kekane
-- Thanks & Best Regards,
Abhishek Kekane
Hi Massimo, Glance multiple store is still experimental and it does not have support for location API in rocky and stein, we are making it compatible in Train release. There is still one bug open for location compatibility with multiple stores and patch for it is not merged yet, I will also encourage you to deploy latest master and apply this patch in your environment and test it. https://review.opendev.org/#/c/617229/19 NOTE: I have tested this on current master after applying patch to local environment and the scenario you mentioned is working fine. Thanks & Best Regards, Abhishek Kekane On Tue, Aug 27, 2019 at 7:47 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Massimo, Cool, thank you for confirmation, I will try to reproduce this in my environment. It will take some time, will revert back to you once done. Meanwhile could you please try the same with single store (disabling the multiple stores)?
Thank you,
Abhishek
On Tue, 27 Aug 2019 at 6:33 PM, Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
Hi Abhishek
Yes, before trying the:
glance location-add --url rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap 6bcc4eab-ed35-42dc-88bd-1d45de73b628
command, I uploaded the image to ceph. And indeed I can see it:
[root@ceph-mon-01 ~]# rbd info images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 rbd image '6bcc4eab-ed35-42dc-88bd-1d45de73b628': size 10GiB in 1280 objects order 23 (8MiB objects) block_name_prefix: rbd_data.b7b56aa1e4f0bc format: 2 features: layering flags: create_timestamp: Tue Aug 27 11:42:32 2019
[root@ceph-mon-01 ~]# rbd snap ls images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 SNAPID NAME SIZE TIMESTAMP 455500 snap 10GiB Tue Aug 27 11:46:37 2019 [root@ceph-mon-01 ~]#
Cheers, Massimo
On Tue, Aug 27, 2019 at 2:41 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Massimo,
I need to reproduce this first, but pretty much sure this issue is not because you configured multiple backends. What you are doing is downloading existing image to your local storage and then uploading it to glance using add-location operation.
So before adding this using add-location operation have you manually uploaded this image to ceph/rbd? if not then how you are building your location url which you are mentioning in the add-location command? If this location is not existing then it might be a problem.
Thank you,
Abhishek
On Tue, 27 Aug 2019 at 5:09 PM, Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I have a Rocky installation where glance is configured with multiple backends. This [*] is the relevant part of the glance configuration.
I now need to dismiss the file backend and move the images-snapshots stored there to rbd.
I had in mind to follow this procedure (already successfully tested with an older version of OpenStack):
1) download the image from the file backend (glance image-download --file <file> <imageid> 2) upload the file <file> to rbd, relying on this function: https://github.com/openstack/glance_store/blob/stable/rocky/glance_store/_dr... 3) glance location-add --url rbd://xyz <imageid> 4) glance location-delete --url file://abc <imageid>
I have problems with 3:
# glance --debug location-add --url rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap 6bcc4eab-ed35-42dc-88bd-1d45de73b628 ... ... DEBUG:keystoneauth.session:PATCH call to image for https://cloud-areapd.pd.infn.it:9292/v2/images/6bcc4eab-ed35-42dc-88bd-1d45d... used request id req-6c0598cc-582c-4ce7-a14c-5d1bb6ec4f14 Request returned failure status 400. DEBUG:glanceclient.common.http:Request returned failure status 400. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 687, in main OpenStackImagesShell().main(argv) File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 591, in main args.func(client, args) File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 749, in do_location_add image = gc.images.add_location(args.id, args.url, metadata) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 448, in add_location response = self._send_image_update_request(image_id, add_patch) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner return RequestIdProxy(wrapped(*args, **kwargs)) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 432, in _send_image_update_request data=json.dumps(patch_body)) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 340, in patch return self.request(url, 'PATCH', **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request return self._handle_response(resp) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response raise exc.from_response(resp, resp.content) HTTPBadRequest: 400 Bad Request: Invalid location (HTTP 400) 400 Bad Request: Invalid location (HTTP 400)
As far as I can see with the "openstack" client there is not something to add/delete a location. So I guess it is necessary to change the 'direct_url' and 'locations' properties.
If I try to change the direct_url property:
# openstack image set --property direct_url='rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap' 6bcc4eab-ed35-42dc-88bd-1d45de73b628 403 Forbidden: Attribute 'direct_url' is read-only. (HTTP 403)
Any hints ? Thanks, Massimo
[*]
[default] ... enabled_backends = file:file,http:http,rbd:rbd show_image_direct_url = true show_multiple_locations = true
[glance_store] default_backend = rbd
[file] filesystem_store_datadir = /var/lib/glance/images/
[rbd] rbd_store_chunk_size = 8 rbd_store_ceph_conf = /etc/ceph/ceph.conf rbd_store_user = glance-prod rbd_store_pool = images-prod
-- Thanks & Best Regards,
Abhishek Kekane
-- Thanks & Best Regards,
Abhishek Kekane
Hi Abhishek First of all thanks to your help This is a production environment and therefore I have problems updating and deploying the current master I think I will revert to the "old way", since its behavior is fine with me (I don't need to be able to select the backend when creating an image) Thanks again Cheers, Massimo On Wed, Aug 28, 2019 at 11:25 AM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Massimo,
Glance multiple store is still experimental and it does not have support for location API in rocky and stein, we are making it compatible in Train release.
There is still one bug open for location compatibility with multiple stores and patch for it is not merged yet, I will also encourage you to deploy latest master and apply this patch in your environment and test it. https://review.opendev.org/#/c/617229/19
NOTE: I have tested this on current master after applying patch to local environment and the scenario you mentioned is working fine.
Thanks & Best Regards,
Abhishek Kekane
On Tue, Aug 27, 2019 at 7:47 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Massimo, Cool, thank you for confirmation, I will try to reproduce this in my environment. It will take some time, will revert back to you once done. Meanwhile could you please try the same with single store (disabling the multiple stores)?
Thank you,
Abhishek
On Tue, 27 Aug 2019 at 6:33 PM, Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
Hi Abhishek
Yes, before trying the:
glance location-add --url rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap 6bcc4eab-ed35-42dc-88bd-1d45de73b628
command, I uploaded the image to ceph. And indeed I can see it:
[root@ceph-mon-01 ~]# rbd info images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 rbd image '6bcc4eab-ed35-42dc-88bd-1d45de73b628': size 10GiB in 1280 objects order 23 (8MiB objects) block_name_prefix: rbd_data.b7b56aa1e4f0bc format: 2 features: layering flags: create_timestamp: Tue Aug 27 11:42:32 2019
[root@ceph-mon-01 ~]# rbd snap ls images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628 SNAPID NAME SIZE TIMESTAMP 455500 snap 10GiB Tue Aug 27 11:46:37 2019 [root@ceph-mon-01 ~]#
Cheers, Massimo
On Tue, Aug 27, 2019 at 2:41 PM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Massimo,
I need to reproduce this first, but pretty much sure this issue is not because you configured multiple backends. What you are doing is downloading existing image to your local storage and then uploading it to glance using add-location operation.
So before adding this using add-location operation have you manually uploaded this image to ceph/rbd? if not then how you are building your location url which you are mentioning in the add-location command? If this location is not existing then it might be a problem.
Thank you,
Abhishek
On Tue, 27 Aug 2019 at 5:09 PM, Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I have a Rocky installation where glance is configured with multiple backends. This [*] is the relevant part of the glance configuration.
I now need to dismiss the file backend and move the images-snapshots stored there to rbd.
I had in mind to follow this procedure (already successfully tested with an older version of OpenStack):
1) download the image from the file backend (glance image-download --file <file> <imageid> 2) upload the file <file> to rbd, relying on this function: https://github.com/openstack/glance_store/blob/stable/rocky/glance_store/_dr... 3) glance location-add --url rbd://xyz <imageid> 4) glance location-delete --url file://abc <imageid>
I have problems with 3:
# glance --debug location-add --url rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap 6bcc4eab-ed35-42dc-88bd-1d45de73b628 ... ... DEBUG:keystoneauth.session:PATCH call to image for https://cloud-areapd.pd.infn.it:9292/v2/images/6bcc4eab-ed35-42dc-88bd-1d45d... used request id req-6c0598cc-582c-4ce7-a14c-5d1bb6ec4f14 Request returned failure status 400. DEBUG:glanceclient.common.http:Request returned failure status 400. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 687, in main OpenStackImagesShell().main(argv) File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 591, in main args.func(client, args) File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 749, in do_location_add image = gc.images.add_location(args.id, args.url, metadata) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 448, in add_location response = self._send_image_update_request(image_id, add_patch) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 598, in inner return RequestIdProxy(wrapped(*args, **kwargs)) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 432, in _send_image_update_request data=json.dumps(patch_body)) File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 340, in patch return self.request(url, 'PATCH', **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 377, in request return self._handle_response(resp) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 126, in _handle_response raise exc.from_response(resp, resp.content) HTTPBadRequest: 400 Bad Request: Invalid location (HTTP 400) 400 Bad Request: Invalid location (HTTP 400)
As far as I can see with the "openstack" client there is not something to add/delete a location. So I guess it is necessary to change the 'direct_url' and 'locations' properties.
If I try to change the direct_url property:
# openstack image set --property direct_url='rbd://8162f291-00b6-4b40-a8b4-1981a8c09b64/images-prod/6bcc4eab-ed35-42dc-88bd-1d45de73b628/snap' 6bcc4eab-ed35-42dc-88bd-1d45de73b628 403 Forbidden: Attribute 'direct_url' is read-only. (HTTP 403)
Any hints ? Thanks, Massimo
[*]
[default] ... enabled_backends = file:file,http:http,rbd:rbd show_image_direct_url = true show_multiple_locations = true
[glance_store] default_backend = rbd
[file] filesystem_store_datadir = /var/lib/glance/images/
[rbd] rbd_store_chunk_size = 8 rbd_store_ceph_conf = /etc/ceph/ceph.conf rbd_store_user = glance-prod rbd_store_pool = images-prod
-- Thanks & Best Regards,
Abhishek Kekane
-- Thanks & Best Regards,
Abhishek Kekane
participants (2)
-
Abhishek Kekane
-
Massimo Sgaravatto