Glance image import to AWS S3 fails for files larger than 5GB
Hi Team, I have a use case where I need to replicate images from file store to S3 store. While image creation directly on S3 works fine with large files over 5G. The same fails when attempted via image-import with copy-image method. The failure is due to AWS rejecting a single object upload operation where the object size is over 5GB. What I see from logs is that direct image creation on S3 uses multipart upload when images are large. However, I fail to see the same happening when image import was used to copy an image from file to s3. Is this a known issue? Or Am I missing some configs related to image-import? Openstack version: Antelope Client version: 6.4.0 Thanks, Shrishail
HI Shrishail, By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior. Thanks & Best Regards, Abhishek Kekane On Wed, Jan 31, 2024 at 3:55 AM Shrishail K <nkshrishail@gmail.com> wrote:
Hi Team,
I have a use case where I need to replicate images from file store to S3 store.
While image creation directly on S3 works fine with large files over 5G.
The same fails when attempted via image-import with copy-image method.
The failure is due to AWS rejecting a single object upload operation where the object size is over 5GB.
What I see from logs is that direct image creation on S3 uses multipart upload when images are large. However, I fail to see the same happening when image import was used to copy an image from file to s3.
Is this a known issue? Or Am I missing some configs related to image-import?
Openstack version: Antelope
Client version: 6.4.0
Thanks,
Shrishail
Hey all, On Wed, Jan 31, 2024 at 3:55 AM Shrishail K <nkshrishail@gmail.com> wrote: Hi Team, I have a use case where I need to replicate images from file store to S3 store. While image creation directly on S3 works fine with large files over 5G. Since Glance moved all stores to the glance_store project, this should be the driver at work here: https://github.com/openstack/glance_store/commits/master/glance_store/_drive... And there you see AWS boto3 lib is used, explicitly using multipart uploads. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s...
The same fails when attempted via image-import with copy-image method.
The failure is due to AWS rejecting a single object upload operation where the object size is over 5GB.
What I see from logs is that direct image creation on S3 uses multipart upload when images are large. However, I fail to see the same happening when image import was used to copy an imagefrom file to s3.
Is this a known issue? Or Am I missing some configs related to image-import?
On 31.01.24 09:13, Abhishek Kekane wrote:
By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior. Abishek, I suppose the copy-image is done via this helper here and which are were referring to? https://github.com/openstack/glance/blob/master/glance/async_/flows/_interna...
Can you elaborate on how this is NOT currently able to use the glance_store drivers and the abstraction they actually provide? So in essence this is not just about S3, but about adding support for using the glance_store drivers in the copy-image functionality. That indeed would be a nice improvement. Regards Christian
On Wed, Jan 31, 2024 at 2:36 PM Christian Rohmann < christian.rohmann@inovex.de> wrote:
Hey all,
On Wed, Jan 31, 2024 at 3:55 AM Shrishail K <nkshrishail@gmail.com> wrote:
Hi Team,
I have a use case where I need to replicate images from file store to S3 store.
While image creation directly on S3 works fine with large files over 5G.
Since Glance moved all stores to the glance_store project, this should be the driver at work here:
https://github.com/openstack/glance_store/commits/master/glance_store/_drive...
And there you see AWS boto3 lib is used, explicitly using multipart uploads.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s...
The same fails when attempted via image-import with copy-image method.
The failure is due to AWS rejecting a single object upload operation where the object size is over 5GB.
What I see from logs is that direct image creation on S3 uses multipart upload when images are large. However, I fail to see the same happening when image import was used to copy an image from file to s3.
Is this a known issue? Or Am I missing some configs related to image-import?
On 31.01.24 09:13, Abhishek Kekane wrote:
By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior.
Abishek, I suppose the copy-image is done via this helper here and which are were referring to?
https://github.com/openstack/glance/blob/master/glance/async_/flows/_interna...
Hi Christian, The helper you mention above is responsible to download the existing data at common storage known as staging area (configured using os_glance_staging_store in glance-api.conf) and from there it will be imported to the destination/target store. However debugging further I found that it internally calls store.add method, which means in fact it is using a particular driver call only. I suspect [1] is where it is using single_part as an upload for s3 while copying the image, because we are not passing the size of an existing image to the import call. I think this is driver specific improvement, and requires additional effort to make it work. [1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/... Thanks, Abhishek
Can you elaborate on how this is NOT currently able to use the glance_store drivers and the abstraction they actually provide?
So in essence this is not just about S3, but about adding support for using the glance_store drivers in the copy-image functionality. That indeed would be a nice improvement.
Regards
Christian
Hey Abhishek! On 31.01.24 10:35, Abhishek Kekane wrote:
On 31.01.24 09:13, Abhishek Kekane wrote:
By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior.
Abishek, I suppose the copy-image is done via this helper here and which are were referring to? https://github.com/openstack/glance/blob/master/glance/async_/flows/_interna...
Hi Christian,
The helper you mention above is responsible to download the existing data at common storage known as staging area (configured using os_glance_staging_store in glance-api.conf) and from there it will be imported to the destination/target store. However debugging further I found that it internally calls store.add method, which means in fact it is using a particular driver call only.
I suspect [1] is where it is using single_part as an upload for s3 while copying the image, because we are not passing the size of an existing image to the import call.
I think this is driver specific improvement, and requires additional effort to make it work.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/...
I cannot (quickly) follow your debugging / the calls you mentioned. Could you please raise a bug with your findings to "fix" this? Seems like this is not intended behavior? Here the image size is actually provided when the image is fetched to the staging store: https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d... But what is the next step then to upload the "staged" image into the new target store? In any case, I tend to also disagree that, if missing image_size is the issue, providing it to the add call is a S3 driver specific thing. Other object storages (GCS, Azure Blob, ...) might "like" to know the size as well to adjust their upload strategy. Regards Christian
On Wed, 31 Jan 2024 at 3:49 PM, Christian Rohmann < christian.rohmann@inovex.de> wrote:
Hey Abhishek!
On 31.01.24 10:35, Abhishek Kekane wrote:
On 31.01.24 09:13, Abhishek Kekane wrote:
By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior.
Abishek, I suppose the copy-image is done via this helper here and which are were referring to?
https://github.com/openstack/glance/blob/master/glance/async_/flows/_interna...
Hi Christian,
The helper you mention above is responsible to download the existing data at common storage known as staging area (configured using os_glance_staging_store in glance-api.conf) and from there it will be imported to the destination/target store. However debugging further I found that it internally calls store.add method, which means in fact it is using a particular driver call only.
I suspect [1] is where it is using single_part as an upload for s3 while copying the image, because we are not passing the size of an existing image to the import call.
I think this is driver specific improvement, and requires additional effort to make it work.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/...
I cannot (quickly) follow your debugging / the calls you mentioned. Could you please raise a bug with your findings to "fix" this? Seems like this is not intended behavior?
Here the image size is actually provided when the image is fetched to the staging store: https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d...
Hey Christian, The store you mentioned here is staging store which is a filesystem store and not intended (s3) store, from here after the image import flow will get called which will give call to upload the data from file (staging) store to actual store. You will find it in a method set_image_data from glance/async_/flows/api_image_import.py file. Abhishek
<https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d6c9b4/glance/async_/flows/_internal_plugins/copy_image.py#L122> But what is the next step then to upload the "staged" image into the new target store?
In any case, I tend to also disagree that, if missing image_size is the issue, providing it to the add call is a S3 driver specific thing. Other object storages (GCS, Azure Blob, ...) might "like" to know the size as well to adjust their upload strategy.
Regards
Christian
Thanks folks for your responses. I have a couple of follow up questions/clarifications to quickly fix/hack this issue in my local environment. 1. Can I manually copy any images larger than 5GB to s3 bucket before running image-import command? Would image-import see the file in the S3 and use it instead of trying to copy it all over again? 2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes. Thanks, Shrishail On Wed, 31 Jan 2024 at 05:28, Abhishek Kekane <akekane@redhat.com> wrote:
On Wed, 31 Jan 2024 at 3:49 PM, Christian Rohmann < christian.rohmann@inovex.de> wrote:
Hey Abhishek!
On 31.01.24 10:35, Abhishek Kekane wrote:
On 31.01.24 09:13, Abhishek Kekane wrote:
By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior.
Abishek, I suppose the copy-image is done via this helper here and which are were referring to?
https://github.com/openstack/glance/blob/master/glance/async_/flows/_interna...
Hi Christian,
The helper you mention above is responsible to download the existing data at common storage known as staging area (configured using os_glance_staging_store in glance-api.conf) and from there it will be imported to the destination/target store. However debugging further I found that it internally calls store.add method, which means in fact it is using a particular driver call only.
I suspect [1] is where it is using single_part as an upload for s3 while copying the image, because we are not passing the size of an existing image to the import call.
I think this is driver specific improvement, and requires additional effort to make it work.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/...
I cannot (quickly) follow your debugging / the calls you mentioned. Could you please raise a bug with your findings to "fix" this? Seems like this is not intended behavior?
Here the image size is actually provided when the image is fetched to the staging store: https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d...
Hey Christian,
The store you mentioned here is staging store which is a filesystem store and not intended (s3) store, from here after the image import flow will get called which will give call to upload the data from file (staging) store to actual store. You will find it in a method set_image_data from glance/async_/flows/api_image_import.py file.
Abhishek
<https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d6c9b4/glance/async_/flows/_internal_plugins/copy_image.py#L122> But what is the next step then to upload the "staged" image into the new target store?
In any case, I tend to also disagree that, if missing image_size is the issue, providing it to the add call is a S3 driver specific thing. Other object storages (GCS, Azure Blob, ...) might "like" to know the size as well to adjust their upload strategy.
Regards
Christian
Hi Shrishail, On Wed, 31 Jan 2024 at 8:49 PM, Shrishail K <nkshrishail@gmail.com> wrote:
Thanks folks for your responses.
I have a couple of follow up questions/clarifications to quickly fix/hack this issue in my local environment.
1. Can I manually copy any images larger than 5GB to s3 bucket before running image-import command? Would image-import see the file in the S3 and use it instead of trying to copy it all over again?
I think (not sure it will work for s3 or not), after manually copying image to s3 you can use add-location command to add location to the existing image. Image import will not work in this case.
2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes.
Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this. Thanks, Abhishek
Thanks, Shrishail
On Wed, 31 Jan 2024 at 05:28, Abhishek Kekane <akekane@redhat.com> wrote:
On Wed, 31 Jan 2024 at 3:49 PM, Christian Rohmann < christian.rohmann@inovex.de> wrote:
Hey Abhishek!
On 31.01.24 10:35, Abhishek Kekane wrote:
On 31.01.24 09:13, Abhishek Kekane wrote:
By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior.
Abishek, I suppose the copy-image is done via this helper here and which are were referring to?
https://github.com/openstack/glance/blob/master/glance/async_/flows/_interna...
Hi Christian,
The helper you mention above is responsible to download the existing data at common storage known as staging area (configured using os_glance_staging_store in glance-api.conf) and from there it will be imported to the destination/target store. However debugging further I found that it internally calls store.add method, which means in fact it is using a particular driver call only.
I suspect [1] is where it is using single_part as an upload for s3 while copying the image, because we are not passing the size of an existing image to the import call.
I think this is driver specific improvement, and requires additional effort to make it work.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/...
I cannot (quickly) follow your debugging / the calls you mentioned. Could you please raise a bug with your findings to "fix" this? Seems like this is not intended behavior?
Here the image size is actually provided when the image is fetched to the staging store: https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d...
Hey Christian,
The store you mentioned here is staging store which is a filesystem store and not intended (s3) store, from here after the image import flow will get called which will give call to upload the data from file (staging) store to actual store. You will find it in a method set_image_data from glance/async_/flows/api_image_import.py file.
Abhishek
<https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d6c9b4/glance/async_/flows/_internal_plugins/copy_image.py#L122> But what is the next step then to upload the "staged" image into the new target store?
In any case, I tend to also disagree that, if missing image_size is the issue, providing it to the add call is a S3 driver specific thing. Other object storages (GCS, Azure Blob, ...) might "like" to know the size as well to adjust their upload strategy.
Regards
Christian
I didn’t find add-location option under openstack image umbrella. Is it “openstack image set —property “store=s3”” ? Thanks, Shrishail On Wed, Jan 31, 2024 at 10:37 AM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Shrishail,
On Wed, 31 Jan 2024 at 8:49 PM, Shrishail K <nkshrishail@gmail.com> wrote:
Thanks folks for your responses.
I have a couple of follow up questions/clarifications to quickly fix/hack this issue in my local environment.
1. Can I manually copy any images larger than 5GB to s3 bucket before running image-import command? Would image-import see the file in the S3 and use it instead of trying to copy it all over again?
I think (not sure it will work for s3 or not), after manually copying image to s3 you can use add-location command to add location to the existing image. Image import will not work in this case.
2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes.
Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this.
Thanks,
Abhishek
Thanks, Shrishail
On Wed, 31 Jan 2024 at 05:28, Abhishek Kekane <akekane@redhat.com> wrote:
On Wed, 31 Jan 2024 at 3:49 PM, Christian Rohmann < christian.rohmann@inovex.de> wrote:
Hey Abhishek!
On 31.01.24 10:35, Abhishek Kekane wrote:
On 31.01.24 09:13, Abhishek Kekane wrote:
By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior.
Abishek, I suppose the copy-image is done via this helper here and which are were referring to?
https://github.com/openstack/glance/blob/master/glance/async_/flows/_interna...
Hi Christian,
The helper you mention above is responsible to download the existing data at common storage known as staging area (configured using os_glance_staging_store in glance-api.conf) and from there it will be imported to the destination/target store. However debugging further I found that it internally calls store.add method, which means in fact it is using a particular driver call only.
I suspect [1] is where it is using single_part as an upload for s3 while copying the image, because we are not passing the size of an existing image to the import call.
I think this is driver specific improvement, and requires additional effort to make it work.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/...
I cannot (quickly) follow your debugging / the calls you mentioned. Could you please raise a bug with your findings to "fix" this? Seems like this is not intended behavior?
Here the image size is actually provided when the image is fetched to the staging store: https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d...
Hey Christian,
The store you mentioned here is staging store which is a filesystem store and not intended (s3) store, from here after the image import flow will get called which will give call to upload the data from file (staging) store to actual store. You will find it in a method set_image_data from glance/async_/flows/api_image_import.py file.
Abhishek
<https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d6c9b4/glance/async_/flows/_internal_plugins/copy_image.py#L122> But what is the next step then to upload the "staged" image into the new target store?
In any case, I tend to also disagree that, if missing image_size is the issue, providing it to the add call is a S3 driver specific thing. Other object storages (GCS, Azure Blob, ...) might "like" to know the size as well to adjust their upload strategy.
Regards
Christian
Support for adding location is not available in the openstack client yet. You need to use the equivalent glance client command for it. glance location-add <image-id> --url <your image url> Thanks, Abhishek On Thu, Feb 1, 2024 at 12:34 AM Shrishail K <nkshrishail@gmail.com> wrote:
I didn’t find add-location option under openstack image umbrella. Is it “openstack image set —property “store=s3”” ?
Thanks, Shrishail
On Wed, Jan 31, 2024 at 10:37 AM Abhishek Kekane <akekane@redhat.com> wrote:
Hi Shrishail,
On Wed, 31 Jan 2024 at 8:49 PM, Shrishail K <nkshrishail@gmail.com> wrote:
Thanks folks for your responses.
I have a couple of follow up questions/clarifications to quickly fix/hack this issue in my local environment.
1. Can I manually copy any images larger than 5GB to s3 bucket before running image-import command? Would image-import see the file in the S3 and use it instead of trying to copy it all over again?
I think (not sure it will work for s3 or not), after manually copying image to s3 you can use add-location command to add location to the existing image. Image import will not work in this case.
2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes.
Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this.
Thanks,
Abhishek
Thanks, Shrishail
On Wed, 31 Jan 2024 at 05:28, Abhishek Kekane <akekane@redhat.com> wrote:
On Wed, 31 Jan 2024 at 3:49 PM, Christian Rohmann < christian.rohmann@inovex.de> wrote:
Hey Abhishek!
On 31.01.24 10:35, Abhishek Kekane wrote:
On 31.01.24 09:13, Abhishek Kekane wrote:
By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior.
Abishek, I suppose the copy-image is done via this helper here and which are were referring to?
https://github.com/openstack/glance/blob/master/glance/async_/flows/_interna...
Hi Christian,
The helper you mention above is responsible to download the existing data at common storage known as staging area (configured using os_glance_staging_store in glance-api.conf) and from there it will be imported to the destination/target store. However debugging further I found that it internally calls store.add method, which means in fact it is using a particular driver call only.
I suspect [1] is where it is using single_part as an upload for s3 while copying the image, because we are not passing the size of an existing image to the import call.
I think this is driver specific improvement, and requires additional effort to make it work.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/...
I cannot (quickly) follow your debugging / the calls you mentioned. Could you please raise a bug with your findings to "fix" this? Seems like this is not intended behavior?
Here the image size is actually provided when the image is fetched to the staging store: https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d...
Hey Christian,
The store you mentioned here is staging store which is a filesystem store and not intended (s3) store, from here after the image import flow will get called which will give call to upload the data from file (staging) store to actual store. You will find it in a method set_image_data from glance/async_/flows/api_image_import.py file.
Abhishek
<https://github.com/openstack/glance/blob/b6b9f043ffe664c643456912148648ecc0d6c9b4/glance/async_/flows/_internal_plugins/copy_image.py#L122> But what is the next step then to upload the "staged" image into the new target store?
In any case, I tend to also disagree that, if missing image_size is the issue, providing it to the add call is a S3 driver specific thing. Other object storages (GCS, Azure Blob, ...) might "like" to know the size as well to adjust their upload strategy.
Regards
Christian
On 31.01.24 16:36, Abhishek Kekane wrote:
2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes.
Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this.
Since we have invested some time looking into this, would it not make sense to write this up in a bug and maybe you could add some of your implementation knowledge as well? This then seems like a fruit (be it low-hanging or not) that someone could pick. Otherwise this bug / missing feature discussion just ends up in the ML archive. Regards Christian
On Thu, Feb 1, 2024 at 1:19 PM Christian Rohmann < christian.rohmann@inovex.de> wrote:
On 31.01.24 16:36, Abhishek Kekane wrote:
2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes.
Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this.
Since we have invested some time looking into this, would it not make sense to write this up in a bug and maybe you could add some of your implementation knowledge as well? This then seems like a fruit (be it low-hanging or not) that someone could pick.
Otherwise this bug / missing feature discussion just ends up in the ML archive.
Hi Shrishail, CC: Christian Is it possible for you to hardcode image_size at [1] and check not passing image_size is the real issue? I have reported a bug [2] and mentioned it as a possible solution, but if that is not the case then we need to investigate further. [1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/... [2] https://bugs.launchpad.net/glance/+bug/2051914 Thanks, Abhishek
Regards
Christian
[Replying-all] Sure, *it looks like setting image size enables the call to multipart upload*, here is what I see 1. With DEBUGs enabled, with no code changes, attempting to *import *an image of around *300MB.* 2024-02-01 15:50:28.516 2024-02-01 15:50:28.323 732 DEBUG glance.async_.flows.api_image_import [-] Image c0b3654d-476f-41b7-8fcf-4eb73075bb3d os_glance_importing_to_stores= merge_store_list /var/lib/kolla/venv/lib/python3.10/site-packages/glance/async_/flows/api_image_import.py:278 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG *glance_store._drivers.s3 [-] Wrote 374656512 bytes to S3 key named c0b3654d-476f-41b7-8fcf-4eb73075bb3d with checksum 34ae3c2ddfe1ed0bfab30d7d6ce69f78 _add_singlepart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:690* 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fda44492620>> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG botocore.retryhandler [-] No retry needed. __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/retryhandler.py:211 2024-02-01 15:50:28.516 2024-02-01 15:50:28.287 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fda44492560> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238 2. With DEBUGs enabled, with *hardcoding the image size*, attempting to *import *an image of around *300MB (with multipart configs left default)* 2024-02-01 16:35:30.511 2024-02-01 16:35:30.299 731 DEBUG g*lance_store._drivers.s3 [-] Multipart initiate key=c0b3654d-476f-41b7-8fcf-4eb73075bb3d*, UploadId=7HLARegXPw.iQVH72rcAKfGWb4GURDa.F_.tQJQgazGS8BjizAaXogifzck1Pqx6jnL7ENoEqmTEwFpNOBEdIqGb0gmmMKK51zIGmZp8jiDkpbHlb_Xh2XuOLehXGXgoUyLRk9S7kLufUnuxz4YnT69ezd08dIJX.GtrWl.6S_L9pB1Aak.hSFhuB8HfkVjH _add_multipart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:721 2024-02-01 16:35:30.260 2024-02-01 16:35:30.117 731 DEBUG glance_store._drivers.s3 [-] *NKSHRISHAIL: IMAGE SIZE IS ZERO... SETTING IT TO 374656512 add * /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:631 On Thu, 1 Feb 2024 at 03:14, Abhishek Kekane <akekane@redhat.com> wrote:
On Thu, Feb 1, 2024 at 1:19 PM Christian Rohmann < christian.rohmann@inovex.de> wrote:
On 31.01.24 16:36, Abhishek Kekane wrote:
2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes.
Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this.
Since we have invested some time looking into this, would it not make sense to write this up in a bug and maybe you could add some of your implementation knowledge as well? This then seems like a fruit (be it low-hanging or not) that someone could pick.
Otherwise this bug / missing feature discussion just ends up in the ML archive.
Hi Shrishail, CC: Christian
Is it possible for you to hardcode image_size at [1] and check not passing image_size is the real issue? I have reported a bug [2] and mentioned it as a possible solution, but if that is not the case then we need to investigate further.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/... [2] https://bugs.launchpad.net/glance/+bug/2051914
Thanks,
Abhishek
Regards
Christian
Hi Folks, Happy Monday. Just checking back if there were any updates on the logs I shared with you. I was able to put a small hack (just to unblock my work), which determines the size of the image from the staging store(whenever image_size is set to 0 in s3 driver). On the other hand, while I was playing with the image-import mechanism, I saw couple other issues, 1. It looks like the *image-import *command fails in CLI validation when *--wait *option is not provided. root@1071c7a187a8:/muon# openstack image import --method copy-image --store file 2fa366df-d28c-43ec-b10c-00454fc8b3d8 usage: openstack image import [-h] [-f {json,shell,table,value,yaml}] [-c COLUMN] [--noindent] [--prefix PREFIX] [--max-width <integer>] [--fit-width] [--print-empty] [--method <method>] [--uri <uri>] [--remote-image <REMOTE_IMAGE>] [--remote-region <REMOTE_GLANCE_REGION>] [--remote-service-interface <REMOTE_SERVICE_INTERFACE>] [--store [<STORE> ...] | --all-stores ALL_STORES] [--allow-failure] [--disallow-failure] [--wait] <image> *openstack image import: error: the following arguments are required: <image>* 2. The *--wait *option when used does not really block the CLI till the import is completed. Thanks, Shrishail On Thu, 1 Feb 2024 at 11:49, Shrishail K <nkshrishail@gmail.com> wrote:
[Replying-all]
Sure, *it looks like setting image size enables the call to multipart upload*, here is what I see
1. With DEBUGs enabled, with no code changes, attempting to *import *an image of around *300MB.*
2024-02-01 15:50:28.516 2024-02-01 15:50:28.323 732 DEBUG glance.async_.flows.api_image_import [-] Image c0b3654d-476f-41b7-8fcf-4eb73075bb3d os_glance_importing_to_stores= merge_store_list /var/lib/kolla/venv/lib/python3.10/site-packages/glance/async_/flows/api_image_import.py:278 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG *glance_store._drivers.s3 [-] Wrote 374656512 bytes to S3 key named c0b3654d-476f-41b7-8fcf-4eb73075bb3d with checksum 34ae3c2ddfe1ed0bfab30d7d6ce69f78 _add_singlepart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:690* 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fda44492620>> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG botocore.retryhandler [-] No retry needed. __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/retryhandler.py:211 2024-02-01 15:50:28.516 2024-02-01 15:50:28.287 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fda44492560> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238
2. With DEBUGs enabled, with *hardcoding the image size*, attempting to *import *an image of around *300MB (with multipart configs left default)*
2024-02-01 16:35:30.511 2024-02-01 16:35:30.299 731 DEBUG g*lance_store._drivers.s3 [-] Multipart initiate key=c0b3654d-476f-41b7-8fcf-4eb73075bb3d*, UploadId=7HLARegXPw.iQVH72rcAKfGWb4GURDa.F_.tQJQgazGS8BjizAaXogifzck1Pqx6jnL7ENoEqmTEwFpNOBEdIqGb0gmmMKK51zIGmZp8jiDkpbHlb_Xh2XuOLehXGXgoUyLRk9S7kLufUnuxz4YnT69ezd08dIJX.GtrWl.6S_L9pB1Aak.hSFhuB8HfkVjH _add_multipart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:721 2024-02-01 16:35:30.260 2024-02-01 16:35:30.117 731 DEBUG glance_store._drivers.s3 [-] *NKSHRISHAIL: IMAGE SIZE IS ZERO... SETTING IT TO 374656512 add * /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:631
On Thu, 1 Feb 2024 at 03:14, Abhishek Kekane <akekane@redhat.com> wrote:
On Thu, Feb 1, 2024 at 1:19 PM Christian Rohmann < christian.rohmann@inovex.de> wrote:
On 31.01.24 16:36, Abhishek Kekane wrote:
2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes.
Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this.
Since we have invested some time looking into this, would it not make sense to write this up in a bug and maybe you could add some of your implementation knowledge as well? This then seems like a fruit (be it low-hanging or not) that someone could pick.
Otherwise this bug / missing feature discussion just ends up in the ML archive.
Hi Shrishail, CC: Christian
Is it possible for you to hardcode image_size at [1] and check not passing image_size is the real issue? I have reported a bug [2] and mentioned it as a possible solution, but if that is not the case then we need to investigate further.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/... [2] https://bugs.launchpad.net/glance/+bug/2051914
Thanks,
Abhishek
Regards
Christian
On Mon, Feb 5, 2024 at 10:03 PM Shrishail K <nkshrishail@gmail.com> wrote: Hi Shrishail,
Hi Folks,
Happy Monday. Just checking back if there were any updates on the logs I shared with you. I was able to put a small hack (just to unblock my work), which determines the size of the image from the staging store(whenever image_size is set to 0 in s3 driver).
I haven't got time to look further into it, unfortunately we are approaching towards end of current development cycle and I need to looked at current tasks in hand for the same. I will take this up in upcoming development cycle PTG for discussion. The other problem for me is I don't have active s3 setup so cannot verify the fix at my end.
On the other hand, while I was playing with the image-import mechanism, I saw couple other issues,
1. It looks like the *image-import *command fails in CLI validation when *--wait *option is not provided.
root@1071c7a187a8:/muon# openstack image import --method copy-image --store file 2fa366df-d28c-43ec-b10c-00454fc8b3d8
usage: openstack image import [-h] [-f {json,shell,table,value,yaml}] [-c COLUMN] [--noindent] [--prefix PREFIX] [--max-width <integer>] [--fit-width] [--print-empty] [--method <method>] [--uri <uri>]
[--remote-image <REMOTE_IMAGE>] [--remote-region <REMOTE_GLANCE_REGION>] [--remote-service-interface <REMOTE_SERVICE_INTERFACE>] [--store [<STORE> ...] | --all-stores ALL_STORES] [--allow-failure]
[--disallow-failure] [--wait]
<image>
*openstack image import: error: the following arguments are required: <image>*
2. The *--wait *option when used does not really block the CLI till the import is completed.
This bug is from openstack client and not glance client. If you use glance client command (glance image-import or image-create-via-import) it will work since we don't have any support for --wait option.
Thanks,
Shrishail
On Thu, 1 Feb 2024 at 11:49, Shrishail K <nkshrishail@gmail.com> wrote:
[Replying-all]
Sure, *it looks like setting image size enables the call to multipart upload*, here is what I see
1. With DEBUGs enabled, with no code changes, attempting to *import *an image of around *300MB.*
2024-02-01 15:50:28.516 2024-02-01 15:50:28.323 732 DEBUG glance.async_.flows.api_image_import [-] Image c0b3654d-476f-41b7-8fcf-4eb73075bb3d os_glance_importing_to_stores= merge_store_list /var/lib/kolla/venv/lib/python3.10/site-packages/glance/async_/flows/api_image_import.py:278 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG *glance_store._drivers.s3 [-] Wrote 374656512 bytes to S3 key named c0b3654d-476f-41b7-8fcf-4eb73075bb3d with checksum 34ae3c2ddfe1ed0bfab30d7d6ce69f78 _add_singlepart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:690* 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fda44492620>> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG botocore.retryhandler [-] No retry needed. __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/retryhandler.py:211 2024-02-01 15:50:28.516 2024-02-01 15:50:28.287 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fda44492560> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238
2. With DEBUGs enabled, with *hardcoding the image size*, attempting to *import *an image of around *300MB (with multipart configs left default)*
2024-02-01 16:35:30.511 2024-02-01 16:35:30.299 731 DEBUG g*lance_store._drivers.s3 [-] Multipart initiate key=c0b3654d-476f-41b7-8fcf-4eb73075bb3d*, UploadId=7HLARegXPw.iQVH72rcAKfGWb4GURDa.F_.tQJQgazGS8BjizAaXogifzck1Pqx6jnL7ENoEqmTEwFpNOBEdIqGb0gmmMKK51zIGmZp8jiDkpbHlb_Xh2XuOLehXGXgoUyLRk9S7kLufUnuxz4YnT69ezd08dIJX.GtrWl.6S_L9pB1Aak.hSFhuB8HfkVjH _add_multipart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:721 2024-02-01 16:35:30.260 2024-02-01 16:35:30.117 731 DEBUG glance_store._drivers.s3 [-] *NKSHRISHAIL: IMAGE SIZE IS ZERO... SETTING IT TO 374656512 add * /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:631
On Thu, 1 Feb 2024 at 03:14, Abhishek Kekane <akekane@redhat.com> wrote:
On Thu, Feb 1, 2024 at 1:19 PM Christian Rohmann < christian.rohmann@inovex.de> wrote:
On 31.01.24 16:36, Abhishek Kekane wrote:
2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes.
Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this.
Since we have invested some time looking into this, would it not make sense to write this up in a bug and maybe you could add some of your implementation knowledge as well? This then seems like a fruit (be it low-hanging or not) that someone could pick.
Otherwise this bug / missing feature discussion just ends up in the ML archive.
Hi Shrishail, CC: Christian
Is it possible for you to hardcode image_size at [1] and check not passing image_size is the real issue? I have reported a bug [2] and mentioned it as a possible solution, but if that is not the case then we need to investigate further.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/... [2] https://bugs.launchpad.net/glance/+bug/2051914
Thanks,
Abhishek
Regards
Christian
participants (3)
-
Abhishek Kekane
-
Christian Rohmann
-
Shrishail K