Hey all, On Wed, Jan 31, 2024 at 3:55 AM Shrishail K <nkshrishail@gmail.com> wrote: Hi Team, I have a use case where I need to replicate images from file store to S3 store. While image creation directly on S3 works fine with large files over 5G. Since Glance moved all stores to the glance_store project, this should be the driver at work here: https://github.com/openstack/glance_store/commits/master/glance_store/_drive... And there you see AWS boto3 lib is used, explicitly using multipart uploads. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s...
The same fails when attempted via image-import with copy-image method.
The failure is due to AWS rejecting a single object upload operation where the object size is over 5GB.
What I see from logs is that direct image creation on S3 uses multipart upload when images are large. However, I fail to see the same happening when image import was used to copy an imagefrom file to s3.
Is this a known issue? Or Am I missing some configs related to image-import?
On 31.01.24 09:13, Abhishek Kekane wrote:
By design copy-image import workflow uses a common uploading mechanism for all stores, so yes it is a known limitation if it is not using multipart upload for s3 backend. Feel free to propose enhancement for the same or participate in the upcoming PTG 'April 8-12, 2024' to discuss the improvements for this behavior. Abishek, I suppose the copy-image is done via this helper here and which are were referring to? https://github.com/openstack/glance/blob/master/glance/async_/flows/_interna...
Can you elaborate on how this is NOT currently able to use the glance_store drivers and the abstraction they actually provide? So in essence this is not just about S3, but about adding support for using the glance_store drivers in the copy-image functionality. That indeed would be a nice improvement. Regards Christian