[Replying-all]

Sure, it looks like setting image size enables the call to multipart upload, here is what I see 

1. With DEBUGs enabled, with no code changes, attempting to import an image of around 300MB.

2024-02-01 15:50:28.516
2024-02-01 15:50:28.323 732 DEBUG glance.async_.flows.api_image_import [-] Image c0b3654d-476f-41b7-8fcf-4eb73075bb3d os_glance_importing_to_stores= merge_store_list /var/lib/kolla/venv/lib/python3.10/site-packages/glance/async_/flows/api_image_import.py:278
2024-02-01 15:50:28.516
2024-02-01 15:50:28.288 732 DEBUG glance_store._drivers.s3 [-] Wrote 374656512 bytes to S3 key named c0b3654d-476f-41b7-8fcf-4eb73075bb3d with checksum 34ae3c2ddfe1ed0bfab30d7d6ce69f78 _add_singlepart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:690
2024-02-01 15:50:28.516
2024-02-01 15:50:28.288 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fda44492620>> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238
2024-02-01 15:50:28.516
2024-02-01 15:50:28.288 732 DEBUG botocore.retryhandler [-] No retry needed. __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/retryhandler.py:211
2024-02-01 15:50:28.516
2024-02-01 15:50:28.287 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fda44492560> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238

2. With DEBUGs enabled, with hardcoding the image size, attempting to import an image of around 300MB (with multipart configs left default)

2024-02-01 16:35:30.511
2024-02-01 16:35:30.299 731 DEBUG glance_store._drivers.s3 [-] Multipart initiate key=c0b3654d-476f-41b7-8fcf-4eb73075bb3d, UploadId=7HLARegXPw.iQVH72rcAKfGWb4GURDa.F_.tQJQgazGS8BjizAaXogifzck1Pqx6jnL7ENoEqmTEwFpNOBEdIqGb0gmmMKK51zIGmZp8jiDkpbHlb_Xh2XuOLehXGXgoUyLRk9S7kLufUnuxz4YnT69ezd08dIJX.GtrWl.6S_L9pB1Aak.hSFhuB8HfkVjH _add_multipart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:721
2024-02-01 16:35:30.260
2024-02-01 16:35:30.117 731 DEBUG glance_store._drivers.s3 [-] NKSHRISHAIL: IMAGE SIZE IS ZERO... SETTING IT TO 374656512 add /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:631

On Thu, 1 Feb 2024 at 03:14, Abhishek Kekane <akekane@redhat.com> wrote:


On Thu, Feb 1, 2024 at 1:19 PM Christian Rohmann <christian.rohmann@inovex.de> wrote:

On 31.01.24 16:36, Abhishek Kekane wrote:

2. Following the discussion, I was getting a sense that the code changes may not be a lot, in that case, if it's possible to provide a patch,
    I can patch my openstack and see how it goes. 

Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this.

Since we have invested some time looking into this, would it not make sense to write this up in a bug and maybe you could add some of your implementation knowledge as well?
This then seems like a fruit (be it low-hanging or not) that someone could pick.


Otherwise this bug / missing feature discussion just ends up in the ML archive.

Hi Shrishail,
CC: Christian

Is it possible for you to hardcode image_size at [1] and check not passing image_size is the real issue?
I have reported a bug [2] and mentioned it as a possible solution, but if that is not the case then we need to investigate further.


[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/s3.py#L628
[2] https://bugs.launchpad.net/glance/+bug/2051914

Thanks,

Abhishek

 


Regards


Christian