Hi Folks,
Happy Monday. Just checking back if there were any updates on the logs I shared with you. I was able to put a small hack (just to unblock my work), which determines the size of the image from the staging store(whenever image_size is set to 0 in s3 driver).
On the other hand, while I was playing with the image-import mechanism, I saw couple other issues,
1. It looks like the *image-import *command fails in CLI validation when *--wait *option is not provided.
root@1071c7a187a8:/muon# openstack image import --method copy-image --store file 2fa366df-d28c-43ec-b10c-00454fc8b3d8
usage: openstack image import [-h] [-f {json,shell,table,value,yaml}] [-c COLUMN] [--noindent] [--prefix PREFIX] [--max-width <integer>] [--fit-width] [--print-empty] [--method <method>] [--uri <uri>]
[--remote-image <REMOTE_IMAGE>] [--remote-region <REMOTE_GLANCE_REGION>] [--remote-service-interface <REMOTE_SERVICE_INTERFACE>] [--store [<STORE> ...] | --all-stores ALL_STORES] [--allow-failure]
[--disallow-failure] [--wait]
<image>
*openstack image import: error: the following arguments are required: <image>*
2. The *--wait *option when used does not really block the CLI till the import is completed.
Thanks,
Shrishail
On Thu, 1 Feb 2024 at 11:49, Shrishail K nkshrishail@gmail.com wrote:
[Replying-all]
Sure, *it looks like setting image size enables the call to multipart upload*, here is what I see
- With DEBUGs enabled, with no code changes, attempting to *import *an
image of around *300MB.*
2024-02-01 15:50:28.516 2024-02-01 15:50:28.323 732 DEBUG glance.async_.flows.api_image_import [-] Image c0b3654d-476f-41b7-8fcf-4eb73075bb3d os_glance_importing_to_stores= merge_store_list /var/lib/kolla/venv/lib/python3.10/site-packages/glance/async_/flows/api_image_import.py:278 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG *glance_store._drivers.s3 [-] Wrote 374656512 bytes to S3 key named c0b3654d-476f-41b7-8fcf-4eb73075bb3d with checksum 34ae3c2ddfe1ed0bfab30d7d6ce69f78 _add_singlepart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:690* 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fda44492620>> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238 2024-02-01 15:50:28.516 2024-02-01 15:50:28.288 732 DEBUG botocore.retryhandler [-] No retry needed. __call__ /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/retryhandler.py:211 2024-02-01 15:50:28.516 2024-02-01 15:50:28.287 732 DEBUG botocore.hooks [-] Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fda44492560> _emit /var/lib/kolla/venv/lib/python3.10/site-packages/botocore/hooks.py:238
- With DEBUGs enabled, with *hardcoding the image size*, attempting to
*import *an image of around *300MB (with multipart configs left default)*
2024-02-01 16:35:30.511 2024-02-01 16:35:30.299 731 DEBUG g*lance_store._drivers.s3 [-] Multipart initiate key=c0b3654d-476f-41b7-8fcf-4eb73075bb3d*, UploadId=7HLARegXPw.iQVH72rcAKfGWb4GURDa.F_.tQJQgazGS8BjizAaXogifzck1Pqx6jnL7ENoEqmTEwFpNOBEdIqGb0gmmMKK51zIGmZp8jiDkpbHlb_Xh2XuOLehXGXgoUyLRk9S7kLufUnuxz4YnT69ezd08dIJX.GtrWl.6S_L9pB1Aak.hSFhuB8HfkVjH _add_multipart /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:721 2024-02-01 16:35:30.260 2024-02-01 16:35:30.117 731 DEBUG glance_store._drivers.s3 [-] *NKSHRISHAIL: IMAGE SIZE IS ZERO... SETTING IT TO 374656512 add * /var/lib/kolla/venv/lib/python3.10/site-packages/glance_store/_drivers/s3.py:631
On Thu, 1 Feb 2024 at 03:14, Abhishek Kekane akekane@redhat.com wrote:
On Thu, Feb 1, 2024 at 1:19 PM Christian Rohmann < christian.rohmann@inovex.de> wrote:
On 31.01.24 16:36, Abhishek Kekane wrote:
- Following the discussion, I was getting a sense that the code
changes may not be a lot, in that case, if it's possible to provide a patch, I can patch my openstack and see how it goes.
Even though changes are trivial, glance uses onion architecture, so it needed changes at a lot places which will take some time. Ideally it’s not possible to provide quick fix for this.
Since we have invested some time looking into this, would it not make sense to write this up in a bug and maybe you could add some of your implementation knowledge as well? This then seems like a fruit (be it low-hanging or not) that someone could pick.
Otherwise this bug / missing feature discussion just ends up in the ML archive.
Hi Shrishail, CC: Christian
Is it possible for you to hardcode image_size at [1] and check not passing image_size is the real issue? I have reported a bug [2] and mentioned it as a possible solution, but if that is not the case then we need to investigate further.
[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/... [2] https://bugs.launchpad.net/glance/+bug/2051914
Thanks,
Abhishek
Regards
Christian