[openstack helm] [Glance] Glance showing OSError: unable to receive chunked part when uploading image
Hello i'm testing using Openstack-helm to install all component but after the installation complete i can't upload the OS image through both openstack CLI or Horizon it stuck at Queueing and when i checking Glance Api log it shows 2024-11-07 07:31:19.362 7 INFO glance.api.v2.image_data [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. 2024-11-07 07:31:19.400 7 ERROR glance.api.v2.image_data [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Failed to upload image data due to internal error: OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Caught error: unable to receive chunked part: OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi Traceback (most recent call last): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1302, in __call__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1345, in dispatch 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return method(*args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 415, in wrapped 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/api/v2/image_data.py", line 299, in upload 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi with excutils.save_and_reraise_exception(): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.force_reraise() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise self.value 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/api/v2/image_data.py", line 162, in upload 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/notifier.py", line 492, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi with excutils.save_and_reraise_exception(): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.force_reraise() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise self.value 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/notifier.py", line 443, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/quota/__init__.py", line 322, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/location.py", line 596, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/location.py", line 487, in _upload_to_store 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/multi_backend.py", line 397, in add_with_multihash 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/multi_backend.py", line 479, in store_add_to_backend_with_multihash 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/driver.py", line 294, in add_adapter 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/capabilities.py", line 176, in op_checker 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/_drivers/filesystem.py", line 764, in add 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise errors.get(e.errno, e) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/_drivers/filesystem.py", line 746, in add 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi for buf in utils.chunkreadable(image_file, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/common/utils.py", line 73, in chunkiter 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi chunk = fp.read(chunk_size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 294, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi result = self.data.read(i) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 121, in readfn 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi result = fd.read(*args) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/format_inspector.py", line 956, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi chunk = self._source.read(size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1045, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi data = uwsgi.chunked_read() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi Ceph Cluster Status: NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL FSID ceph /var/lib/rook 3 82m Ready Cluster created successfully HEALTH_OK 5cf8b9bb-13e1-4c31-b47d-9e7d92c3c05d Ceph osd status: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 0.35156 - 360 GiB 3.3 GiB 493 MiB 0 B 2.8 GiB 357 GiB 0.91 1.00 - root default -5 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 964 MiB 119 GiB 0.92 1.01 - host instance-lu-1 2 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 964 MiB 119 GiB 0.92 1.01 195 up osd.2 -3 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 - host instance-lu-2 0 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 195 up osd.0 -7 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 - host instance-lu-3 1 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 197 up osd.1 TOTAL 360 GiB 3.3 GiB 493 MiB 0 B 2.8 GiB 357 GiB 0.91 MIN/MAX VAR: 1.00/1.01 STDDEV: 0.01 now i'm able to create Empty Volume using horizon(can't create volume using default cirros image ) and i can start an Instance without volume and mount the empty volume in(Ceph also can detect when i use that volume) is there something i need to set when starting glance? because i'm following the Openstack-helm installation document to install openstack with helm
Update on this it turn out using Openstack-CLI actually works fine when creating image (but still not able to create one from horizon) but using that image to create a volume shows error on cinder (I'm able to create Empty volume) <Error log on cinder scheduler>: 2024-11-08 00:24:53.390 1 ERROR cinder.scheduler.flows.create_volume [None req-bf79602b-ed01-458d-ad99-6ffbb2fd5418 123b6c97e5334e4095792ccb8d25d24a 39fd6b630b8d4dcd942aa440567a4c89 - - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource 230b4903-b660-4270-8a74-11f818786a7d: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource 230b4903-b660-4270-8a74-11f818786a7d <Error log on cinder volume>: 2024-11-08 00:24:53.307 8 ERROR cinder.volume.manager File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 201, in exchange_ping 2024-11-08 00:24:53.307 8 ERROR cinder.volume.manager raise FailedToDropPrivileges(msg) 2024-11-08 00:24:53.307 8 ERROR cinder.volume.manager oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start 2024-11-08 00:24:53.307 8 ERROR cinder.volume.manager now it looks like it's cinder's problem ?
So yes. Unfortunately there’s not a good fix because of how uWSGI works. Essentially the default in OpenStack Helm is to use 1 uWSGI process, which would normally be the Kubernetes way (not having a process manager inside of a pod and using horizontal pod scaling). But when uWSGI is servicing a request when it’s only 1 process, it won’t even service the health check from Kubernetes. So Kubernetes marks it unhealthy and sends it a signal to shutdown. Except uWSGI never handled signals correctly. OpenStack Helm attempts to handle that by setting https://opendev.org/openstack/openstack-helm/src/commit/79d4b68951d17a00d845... to “gracefully_kill_them_all”. Unfortunately that just means it spikes the http router and uWSGI dies on the next epoll() fire so it doesn’t finish the interaction. This has been broken since https://review.opendev.org/c/openstack/openstack-helm/+/899864 merged. The issue with uWSGI has been known since at least 2018 (see https://github.com/unbit/uwsgi/issues/1795 ) It won’t be fixed either since uWSGI is deprecated by upstream and the underlying issue is called out in their docs as suppose to be fixed in the next version (2.1) but that version won’t happen. The “hack” is to increase the number of processes. But the real fix is to move away from uWSGI. Hope that helps. — Doug
On Feb 1, 2025, at 7:00 AM, cbh12032@gmail.com wrote:
Did you find the solution? I also had the same problem as you. (exactly)
Thanks for the reply I Was able to fix Cinder but Not Glance that's why i jump to KubeVirt and leave Openstack while i'm on Kubernetes Now i know the reason behind this error ( I thought it's Network issue before ) Daniel
I suspect the ultimate reason why gracefully_kill_them_all isn’t working correctly with uWSGI is because the OpenStack Helm images are based on a distro that doesn’t have the following patch applied to its uWSGI. https://github.com/unbit/uwsgi/commit/337ba4cb6b2c3a6aabdf629083d883b35b8c46... — Doug
On Feb 3, 2025, at 6:31 PM, Doug Goldstein <cardoe@cardoe.com> wrote:
So yes. Unfortunately there’s not a good fix because of how uWSGI works. Essentially the default in OpenStack Helm is to use 1 uWSGI process, which would normally be the Kubernetes way (not having a process manager inside of a pod and using horizontal pod scaling). But when uWSGI is servicing a request when it’s only 1 process, it won’t even service the health check from Kubernetes. So Kubernetes marks it unhealthy and sends it a signal to shutdown. Except uWSGI never handled signals correctly. OpenStack Helm attempts to handle that by setting https://opendev.org/openstack/openstack-helm/src/commit/79d4b68951d17a00d845... to “gracefully_kill_them_all”. Unfortunately that just means it spikes the http router and uWSGI dies on the next epoll() fire so it doesn’t finish the interaction.
This has been broken since https://review.opendev.org/c/openstack/openstack-helm/+/899864 merged. The issue with uWSGI has been known since at least 2018 (see https://github.com/unbit/uwsgi/issues/1795 ) It won’t be fixed either since uWSGI is deprecated by upstream and the underlying issue is called out in their docs as suppose to be fixed in the next version (2.1) but that version won’t happen.
The “hack” is to increase the number of processes. But the real fix is to move away from uWSGI.
Hope that helps. — Doug
On Feb 1, 2025, at 7:00 AM, cbh12032@gmail.com wrote:
Did you find the solution? I also had the same problem as you. (exactly)
Hi Doug, I increased the number of processes like you recommended, but I still receive the "unable to receive chunked part" error. Anything you would recommend I investigate to rectify this issue?
I had a similar issue, and what worked for me was copying certain relevant configurations from Atmosphere's values into my own configuration. My values are: ``` conf: glance: # DEFAULT: # workers: 2 glance_api_uwsgi: uwsgi: chunked-input-limit: "4096000" http-auto-chunked: true http-raw-body: true socket-timeout: 10 ``` I copied unique, unset values from here: https://github.com/vexxhost/atmosphere/blob/main/charts/glance/values.yaml
participants (5)
-
cbh12032＠gmail.com
-
daniel890723＠gmail.com
-
Doug Goldstein
-
jeff.fonseca777＠gmail.com
-
jeff.slapp＠steeldomecyber.com