Folks,

Following options fixed my problem but it has some security issues involved. 
show_multiple_locations=True 

On Fri, Jun 23, 2023 at 6:26 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,

I am running kolla-ansible on small environments with ceph. When I am getting the following error when performing a VM snapshot. 

This started happening after I upgraded from yoga to Zed. Any idea what changed here? 


2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return self._client.call(
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 191, in call
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     result = getattr(controller, method)(*args, **kwargs)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/v2/images.py", line 503, in add_location
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     response = self._send_image_update_request(image_id, add_patch)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/utils.py", line 670, in inner
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return RequestIdProxy(wrapped(*args, **kwargs))
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/v2/images.py", line 483, in _send_image_update_request
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     resp, body = self.http_client.patch(url, headers=hdrs,
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 407, in patch
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return self.request(url, 'PATCH', **kwargs)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 380, in request
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return self._handle_response(resp)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 120, in _handle_response
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     raise exc.from_response(resp, resp.content)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPForbidden: HTTP 403 Forbidden: It&#x27;s not allowed to add locations if locations are invisible.
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py", line 65, in wrapped
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     raise self.value
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/exception_wrapper.py", line 63, in wrapped
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 164, in decorated_function
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     raise self.value
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 155, in decorated_function
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/utils.py", line 1439, in decorated_function
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 211, in decorated_function
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     raise self.value
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 201, in decorated_function
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 231, in decorated_function
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     raise self.value
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 228, in decorated_function
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return function(self, context, image_id, instance,
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 4219, in snapshot_instance
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     self._snapshot_instance(context, image_id, instance,
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 4252, in _snapshot_instance
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     self.driver.snapshot(context, instance, image_id,
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 3116, in snapshot
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     with excutils.save_and_reraise_exception():
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     raise self.value
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 3045, in snapshot
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     self._image_api.update(context, image_id, metadata,
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 1243, in update
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return session.update(context, image_id, image_info, data=data,
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 693, in update
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     _reraise_translated_image_exception(image_id)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 1031, in _reraise_translated_image_exception
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     raise new_exc.with_traceback(exc_trace)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 691, in update
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     image = self._update_v2(context, sent_service_image_meta, data)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 705, in _update_v2
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     image = self._add_location(context, image_id, location)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 561, in _add_location
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return self._client.call(
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/image/glance.py", line 191, in call
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     result = getattr(controller, method)(*args, **kwargs)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/v2/images.py", line 503, in add_location
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     response = self._send_image_update_request(image_id, add_patch)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/utils.py", line 670, in inner
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return RequestIdProxy(wrapped(*args, **kwargs))
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/v2/images.py", line 483, in _send_image_update_request
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     resp, body = self.http_client.patch(url, headers=hdrs,
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystoneauth1/adapter.py", line 407, in patch
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return self.request(url, 'PATCH', **kwargs)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 380, in request
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     return self._handle_response(resp)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 120, in _handle_response
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server     raise exc.from_response(resp, resp.content)
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server nova.exception.ImageNotAuthorized: Not authorized for image 6d39ead7-e543-4ab6-b54c-78ca16421242.
2023-06-23 22:18:17.075 7 ERROR oslo_messaging.rpc.server