[Openstack] [Ocata] config option show_multiple_locations
Hello list, I have a strange issue I'd like to report here, I'm not sure whether this could be a bug or a config issue on my side. The environment has developed from Liberty to Ocata over the last 3 years, backend for glance, cinder and nova is Ceph since Mitaka release. So according to [1] these two config options should be set to true.
show_multiple_locations = True show_image_direct_url = True
This setup has worked just fine, live snapshots of nova worked as expected. Last year the environment was upgraded to Ocata (successfully), and some time later I decided to clean up the configs, I set show_multiple_locations to false, also because glance reports:
Option "show_multiple_locations" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
Since this change the nova live snapshots stopped working, resulting in this stack trace: ---cut here--- [req-5bd2fef2-2155-4a89-b346-e20fb0b0d14a df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] Failed to snapshot image Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1626, in snapshot purge_props=False) File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 132, in update purge_props=purge_props) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 733, in update _reraise_translated_image_exception(image_id) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 1050, in _reraise_translated_image_exception six.reraise(type(new_exc), new_exc, exc_trace) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 731, in update image = self._update_v2(context, sent_service_image_meta, data) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 745, in _update_v2 image = self._add_location(context, image_id, location) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 630, in _add_location location, {}) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 168, in call result = getattr(controller, method)(*args, **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 340, in add_location response = self._send_image_update_request(image_id, add_patch) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 535, in inner return RequestIdProxy(wrapped(*args, **kwargs)) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 324, in _send_image_update_request data=json.dumps(patch_body)) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 294, in patch return self._request('PATCH', url, **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 277, in _request resp, body_iter = self._handle_response(resp) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response raise exc.from_response(resp, resp.content) ImageNotAuthorized: Not authorized for image e99b2dfd-db33-4475-a51f-af4b913a7041. INFO nova.compute.manager [req-5bd2fef2-2155-4a89-b346-e20fb0b0d14a df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] [instance: 751b3731-de0d-42cd-a105-b92e326294aa] Successfully reverted task state from image_uploading on failure for instance. ---cut here--- A couple of weeks passed until this problem occured (oviously nobody took snapshots), so I didn't immediately connect it to the config change, but when I followed the stack trace, I found this comment: ---cut here--- def _add_location(self, context, image_id, location): # 'show_multiple_locations' must be enabled in glance api conf file. [...] ---cut here--- I wouldn't expect this dependency if the option is marked as deprecated. Is this my misunderstanding or did I forget other configs that would prevent this behavior? Thank you for any information about this topic. Regards, Eugen [1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/#for-mitaka-only
Apoologies for top-posting, but the answer is that show_multiple_locations is deprecated, but its removal has been postponed, so you should continue to use it (but keep an eye on the Glance release notes). The original idea behind the deprecation was that because image locations are also governed by policies, it would simplify things to use only policies and eliminate the configuration option. In the meantime, an OSSN [0] was issued where the easiest way to mitigate the exploit is to set show_multiple_locations=False, so the deprecation period was extended [1]. Finally, closer inspection has revealed that show_multiple_locations cannot be removed without some major refactoring. There's a draft spec explaining the situation [2], but no one has been able to commit time to work on the issue (or even finish the spec). The Glance team would be happy to discuss this more with anyone interested in working on the issue. cheers, brian [0] https://wiki.openstack.org/wiki/OSSN/OSSN-0065 [1] https://docs.openstack.org/releasenotes/glance/ocata.html#relnotes-14-0-0-or... [2] https://review.openstack.org/#/c/528021/ On 11/29/18 6:01 AM, Eugen Block wrote:
Hello list,
I have a strange issue I'd like to report here, I'm not sure whether this could be a bug or a config issue on my side.
The environment has developed from Liberty to Ocata over the last 3 years, backend for glance, cinder and nova is Ceph since Mitaka release. So according to [1] these two config options should be set to true.
show_multiple_locations = True show_image_direct_url = True
This setup has worked just fine, live snapshots of nova worked as expected. Last year the environment was upgraded to Ocata (successfully), and some time later I decided to clean up the configs, I set show_multiple_locations to false, also because glance reports:
Option "show_multiple_locations" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
Since this change the nova live snapshots stopped working, resulting in this stack trace:
---cut here--- [req-5bd2fef2-2155-4a89-b346-e20fb0b0d14a df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] Failed to snapshot image Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1626, in snapshot purge_props=False) File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 132, in update purge_props=purge_props) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 733, in update _reraise_translated_image_exception(image_id) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 1050, in _reraise_translated_image_exception six.reraise(type(new_exc), new_exc, exc_trace) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 731, in update image = self._update_v2(context, sent_service_image_meta, data) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 745, in _update_v2 image = self._add_location(context, image_id, location) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 630, in _add_location location, {}) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 168, in call result = getattr(controller, method)(*args, **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 340, in add_location response = self._send_image_update_request(image_id, add_patch) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 535, in inner return RequestIdProxy(wrapped(*args, **kwargs)) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 324, in _send_image_update_request data=json.dumps(patch_body)) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 294, in patch return self._request('PATCH', url, **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 277, in _request resp, body_iter = self._handle_response(resp) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response raise exc.from_response(resp, resp.content) ImageNotAuthorized: Not authorized for image e99b2dfd-db33-4475-a51f-af4b913a7041.
INFO nova.compute.manager [req-5bd2fef2-2155-4a89-b346-e20fb0b0d14a df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] [instance: 751b3731-de0d-42cd-a105-b92e326294aa] Successfully reverted task state from image_uploading on failure for instance. ---cut here---
A couple of weeks passed until this problem occured (oviously nobody took snapshots), so I didn't immediately connect it to the config change, but when I followed the stack trace, I found this comment:
---cut here--- def _add_location(self, context, image_id, location): # 'show_multiple_locations' must be enabled in glance api conf file. [...] ---cut here---
I wouldn't expect this dependency if the option is marked as deprecated. Is this my misunderstanding or did I forget other configs that would prevent this behavior?
Thank you for any information about this topic.
Regards, Eugen
[1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/#for-mitaka-only
Thank you very much for the explanation, Brian. I'll keep the config option (it doesn't work without, so what choice is there?). Thank! Eugen Zitat von Brian Rosmaita <rosmaita.fossdev@gmail.com>:
Apoologies for top-posting, but the answer is that show_multiple_locations is deprecated, but its removal has been postponed, so you should continue to use it (but keep an eye on the Glance release notes).
The original idea behind the deprecation was that because image locations are also governed by policies, it would simplify things to use only policies and eliminate the configuration option. In the meantime, an OSSN [0] was issued where the easiest way to mitigate the exploit is to set show_multiple_locations=False, so the deprecation period was extended [1].
Finally, closer inspection has revealed that show_multiple_locations cannot be removed without some major refactoring. There's a draft spec explaining the situation [2], but no one has been able to commit time to work on the issue (or even finish the spec).
The Glance team would be happy to discuss this more with anyone interested in working on the issue.
cheers, brian
[0] https://wiki.openstack.org/wiki/OSSN/OSSN-0065 [1] https://docs.openstack.org/releasenotes/glance/ocata.html#relnotes-14-0-0-or... [2] https://review.openstack.org/#/c/528021/
On 11/29/18 6:01 AM, Eugen Block wrote:
Hello list,
I have a strange issue I'd like to report here, I'm not sure whether this could be a bug or a config issue on my side.
The environment has developed from Liberty to Ocata over the last 3 years, backend for glance, cinder and nova is Ceph since Mitaka release. So according to [1] these two config options should be set to true.
show_multiple_locations = True show_image_direct_url = True
This setup has worked just fine, live snapshots of nova worked as expected. Last year the environment was upgraded to Ocata (successfully), and some time later I decided to clean up the configs, I set show_multiple_locations to false, also because glance reports:
Option "show_multiple_locations" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
Since this change the nova live snapshots stopped working, resulting in this stack trace:
---cut here--- [req-5bd2fef2-2155-4a89-b346-e20fb0b0d14a df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] Failed to snapshot image Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1626, in snapshot purge_props=False) File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 132, in update purge_props=purge_props) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 733, in update _reraise_translated_image_exception(image_id) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 1050, in _reraise_translated_image_exception six.reraise(type(new_exc), new_exc, exc_trace) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 731, in update image = self._update_v2(context, sent_service_image_meta, data) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 745, in _update_v2 image = self._add_location(context, image_id, location) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 630, in _add_location location, {}) File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 168, in call result = getattr(controller, method)(*args, **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 340, in add_location response = self._send_image_update_request(image_id, add_patch) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 535, in inner return RequestIdProxy(wrapped(*args, **kwargs)) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 324, in _send_image_update_request data=json.dumps(patch_body)) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 294, in patch return self._request('PATCH', url, **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 277, in _request resp, body_iter = self._handle_response(resp) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response raise exc.from_response(resp, resp.content) ImageNotAuthorized: Not authorized for image e99b2dfd-db33-4475-a51f-af4b913a7041.
INFO nova.compute.manager [req-5bd2fef2-2155-4a89-b346-e20fb0b0d14a df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] [instance: 751b3731-de0d-42cd-a105-b92e326294aa] Successfully reverted task state from image_uploading on failure for instance. ---cut here---
A couple of weeks passed until this problem occured (oviously nobody took snapshots), so I didn't immediately connect it to the config change, but when I followed the stack trace, I found this comment:
---cut here--- def _add_location(self, context, image_id, location): # 'show_multiple_locations' must be enabled in glance api conf file. [...] ---cut here---
I wouldn't expect this dependency if the option is marked as deprecated. Is this my misunderstanding or did I forget other configs that would prevent this behavior?
Thank you for any information about this topic.
Regards, Eugen
[1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/#for-mitaka-only
participants (2)
-
Brian Rosmaita
-
Eugen Block