[glance] Replaced parameters "default_store" and "glance_store" breaks cooperation between cinder and glance in rocky release
Hello, I am writing to you, because we faced an issue with broken cooperation between cinder and glance. In our logs(glance-api) we observed such warning message: 2021-02-22 15:30:02.894 756 WARNING oslo_config.cfg [-] Deprecated: Option "stores" from group "glance_store" is deprecated for removal ( This option is deprecated against new config option ``enabled_backends`` which helps to configure multiple backend stores of different schemes. This option is scheduled for removal in the Stein development cycle. ). Its value may be silently ignored in the future. 2021-02-22 15:30:02.896 756 WARNING oslo_config.cfg [-] Deprecated: Option "default_store" from group "glance_store" is deprecated for removal ( This option is deprecated against new config option ``default_backend`` which acts similar to ``default_store`` config option. This option is scheduled for removal in the Stein development cycle. ). Its value may be silently ignored in the future. And we simply wanted to solve this problem, by changing option glance_store to enabled_backends and default_store to default_backend: @@ -2079,7 +2079,7 @@ pool_timeout = 60 # cycle. #stores = file,http -stores = glance.store.rbd.Store +enabled_backends = glance.store.rbd.Store # DEPRECATED: # The default scheme to use for storing images. @@ -2134,7 +2134,7 @@ stores = glance.store.rbd.Store # cycle. #default_store = file -default_store = rbd +default_backend = rbd But then we realized that we're not able to create volumes from new images, because we hit an issue: 2021-02-22 22:45:08.805 5910 ERROR cinder.scheduler.filter_scheduler [req-fe8e8aa2-dd8a-470b-9d70-dbdc757b48d9 54df39e1b74149bcb8cf9ad3eb957d48 afdfb965515f4724a21b6823764ec36d - default default] Error scheduling 9de37fe8-4f9e-4c29-8cfa-c1a71e8c68b7 from last vol-service: cinder-05@huawei_backend#StoragePool001 : [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task\n result = task.execute(**arguments)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 1059, in execute\n **volume_spec)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 818, in _wrapper\n return r.call(f, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n', u' File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n', u' File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 959, in _create_from_image\n image_service)\n', u' File "<decorator-gen-238>", line 2, in _prepare_image_cache_entry\n', u' File "/usr/lib/python2.7/dist-packages/cinder/coordination.py", line 151, in _synchronized\n return f(*a, **k)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 757, in _prepare_image_cache_entry\n update_cache=True)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 831, in _create_from_image_cache_or_download\n backend_name) as tmp_image:\n', u' File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__\n return self.gen.next()\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 832, in fetch\n fetch_verify_image(context, image_service, image_id, tmp)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 451, in fetch_verify_image\n None, None)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 377, in fetch\n tpool.Proxy(image_file))\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 358, in download\n _reraise_translated_image_exception(image_id)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 602, in _reraise_translated_image_exception\n six.reraise(type(new_exc), new_exc, exc_trace)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 356, in download\n image_chunks = self._client.call(context, \'data\', image_id)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 225, in call\n return getattr(controller, method)(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 545, in inner\n return RequestIdProxy(wrapped(*args, **kwargs))\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 208, in data\n resp, body = self.http_client.get(url)\n', u' File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 328, in get\n return self.request(url, \'GET\', **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 349, in request\n return self._handle_response(resp)\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 98, in _handle_response\n raise exc.from_response(resp, resp.content)\n', u'HTTPInternalServerError: HTTPInternalServerError (HTTP 500)\n'] Which is visible in cinder-scheduler log file. When we reverted a change, everything started to operate normally. I'm just wondering how we can skip this warning message and makes cinder and glance to cooperate correctly, especially that we're planning to make an upgrade from Rocky to Stein soon and this option is planned to be removed in the Stein release. Thanks in advance for your help here. Pozdrawiam, Jan Wasilewski Administrator systemów i sieci CI ATM Tel: +48 691 493 327 jan.wasilewski@atman.pl<mailto:jan.wasilewski@atman.pl>, www.atman.eu<http://www.atman.eu/> Informacja o zasadach przetwarzania danych osobowych przez ATM S.A. znajduje się pod tym linkiem<http://www.atman.pl/Informacja-o-przetwarzaniu-danych-osobowych-cabout-pol-113.html>.
Sorry to top post, but I think your Glance configuration may be incorrect. The 'enabled_backends' value is a comma separated string of keys:values of name:store_type, and then you use the name to create a config group for that store's options. So if you name your store 'reliable', you'll want something like: [DEFAULT] enabled_backends = reliable:rbd [glance_store] default_backend = reliable [reliable] store_description = "Reliable backend for storing your images" # your current rbd options should also go here You'll also need to add a [os_staging] section to the config file to handle image import; see this doc for information about all of the above: https://docs.openstack.org/glance/rocky/admin/multistores.html I think Cinder is getting a 500 from Glance when it requests the image data, so hopefully this Glance config change will also fix your cinder issue. cheers, brian On 2/23/21 5:02 AM, Jan Wasilewski wrote:
Hello,
I am writing to you, because we faced an issue with broken cooperation between cinder and glance. In our logs(glance-api) we observed such warning message:
2021-02-22 15:30:02.894 756 WARNING oslo_config.cfg [-] Deprecated: Option "stores" from group "glance_store" is deprecated for removal (
This option is deprecated against new config option
``enabled_backends`` which helps to configure multiple backend stores
of different schemes.
This option is scheduled for removal in the Stein development
cycle.
). Its value may be silently ignored in the future.
2021-02-22 15:30:02.896 756 WARNING oslo_config.cfg [-] Deprecated: Option "default_store" from group "glance_store" is deprecated for removal (
This option is deprecated against new config option
``default_backend`` which acts similar to ``default_store`` config
option.
This option is scheduled for removal in the Stein development
cycle.
). Its value may be silently ignored in the future.
And we simply wanted to solve this problem, by changing option glance_store to enabled_backends and default_store to default_backend:
@@ -2079,7 +2079,7 @@pool_timeout = 60
# cycle.
#stores = file,http
-stores = glance.store.rbd.Store
+enabled_backends = glance.store.rbd.Store
# DEPRECATED:
# The default scheme to use for storing images.
@@ -2134,7 +2134,7 @@stores = glance.store.rbd.Store
# cycle.
#default_store = file
-default_store = rbd
+default_backend = rbd
But then we realized that we’re not able to create volumes from new images, because we hit an issue:
2021-02-22 22:45:08.805 5910 ERROR cinder.scheduler.filter_scheduler [req-fe8e8aa2-dd8a-470b-9d70-dbdc757b48d9 54df39e1b74149bcb8cf9ad3eb957d48 afdfb965515f4724a21b6823764ec36d - default default] Error scheduling 9de37fe8-4f9e-4c29-8cfa-c1a71e8c68b7 from last vol-service: cinder-05@huawei_backend#StoragePool001 : [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task\n result = task.execute(**arguments)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 1059, in execute\n **volume_spec)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 818, in _wrapper\n return r.call(f, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n', u' File "/usr/lib/python2.7/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n', u' File "/usr/lib/python2.7/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 959, in _create_from_image\n image_service)\n', u' File "<decorator-gen-238>", line 2, in _prepare_image_cache_entry\n', u' File "/usr/lib/python2.7/dist-packages/cinder/coordination.py", line 151, in _synchronized\n return f(*a, **k)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 757, in _prepare_image_cache_entry\n update_cache=True)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 831, in _create_from_image_cache_or_download\n backend_name) as tmp_image:\n', u' File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__\n return self.gen.next()\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 832, in fetch\n fetch_verify_image(context, image_service, image_id, tmp)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 451, in fetch_verify_image\n None, None)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 377, in fetch\n tpool.Proxy(image_file))\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 358, in download\n _reraise_translated_image_exception(image_id)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 602, in _reraise_translated_image_exception\n six.reraise(type(new_exc), new_exc, exc_trace)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 356, in download\n image_chunks = self._client.call(context, \'data\', image_id)\n', u' File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 225, in call\n return getattr(controller, method)(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 545, in inner\n return RequestIdProxy(wrapped(*args, **kwargs))\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 208, in data\n resp, body = self.http_client.get(url)\n', u' File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 328, in get\n return self.request(url, \'GET\', **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 349, in request\n return self._handle_response(resp)\n', u' File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 98, in _handle_response\n raise exc.from_response(resp, resp.content)\n', u'HTTPInternalServerError: HTTPInternalServerError (HTTP 500)\n']
Which is visible in cinder-scheduler log file. When we reverted a change, everything started to operate normally. I’m just wondering how we can skip this warning message and makes cinder and glance to cooperate correctly, especially that we’re planning to make an upgrade from Rocky to Stein soon and this option is planned to be removed in the Stein release. Thanks in advance for your help here.
Pozdrawiam, Jan Wasilewski Administrator systemów i sieci CI ATM Tel: +48 691 493 327 _jan.wasilewski@atman.pl <mailto:jan.wasilewski@atman.pl>, _www.atman.eu <http://www.atman.eu/>
Informacja o zasadach przetwarzania danych osobowych przez ATM S.A. znajduje się pod tym linkiem <http://www.atman.pl/Informacja-o-przetwarzaniu-danych-osobowych-cabout-pol-113.html>.
participants (2)
-
Brian Rosmaita
-
Jan Wasilewski