Re: Cinder error while creating Volumes in Openstack
Hello Team,
I am getting the above error, while attaching volumes, My volume service list are up and running.
also my rabbitmq services are fine, any other suggestion we can check, we use qnap as storage
Regards Adivya Singh
On Wed, Jul 14, 2021 at 7:30 PM Adivya Singh adivya1.singh@gmail.com wrote:
Hi Eugen,
Please find the error logs which i am getting, particular during attaching a instance to a Instance
Jul 14 13:54:17 c1w-cinder-volumes-container-aa023a91 cinder-volume[38100]: 2021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server [req-7ec417bc-0692-4766-8513-86a979c23e07 4441034aeea2551710631bc8fe381b1094c260e64501c736bfa51987aaefd7cc 4e1398294816433bab5f3cebb2210788 - f7834cb0083b4f8f81184b6595b46b34 f7834cb0083b4f8f81184b6595b46b34] Exception during message handling: ParseError: no element found: line 20, column 0#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server Traceback (most recent call last):#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/manager.py", line 4420, in attachment_update#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server connector)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/manager.py", line 4348, in _connection_create#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server volume, connector)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 853, in create_export#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server self._get_portal_info(volume, connector, lun_slot_id, lun_owner))#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 706, in _get_portal_info#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server ret = self.api_executor.get_iscsi_portal_info()#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 1109, in inner_connection_checker#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 1520, in get_iscsi_portal_info#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server root = ET.fromstring(res_details['data'])#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/defusedxml/common.py", line 118, in fromstring#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return parser.close()#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1671, in close#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server self._raiseerror(v)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server raise err#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server ParseError: no element found: line 20, column 0#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server
Regards Adivya Singh 9590986094
On Wed, Jul 14, 2021 at 1:29 PM Eugen Block eblock@nde.ag wrote:
Hi,
those are only INFO logs, they don't show anything related to errors you're experiencing.
Zitat von Adivya Singh adivya1.singh@gmail.com:
Hi Team,
any input on this, Logs are attached
Regards Adivya Singh
---------- Forwarded message --------- From: Adivya Singh adivya1.singh@gmail.com Date: Mon, Jul 12, 2021 at 9:58 PM Subject: Re: Cinder error while creating Volumes in Openstack To: Brian Rosmaita rosmaita.fossdev@gmail.com
hi Brian,
Thanks for your response, I am attaching the logs hereby.
Earlier we were able to create the cinder volumes and delete , but
unable
to attach, but now even create and delete not working
Regards Adivya Singh
On Mon, Jul 12, 2021 at 9:31 PM Brian Rosmaita <
rosmaita.fossdev@gmail.com>
wrote:
On 7/12/21 11:20 AM, Adivya Singh wrote:
Hello Team,
i am seeing the below error, while creating volumes in cinder log,
Any
solution or fix for these error, as of now i am not able to create volumes, neither i am unable to delete, it just sits in hung state
DISTRIB_RELEASE="18.1.9" DISTRIB_CODENAME="Rocky" DISTRIB_DESCRIPTION="OpenStack-Ansible"
Hello Adivya,
I think you forgot to include the error? If your log extracts are
long,
you can post them in http://paste.openstack.org/ and send the links to this mailing list. (Don't forget to scrub any sensitive info first.)
Regards Adivya Singh
Have you checked the storage backend for errors? I’m not familiar with qnap, but if rabbitmq is healthy I would expect something in the qnap backend.
Zitat von Adivya Singh adivya1.singh@gmail.com:
Hello Team,
I am getting the above error, while attaching volumes, My volume service list are up and running.
also my rabbitmq services are fine, any other suggestion we can check, we use qnap as storage
Regards Adivya Singh
On Wed, Jul 14, 2021 at 7:30 PM Adivya Singh adivya1.singh@gmail.com wrote:
Hi Eugen,
Please find the error logs which i am getting, particular during attaching a instance to a Instance
Jul 14 13:54:17 c1w-cinder-volumes-container-aa023a91 cinder-volume[38100]: 2021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server [req-7ec417bc-0692-4766-8513-86a979c23e07 4441034aeea2551710631bc8fe381b1094c260e64501c736bfa51987aaefd7cc 4e1398294816433bab5f3cebb2210788 - f7834cb0083b4f8f81184b6595b46b34 f7834cb0083b4f8f81184b6595b46b34] Exception during message handling: ParseError: no element found: line 20, column 0#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server Traceback (most recent call last):#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/manager.py", line 4420, in attachment_update#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server connector)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/manager.py", line 4348, in _connection_create#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server volume, connector)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 853, in create_export#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server self._get_portal_info(volume, connector, lun_slot_id, lun_owner))#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 706, in _get_portal_info#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server ret = self.api_executor.get_iscsi_portal_info()#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 1109, in inner_connection_checker#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 1520, in get_iscsi_portal_info#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server root = ET.fromstring(res_details['data'])#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/defusedxml/common.py", line 118, in fromstring#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return parser.close()#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1671, in close#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server self._raiseerror(v)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server raise err#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server ParseError: no element found: line 20, column 0#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server
Regards Adivya Singh 9590986094
On Wed, Jul 14, 2021 at 1:29 PM Eugen Block eblock@nde.ag wrote:
Hi,
those are only INFO logs, they don't show anything related to errors you're experiencing.
Zitat von Adivya Singh adivya1.singh@gmail.com:
Hi Team,
any input on this, Logs are attached
Regards Adivya Singh
---------- Forwarded message --------- From: Adivya Singh adivya1.singh@gmail.com Date: Mon, Jul 12, 2021 at 9:58 PM Subject: Re: Cinder error while creating Volumes in Openstack To: Brian Rosmaita rosmaita.fossdev@gmail.com
hi Brian,
Thanks for your response, I am attaching the logs hereby.
Earlier we were able to create the cinder volumes and delete , but
unable
to attach, but now even create and delete not working
Regards Adivya Singh
On Mon, Jul 12, 2021 at 9:31 PM Brian Rosmaita <
rosmaita.fossdev@gmail.com>
wrote:
On 7/12/21 11:20 AM, Adivya Singh wrote:
Hello Team,
i am seeing the below error, while creating volumes in cinder log,
Any
solution or fix for these error, as of now i am not able to create volumes, neither i am unable to delete, it just sits in hung state
DISTRIB_RELEASE="18.1.9" DISTRIB_CODENAME="Rocky" DISTRIB_DESCRIPTION="OpenStack-Ansible"
Hello Adivya,
I think you forgot to include the error? If your log extracts are
long,
you can post them in http://paste.openstack.org/ and send the links to this mailing list. (Don't forget to scrub any sensitive info first.)
Regards Adivya Singh
On Wed, Jul 14, 2021 at 06:20:15PM +0000, Eugen Block wrote:
Have you checked the storage backend for errors? I’m not familiar with qnap, but if rabbitmq is healthy I would expect something in the qnap backend.
Zitat von Adivya Singh adivya1.singh@gmail.com:
Hello Team,
I am getting the above error, while attaching volumes, My volume service list are up and running.
also my rabbitmq services are fine, any other suggestion we can check, we use qnap as storage
Regards Adivya Singh
The QNAP driver has not seen any vendor support for quite awhile, so it's very possible the driver is broken with new version of OpenStack or newer versions of the QNAP storage.
I have submitted a patch to mark it as unsupported so the state of the driver is clear:
https://review.opendev.org/c/openstack/cinder/+/800830
Sorry, I know that doesn't help you now, but it might be a good indication that it would be worth looking in to alternative storage to use.
Sean
On 14/07, Adivya Singh wrote:
Hello Team,
I am getting the above error, while attaching volumes, My volume service list are up and running.
also my rabbitmq services are fine, any other suggestion we can check, we use qnap as storage
Regards Adivya Singh
Hi,
Looks like the XML returned by the QNAP system is missing the 'data' element.
What is the QTS version of your QNAP system?
I believe your QNAP firmware version is not supported by the Rocky driver.
If it's 4.4, then it won't work with Rocky, since support was added in a later release [1].
If that's not it, I would recommend enabling debug logging in the cinder-volume service to see if there is additional information.
Cheers, Gorka.
[1]: https://review.opendev.org/c/openstack/cinder/+/632953
On Wed, Jul 14, 2021 at 7:30 PM Adivya Singh adivya1.singh@gmail.com wrote:
Hi Eugen,
Please find the error logs which i am getting, particular during attaching a instance to a Instance
Jul 14 13:54:17 c1w-cinder-volumes-container-aa023a91 cinder-volume[38100]: 2021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server [req-7ec417bc-0692-4766-8513-86a979c23e07 4441034aeea2551710631bc8fe381b1094c260e64501c736bfa51987aaefd7cc 4e1398294816433bab5f3cebb2210788 - f7834cb0083b4f8f81184b6595b46b34 f7834cb0083b4f8f81184b6595b46b34] Exception during message handling: ParseError: no element found: line 20, column 0#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server Traceback (most recent call last):#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/manager.py", line 4420, in attachment_update#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server connector)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/manager.py", line 4348, in _connection_create#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server volume, connector)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 853, in create_export#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server self._get_portal_info(volume, connector, lun_slot_id, lun_owner))#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 706, in _get_portal_info#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server ret = self.api_executor.get_iscsi_portal_info()#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 1109, in inner_connection_checker#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/cinder/volume/drivers/qnap.py", line 1520, in get_iscsi_portal_info#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server root = ET.fromstring(res_details['data'])#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/openstack/venvs/cinder-18.1.9/lib/python2.7/site-packages/defusedxml/common.py", line 118, in fromstring#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server return parser.close()#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1671, in close#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server self._raiseerror(v)#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server raise err#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server ParseError: no element found: line 20, column 0#0122021-07-14 13:54:17.246 38100 ERROR oslo_messaging.rpc.server
Regards Adivya Singh 9590986094
On Wed, Jul 14, 2021 at 1:29 PM Eugen Block eblock@nde.ag wrote:
Hi,
those are only INFO logs, they don't show anything related to errors you're experiencing.
Zitat von Adivya Singh adivya1.singh@gmail.com:
Hi Team,
any input on this, Logs are attached
Regards Adivya Singh
---------- Forwarded message --------- From: Adivya Singh adivya1.singh@gmail.com Date: Mon, Jul 12, 2021 at 9:58 PM Subject: Re: Cinder error while creating Volumes in Openstack To: Brian Rosmaita rosmaita.fossdev@gmail.com
hi Brian,
Thanks for your response, I am attaching the logs hereby.
Earlier we were able to create the cinder volumes and delete , but
unable
to attach, but now even create and delete not working
Regards Adivya Singh
On Mon, Jul 12, 2021 at 9:31 PM Brian Rosmaita <
rosmaita.fossdev@gmail.com>
wrote:
On 7/12/21 11:20 AM, Adivya Singh wrote:
Hello Team,
i am seeing the below error, while creating volumes in cinder log,
Any
solution or fix for these error, as of now i am not able to create volumes, neither i am unable to delete, it just sits in hung state
DISTRIB_RELEASE="18.1.9" DISTRIB_CODENAME="Rocky" DISTRIB_DESCRIPTION="OpenStack-Ansible"
Hello Adivya,
I think you forgot to include the error? If your log extracts are
long,
you can post them in http://paste.openstack.org/ and send the links to this mailing list. (Don't forget to scrub any sensitive info first.)
Regards Adivya Singh
participants (4)
-
Adivya Singh
-
Eugen Block
-
Gorka Eguileor
-
Sean McGinnis