[cinder] cinder-backup volume stuck in creating
Folks, I have configured nova and cinder with ceph storage. VMs running on ceph storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad. My cinder.conf [DEFAULT] enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776 Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug? # openstack volume backup list --all +--------------------------------------+------+-------------+----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+------+-------------+----------+------+ | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 | +--------------------------------------+------+-------------+----------+------+
Hello Satish, Are you able to track the API request in c-api logs? Does the c-bak logs show that it is creating a backup at least or nothing at all? Regards, On Thu, Jan 26, 2023 at 7:22 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have configured nova and cinder with ceph storage. VMs running on ceph storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad.
My cinder.conf
[DEFAULT]
enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776
Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug?
# openstack volume backup list --all
+--------------------------------------+------+-------------+----------+------+ | ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+----------+------+ | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 |
+--------------------------------------+------+-------------+----------+------+
-- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>
Without the logs themselves it's really hard to say. One way to proceed would be to file a bug [1] and the team can work with you there. You could also enable debugging (debug = True), reproduce the failure, and upload the relevant logs there as well. [1]: https://bugs.launchpad.net/cinder/+filebug -- Jon On Thu, Jan 26, 2023 at 2:20 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have configured nova and cinder with ceph storage. VMs running on ceph storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad.
My cinder.conf
[DEFAULT]
enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776
Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug?
# openstack volume backup list --all
+--------------------------------------+------+-------------+----------+------+ | ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+----------+------+ | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 |
+--------------------------------------+------+-------------+----------+------+
Thank you Jon/Sofia, Biggest issue is even if I turn on debugging, it's not producing enough logs to see what is going on. See following output. https://paste.opendev.org/show/bh9OF9l2OrozrNMglv2Y/ On Fri, Jan 27, 2023 at 10:50 AM Jon Bernard <jobernar@redhat.com> wrote:
Without the logs themselves it's really hard to say. One way to proceed would be to file a bug [1] and the team can work with you there. You could also enable debugging (debug = True), reproduce the failure, and upload the relevant logs there as well.
[1]: https://bugs.launchpad.net/cinder/+filebug
-- Jon
On Thu, Jan 26, 2023 at 2:20 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have configured nova and cinder with ceph storage. VMs running on ceph storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad.
My cinder.conf
[DEFAULT]
enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776
Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug?
# openstack volume backup list --all
+--------------------------------------+------+-------------+----------+------+ | ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+----------+------+ | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 |
+--------------------------------------+------+-------------+----------+------+
On 27/01, Satish Patel wrote:
Thank you Jon/Sofia,
Biggest issue is even if I turn on debugging, it's not producing enough logs to see what is going on. See following output.
Hi, I don't see the cinder-volume logs, which are necessary to debug this issue, because we can see in the backup logs that it is doing an RPC call to the cinder-volume service "Call Volume Manager to get_backup_device". What is the value of the "host" field of the source volume? Because if it's anything other than "kolla-infra-1.example.com@rbd-1", then the problem is that the cinder-volume service for that backend is currently down. Cheers, Gorka.
On Fri, Jan 27, 2023 at 10:50 AM Jon Bernard <jobernar@redhat.com> wrote:
Without the logs themselves it's really hard to say. One way to proceed would be to file a bug [1] and the team can work with you there. You could also enable debugging (debug = True), reproduce the failure, and upload the relevant logs there as well.
[1]: https://bugs.launchpad.net/cinder/+filebug
-- Jon
On Thu, Jan 26, 2023 at 2:20 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have configured nova and cinder with ceph storage. VMs running on ceph storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad.
My cinder.conf
[DEFAULT]
enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776
Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug?
# openstack volume backup list --all
+--------------------------------------+------+-------------+----------+------+ | ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+----------+------+ | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 |
+--------------------------------------+------+-------------+----------+------+
Hello guys, I had the same problem, there is no more error log. I resolved by turning off cinder-api, cinder-scheduler and cinder-backup then removing these queues, such as cinder, cinder-scheduler and cinder-backup.. I enabled debug log on rabbitmq and cinder, however, I cannot see any related logs, 2023-07-13 20:57:22.903 666 INFO cinder.backup.manager [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d 183e47a6dd14489991db5c3cf4132a2a - - -] Create backup started, backup: 2cebd275-6fc3-47ae-8c8d-3337f9227057 volume: d8446fb4-032a-44a1-b47d-f922c4a0020e. 2023-07-13 20:57:22.919 666 INFO cinder.backup.manager [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d 183e47a6dd14489991db5c3cf4132a2a - - -] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2023-07-13T13:57:22Z,data_timestamp=2023-07-13T13:57:22Z,deleted=False,deleted_at=None,display_description='',display_name='ijoo',encryption_key_id=None,fail_reason=None,host='controller01',id=2cebd275-6fc3-47ae-8c8d-3337f9227057,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='183e47a6dd14489991db5c3cf4132a2a',restore_volume_id=None,service='cinder.backup.drivers.nfs.NFSBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='6371ebfe0fce499983b1f07e7c34bf6d',volume_id=d8446fb4-032a-44a1-b47d-f922c4a0020e) It is hard to reproduce this problems. Nguyen Huu Khoi On Wed, Feb 1, 2023 at 7:08 PM Gorka Eguileor <geguileo@redhat.com> wrote:
On 27/01, Satish Patel wrote:
Thank you Jon/Sofia,
Biggest issue is even if I turn on debugging, it's not producing enough logs to see what is going on. See following output.
Hi,
I don't see the cinder-volume logs, which are necessary to debug this issue, because we can see in the backup logs that it is doing an RPC call to the cinder-volume service "Call Volume Manager to get_backup_device".
What is the value of the "host" field of the source volume?
Because if it's anything other than "kolla-infra-1.example.com@rbd-1", then the problem is that the cinder-volume service for that backend is currently down.
Cheers, Gorka.
On Fri, Jan 27, 2023 at 10:50 AM Jon Bernard <jobernar@redhat.com>
wrote:
Without the logs themselves it's really hard to say. One way to
proceed
would be to file a bug [1] and the team can work with you there. You could also enable debugging (debug = True), reproduce the failure, and upload the relevant logs there as well.
[1]: https://bugs.launchpad.net/cinder/+filebug
-- Jon
On Thu, Jan 26, 2023 at 2:20 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have configured nova and cinder with ceph storage. VMs running on ceph storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad.
My cinder.conf
[DEFAULT]
enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776
Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug?
# openstack volume backup list --all
+--------------------------------------+------+-------------+----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+----------+------+
| bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 |
+--------------------------------------+------+-------------+----------+------+
On 15/07, Nguyễn Hữu Khôi wrote:
Hello guys, I had the same problem, there is no more error log.
I resolved by turning off cinder-api, cinder-scheduler and cinder-backup then removing these queues, such as cinder, cinder-scheduler and cinder-backup..
I enabled debug log on rabbitmq and cinder, however, I cannot see any related logs,
2023-07-13 20:57:22.903 666 INFO cinder.backup.manager [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d 183e47a6dd14489991db5c3cf4132a2a - - -] Create backup started, backup: 2cebd275-6fc3-47ae-8c8d-3337f9227057 volume: d8446fb4-032a-44a1-b47d-f922c4a0020e. 2023-07-13 20:57:22.919 666 INFO cinder.backup.manager [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d 183e47a6dd14489991db5c3cf4132a2a - - -] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2023-07-13T13:57:22Z,data_timestamp=2023-07-13T13:57:22Z,deleted=False,deleted_at=None,display_description='',display_name='ijoo',encryption_key_id=None,fail_reason=None,host='controller01',id=2cebd275-6fc3-47ae-8c8d-3337f9227057,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='183e47a6dd14489991db5c3cf4132a2a',restore_volume_id=None,service='cinder.backup.drivers.nfs.NFSBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='6371ebfe0fce499983b1f07e7c34bf6d',volume_id=d8446fb4-032a-44a1-b47d-f922c4a0020e)
It is hard to reproduce this problems.
Nguyen Huu Khoi
Hi, Next time it happens you may want to explore the status of the RabbitMQ service and queues. I remember seeing at least once a RabbitMQ service going crazy and not delivering messages in some of its queues. Cheers, Gorka.
On Wed, Feb 1, 2023 at 7:08 PM Gorka Eguileor <geguileo@redhat.com> wrote:
On 27/01, Satish Patel wrote:
Thank you Jon/Sofia,
Biggest issue is even if I turn on debugging, it's not producing enough logs to see what is going on. See following output.
Hi,
I don't see the cinder-volume logs, which are necessary to debug this issue, because we can see in the backup logs that it is doing an RPC call to the cinder-volume service "Call Volume Manager to get_backup_device".
What is the value of the "host" field of the source volume?
Because if it's anything other than "kolla-infra-1.example.com@rbd-1", then the problem is that the cinder-volume service for that backend is currently down.
Cheers, Gorka.
On Fri, Jan 27, 2023 at 10:50 AM Jon Bernard <jobernar@redhat.com>
wrote:
Without the logs themselves it's really hard to say. One way to
proceed
would be to file a bug [1] and the team can work with you there. You could also enable debugging (debug = True), reproduce the failure, and upload the relevant logs there as well.
[1]: https://bugs.launchpad.net/cinder/+filebug
-- Jon
On Thu, Jan 26, 2023 at 2:20 PM Satish Patel <satish.txt@gmail.com> wrote:
Folks,
I have configured nova and cinder with ceph storage. VMs running on ceph storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad.
My cinder.conf
[DEFAULT]
enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776
Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug?
# openstack volume backup list --all
+--------------------------------------+------+-------------+----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+----------+------+
| bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 |
+--------------------------------------+------+-------------+----------+------+
Hello. I will debug as you suggested next time. Thank you much. On Wed, Jul 26, 2023, 8:44 PM Gorka Eguileor <geguileo@redhat.com> wrote:
On 15/07, Nguyễn Hữu Khôi wrote:
Hello guys, I had the same problem, there is no more error log.
I resolved by turning off cinder-api, cinder-scheduler and cinder-backup then removing these queues, such as cinder, cinder-scheduler and cinder-backup..
I enabled debug log on rabbitmq and cinder, however, I cannot see any related logs,
2023-07-13 20:57:22.903 666 INFO cinder.backup.manager [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d 183e47a6dd14489991db5c3cf4132a2a - - -] Create backup started, backup: 2cebd275-6fc3-47ae-8c8d-3337f9227057 volume: d8446fb4-032a-44a1-b47d-f922c4a0020e. 2023-07-13 20:57:22.919 666 INFO cinder.backup.manager [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d 183e47a6dd14489991db5c3cf4132a2a - - -] Call Volume Manager to get_backup_device for
Backup(availability_zone=None,container=None,created_at=2023-07-13T13:57:22Z,data_timestamp=2023-07-13T13:57:22Z,deleted=False,deleted_at=None,display_description='',display_name='ijoo',encryption_key_id=None,fail_reason=None,host='controller01',id=2cebd275-6fc3-47ae-8c8d-3337f9227057,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='183e47a6dd14489991db5c3cf4132a2a',restore_volume_id=None,service='cinder.backup.drivers.nfs.NFSBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='6371ebfe0fce499983b1f07e7c34bf6d',volume_id=d8446fb4-032a-44a1-b47d-f922c4a0020e)
It is hard to reproduce this problems.
Nguyen Huu Khoi
Hi,
Next time it happens you may want to explore the status of the RabbitMQ service and queues.
I remember seeing at least once a RabbitMQ service going crazy and not delivering messages in some of its queues.
Cheers, Gorka.
On Wed, Feb 1, 2023 at 7:08 PM Gorka Eguileor <geguileo@redhat.com>
wrote:
On 27/01, Satish Patel wrote:
Thank you Jon/Sofia,
Biggest issue is even if I turn on debugging, it's not producing
enough
logs to see what is going on. See following output.
Hi,
I don't see the cinder-volume logs, which are necessary to debug this issue, because we can see in the backup logs that it is doing an RPC call to the cinder-volume service "Call Volume Manager to get_backup_device".
What is the value of the "host" field of the source volume?
Because if it's anything other than "kolla-infra-1.example.com@rbd-1", then the problem is that the cinder-volume service for that backend is currently down.
Cheers, Gorka.
On Fri, Jan 27, 2023 at 10:50 AM Jon Bernard <jobernar@redhat.com>
wrote:
Without the logs themselves it's really hard to say. One way to
proceed
would be to file a bug [1] and the team can work with you there. You could also enable debugging (debug = True), reproduce the failure, and upload the relevant logs there as well.
[1]: https://bugs.launchpad.net/cinder/+filebug
-- Jon
On Thu, Jan 26, 2023 at 2:20 PM Satish Patel <satish.txt@gmail.com
wrote:
Folks,
I have configured nova and cinder with ceph storage. VMs running
on ceph
storage but now when i am trying to create a backup of cinder volume its getting stuck on creating and doing nothing. Logs also do not give any indication of bad.
My cinder.conf
[DEFAULT]
enabled_backends = rbd-1 backup_driver = cinder.backup.drivers.ceph.CephBackupDriver backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true osapi_volume_listen = 10.73.0.181 osapi_volume_listen_port = 8776
Output of "openstack volume service list" showing cinder-backup service is up but when i create a backup it's getting stuck in this stage and no activity. I am not seeing anything getting transferred to the ceph backups pool also. Any clue? or method to debug?
# openstack volume backup list --all
+--------------------------------------+------+-------------+----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+----------+------+
| bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | creating | 20 |
+--------------------------------------+------+-------------+----------+------+
participants (5)
-
Gorka Eguileor
-
Jon Bernard
-
Nguyễn Hữu Khôi
-
Satish Patel
-
Sofia Enriquez