Hi Noguchi,

In the event of live-migration failure, new record of table cinder.volume_attachement which corresponding to the destination host was not clean correctly so it has two attachments as your CLI result. My suggestion is clean up that stale record from cinder database manually.

-- Best Regards, Sang Chan – FPT Smart Cloud

From: Junya Noguchi (Fujitsu) noguchi.junya@fujitsu.com Date: Friday, 28 June 2024 at 19:19 To: 'openstack-discuss@lists.openstack.org' openstack-discuss@lists.openstack.org Cc: Hiroki Yokota (Fujitsu) yokota.hiroki@fujitsu.com Subject: [nova] volume information was duplicated when Live Migration failed To whom it may concern

Hi. I encountered an issue with OpenStack where volume information was duplicated when Live Migration failed. Could you please provide a solution to this issue?

[Issue]

After a Live Migration failure, when I checked the volume where the error occurred using the 'openstack volume list' command, the following was displayed:

+--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 9ba9734f-575e-41de-8bb4-60839388e0ad | | in-use | 10 | Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda |

I expect that the 'Attached to' display should only appear once as there is one volume connected, but it is duplicated for each failed attempt. The versions of OpenStack and nova are as follows:

OpenStack Zed

$ openstack --version openstack 6.0.0 $ nova --version nova CLI is deprecated and will be a removed in a future release 18.1.0

[Details]

The volume display is duplicated under the following conditions:

- When Live Migration of an instance fails

If the communication between the ports required for Live Migration is closed, Live Migration will fail when executed.

$ sudo netstat -tuln | grep 16509 $

The output of nova-compute.log at the time of Live Migration failure is as follows:

--- 2024-02-28 17:04:32.119 2785 ERROR nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Live Migration failure: unable to connect to server at 'XXXXX.com:16509': Connection refused: libvirt.libvirtError: unable to connect to server at 'XXXXX.com:16509': Connection refused 2024-02-28 17:04:32.119 2785 DEBUG nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Migration operation thread notification thread_finished /usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py:10516 2024-02-28 17:04:32.607 2785 DEBUG nova.virt.libvirt.migration [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] VM running on src, migration failed _log /usr/lib/python3.9/site-packages/nova/virt/libvirt/migration.py:432 2024-02-28 17:04:32.608 2785 DEBUG nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py:10330 2024-02-28 17:04:32.608 2785 ERROR nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Migration operation has aborted ---

After Live Migration, when I check the volume where the error occurred using the 'openstack volume list' command, the following is displayed:

+--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 9ba9734f-575e-41de-8bb4-60839388e0ad | | in-use | 10 | Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda |

'73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda' is displayed as many times as Live Migration failed. The duplicate display did not disappear even after Live Migration was successful.

Thanks,