[nova] Live migration of a RAM intensive instance failed

Rafa rafaa.haji3 at gmail.com
Thu Sep 14 09:38:16 UTC 2023


Hello,
I have a volume-backed instance with 16 vCPU and 64GB RAM. The instance
uses 60GB RAM out of 64GB (used: 22GB; buff/cache 38GB). When I do a
live migration of this instance, it fails without any timeouts.
It copies almost all the RAM (within 150 - 250 seconds) to the target
compute host without any problems according to the logs. Then the
instance is paused to copy the rest of the RAM. Everything seems to be
working correctly up to this point, but then the instance resumes and
the following error message appears:

Live Migration failure: operation failed: migration out job:
unexpectedly failed: libvirt.libvirtError: operation failed:
migration out job: unexpectedly failed

Unfortunately, this error message does not say much. It doesn't look
like it's due to any timeouts or short downtimes, but I still tested
different (higher) values for the following configurations.
Unfortunately without success.
- live_migration_completion_timeout
- live_migration_timeout_action: abort / force_complete (pause)
- live_migration_downtime
- live_migration_downtime_steps
- live_migration_downtime_delay
- live_migration_permit_auto_converge: True / False

All other instances on the same source and destination hosts can be
live migrated without any issues. This instance can also be
successfully live migrated after a restart, as it is probably not yet
heavily loaded. After a few hours, however, the live migration no
longer works.

Any ideas what the problem could be?

Logs:
- nova-compute.log from source compute host:
https://paste.openstack.org/show/bJMFxnPKQBEVaPakud61/
- i found this Traceback using journalctrl:
https://paste.openstack.org/show/bIS4GFAd2RJ5fHVN9I8d/
- there was also an error in /var/log/libvirt/qemu/:
https://paste.openstack.org/show/bImT89IelDcXXBPSgTCO/

Enviroment:
- Libvirt: 8.0.0
- QEMU: 4.2.1
- Nova: 25.1.1
- OpenStack: Yoga
- Compute operating system: Ubuntu 20.04



More information about the openstack-discuss mailing list