[nova] Live migration of a RAM intensive instance failed

Eugen Block eblock at nde.ag
Thu Sep 14 14:18:21 UTC 2023


Hi, could you share logs from the target compute node as well?

Zitat von Rafa <rafaa.haji3 at gmail.com>:

> Hello,
> I have a volume-backed instance with 16 vCPU and 64GB RAM. The instance
> uses 60GB RAM out of 64GB (used: 22GB; buff/cache 38GB). When I do a
> live migration of this instance, it fails without any timeouts.
> It copies almost all the RAM (within 150 - 250 seconds) to the target
> compute host without any problems according to the logs. Then the
> instance is paused to copy the rest of the RAM. Everything seems to be
> working correctly up to this point, but then the instance resumes and
> the following error message appears:
>
> Live Migration failure: operation failed: migration out job:
> unexpectedly failed: libvirt.libvirtError: operation failed:
> migration out job: unexpectedly failed
>
> Unfortunately, this error message does not say much. It doesn't look
> like it's due to any timeouts or short downtimes, but I still tested
> different (higher) values for the following configurations.
> Unfortunately without success.
> - live_migration_completion_timeout
> - live_migration_timeout_action: abort / force_complete (pause)
> - live_migration_downtime
> - live_migration_downtime_steps
> - live_migration_downtime_delay
> - live_migration_permit_auto_converge: True / False
>
> All other instances on the same source and destination hosts can be
> live migrated without any issues. This instance can also be
> successfully live migrated after a restart, as it is probably not yet
> heavily loaded. After a few hours, however, the live migration no
> longer works.
>
> Any ideas what the problem could be?
>
> Logs:
> - nova-compute.log from source compute host:
> https://paste.openstack.org/show/bJMFxnPKQBEVaPakud61/
> - i found this Traceback using journalctrl:
> https://paste.openstack.org/show/bIS4GFAd2RJ5fHVN9I8d/
> - there was also an error in /var/log/libvirt/qemu/:
> https://paste.openstack.org/show/bImT89IelDcXXBPSgTCO/
>
> Enviroment:
> - Libvirt: 8.0.0
> - QEMU: 4.2.1
> - Nova: 25.1.1
> - OpenStack: Yoga
> - Compute operating system: Ubuntu 20.04






More information about the openstack-discuss mailing list