Nova live migration fails - InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.
Having issues getting live migrations to functions between two compute nodes. One is an Opteron G4 and the other is an Opteron G5. I can migrate from the G4 to the G5 just fine. When I try to migrate an instance from the G5 node to the G4 node I get an error message: InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. This is despite having the following setting in my /etc/nova/nova.conf on the G5 machine. [libvirt] cpu_mode = custom cpu_model = Opteron_G4 Is this not the setting I'm looking for? What can I do to make this work? Torin Woltjer Grand Dial Communications - A ZK Tech Inc. Company 616.776.1066 ext. 2006 www.granddial.com
On 12/13/2018 4:36 PM, Torin Woltjer wrote:
Having issues getting live migrations to functions between two compute nodes. One is an Opteron G4 and the other is an Opteron G5. I can migrate from the G4 to the G5 just fine. When I try to migrate an instance from the G5 node to the G4 node I get an error message: InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. This is despite having the following setting in my /etc/nova/nova.conf on the G5 machine.
[libvirt] cpu_mode = custom cpu_model = Opteron_G4
Is this not the setting I'm looking for? What can I do to make this work?
This summit talk from Kashyap might be helpful: https://www.openstack.org/videos/summits/berlin-2018/effective-virtual-cpu-c... -- Thanks, Matt
On Thu, Dec 13, 2018 at 5:37 PM Torin Woltjer <torin.woltjer@granddial.com> wrote:
Having issues getting live migrations to functions between two compute nodes. One is an Opteron G4 and the other is an Opteron G5. I can migrate from the G4 to the G5 just fine. When I try to migrate an instance from the G5 node to the G4 node I get an error message: InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. This is despite having the following setting in my /etc/nova/nova.conf on the G5 machine.
[libvirt] cpu_mode = custom cpu_model = Opteron_G4
Is this not the setting I'm looking for? What can I do to make this work?
Generally, yes this should do the trick. So long as the lowest common set of features is used, then you should be able to live migrate. You can do a diff on the feature set of each host CPU with: virsh capabilities Did you create the instances you are trying to migrate after you applied the setting? If not, do a cold migrate and then try the live one again. Did you apply the setting on both computes? I know this maybe is obvious, but I've seen people put it on the controllers and not the computes and wonder why it wasn't working. Doesn't hurt to ask :).
Torin Woltjer
Grand Dial Communications - A ZK Tech Inc. Company
616.776.1066 ext. 2006 www.granddial.com
-Erik
On Thu, Dec 13, 2018 at 06:48:10PM -0500, Erik McCormick wrote: :On Thu, Dec 13, 2018 at 5:37 PM Torin Woltjer :<torin.woltjer@granddial.com> wrote: :> :> Having issues getting live migrations to functions between two compute nodes. :> One is an Opteron G4 and the other is an Opteron G5. I can migrate from the G4 to the G5 just fine. When I try to migrate an instance from the G5 node to the G4 node I get an error message: :> InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. :> This is despite having the following setting in my /etc/nova/nova.conf on the G5 machine. :> :> [libvirt] :> cpu_mode = custom :> cpu_model = Opteron_G4 :> :> Is this not the setting I'm looking for? What can I do to make this work? :> :Did you create the instances you are trying to migrate after you :applied the setting? If not, do a cold migrate and then try the live :one again. My bets are on this one, it's the sort of thing I do anyway. An `openstack server reboot --hard` should also recreate the VM inplace with new config settings I think. you can double check that the instance you are trying to move has the correct CPU setting by looking at the libvirt XML, either in: /var/lib/nova/instances/<instance uuid>/libvirt.xml or dumping libvirt's info on running VM: virsh dumpxml <virt instance-name> what you don't want to see is: <cpu mode="host-passthrough" match="exact"> This is what I use and just live with somewhat complex mappings of what will move where because my user want all thw feature they can get. So I'm not 100% sure what you wnat to see here, but I know it's not that. -Jon
Let me chip in on this as we are experiencing a similar issue: We have started encountering this behaviour following a ubuntu trusty to xenial upgrade (we are still running Mitaka). It affects VMs that previously were successfully migrated cross architecture (xeon v2 -> skylake and vice versa). We are yet unable to pinpoint the exact cause because the issue is difficult to reproduce: VMs started on the affected node migrate freely and their xml is *identical* (except for UUIDs and such) to the one of the VMs that refuse to migrate. As a side note I think the key is the: "InvalidCPUInfo: Unacceptable CPU info". Normally, infact, the error would also include which are the problematic flags and despite the exception coming always from driver.py the message is different. If anyone has any suggestions other than reboot the affected VMs I'll gladly give them a try. Best Mattia On 12/14/18 3:56 PM, Jonathan D. Proulx wrote:
On Thu, Dec 13, 2018 at 06:48:10PM -0500, Erik McCormick wrote: :On Thu, Dec 13, 2018 at 5:37 PM Torin Woltjer :<torin.woltjer@granddial.com> wrote: :> :> Having issues getting live migrations to functions between two compute nodes. :> One is an Opteron G4 and the other is an Opteron G5. I can migrate from the G4 to the G5 just fine. When I try to migrate an instance from the G5 node to the G4 node I get an error message: :> InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. :> This is despite having the following setting in my /etc/nova/nova.conf on the G5 machine. :> :> [libvirt] :> cpu_mode = custom :> cpu_model = Opteron_G4 :> :> Is this not the setting I'm looking for? What can I do to make this work? :>
:Did you create the instances you are trying to migrate after you :applied the setting? If not, do a cold migrate and then try the live :one again.
My bets are on this one, it's the sort of thing I do anyway. An `openstack server reboot --hard` should also recreate the VM inplace with new config settings I think.
you can double check that the instance you are trying to move has the correct CPU setting by looking at the libvirt XML, either in:
/var/lib/nova/instances/<instance uuid>/libvirt.xml
or dumping libvirt's info on running VM:
virsh dumpxml <virt instance-name>
what you don't want to see is:
<cpu mode="host-passthrough" match="exact">
This is what I use and just live with somewhat complex mappings of what will move where because my user want all thw feature they can get. So I'm not 100% sure what you wnat to see here, but I know it's not that.
-Jon
-- Mattia Belluco S3IT Services and Support for Science IT Office Y11 F 52 University of Zürich Winterthurerstrasse 190, CH-8057 Zürich (Switzerland) Tel: +41 44 635 42 22
participants (5)
-
Erik McCormick
-
Jonathan D. Proulx
-
Matt Riedemann
-
Mattia Belluco
-
Torin Woltjer