Open Stack

Thanks for the advice and feedback!

>>  I'm encountering an issue with live migrating between 2 Compute nodes with different CPU models. Host A has a Westmere CPU and host B has a Broadwell.
>That’s expected unless you specify the least common denominator with the cpu model, in this case Westmere, when you boot the instances.
 >
 >> I can migrate an instance from the Westmere host A to the Broadwell host B  Yes, you can move an instance from a host with less cpu features to more cpu features as long as you do not configure cpu-passthrough AFAIK.
>
>> but when I attempt to live migrate a VM from the Broadwell host B to
>> the older Westmere host I get the following error; Live Migration failure: unsupported configuration: guest and host CPU are not compatible:
>> Host CPU does not provide required features: fma, x2apic, movbe, tsc-deadline, xsave, osxsave, avx, ….
>That’s because the instances booted with those extra features enabled.
>(you can also see feature when grepping for the qemu process in the processlist, you will see the configured cpu in the qemu commandline) The Westmere CPU does not support avx/avx2 etc.


Therein lies the problem, the VM is running using only CPU features that are compatible with both CPUs. The CPU features libvirt says are missing are not in use by the VM. It was first booted on a host with a Westmere CPU, successfully live migrated to host with Broadwell CPU, but then refuses to migrate BACK to the Westmere host. The qemu-kvm process was launched with the following CPU parameters;

"-cpu Westmere,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+pclmuldq,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+arat,+pdpe1gb,+rdtscp"

And from "virsh dumpxml <instance-id>" the VM running with the following XML CPU config;

<cpu mode='host-model'>
    <model fallback='allow'>Westmere</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='1' threads='1'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='pclmuldq'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='rdtscp'/>
  </cpu>

Which is compatible with the Westmere host CPU. The Westmere hosts capabilities from "virsh capabilities";

  <host>
    <uuid>35333036-3936-4e43-3731-343030485837</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Westmere</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='2'/>
      <feature name='vme'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='pclmuldq'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='pcid'/>
      <feature name='dca'/>
      <feature name='arat'/>
      <feature name='pdpe1gb'/>
      <feature name='rdtscp'/>
      <feature name='invtsc'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>

The instance can also be migrated between 2 Westmere hosts. As I outline in another email, I suspect (perhaps incorrectly) as the VM has "cpu mode='host-model'" in its running config, libvirt may be comparing source and destination host CPUs and not guest and destination CPUs.

>>How can I avoid this error, without rebooting any VMs?
>You cannot do that for any instance which is already running with the extra cpu flags.
>Basically, you would need to remove cpu-features on the fly when live-migrating.
>That’s not possible.

>> Host B with the Broadwell CPU has the libvirt CPU masking enabled in its nova.conf;
>>     [libvirt]
>>   cpu_mode = custom
>>    cpu_model = Westmere
>>
>>    Host A with the Westmere CPU, never had the cpu_mode set in nova.conf but as its KVM it should default to the following;
>>    [libvirt]
>>    cpu_mode = host-model
>>

>According the docs :"host-model" - this causes libvirt to identify the named CPU model which most closely matches the host from the above list, and then request additional CPU flags to complete the match.
>If you want full live migration compatibility across different cpu generations I would recommend to set the cpu mode and model everywhere to the same values.

>> Interestingly once a host is booted on Host B and has the above CPU config, it can be migrated back to Host A without any error.
>I would expect that, as long as the destination CPU has all the cpu features where the qemu process was started with (in this case Westmere profile) it should work.
>
>There have been a couple of bugs in the past with the comparison of cpu features but I think what you are seeing is expected behavior.
>
>Note that there is quite a bit of change between Westmere and Broadwell and you might really want those extra cpu features.
>e.g. avx2 can seriously improve SSL offloading (200% improvement with the correct cypher suites)

This is defiantly in the longer term plan, but for the moment backwards compatibility is a requirement for us.

>Cheers,
>Robert van Leeuwen





Oisin O'Malley
Systems Engineer
Iocane Pty Ltd
763 South Road
Black Forest SA 5035

Office:+61 (8) 8413 1010
Fax:+61 (8) 8231 2050
Email:oisin.omalley at iocane.com.au
Web:www.iocane.com.au

Better for business