Open Stack

>  I'm encountering an issue with live migrating between 2 Compute nodes with different CPU models. Host A has a Westmere CPU and host B has a Broadwell.
That’s expected unless you specify the least common denominator with the cpu model, in this case Westmere, when you boot the instances.
  
 > I can migrate an instance from the Westmere host A to the Broadwell host B
 Yes, you can move an instance from a host with less cpu features to more cpu features as long as you do not configure cpu-passthrough AFAIK.

> but when I attempt to live migrate a VM from the Broadwell host B to the older Westmere host I get the following error;
> Live Migration failure: unsupported configuration: guest and host CPU are not compatible: 
> Host CPU does not provide required features: fma, x2apic, movbe, tsc-deadline, xsave, osxsave, avx, ….
That’s because the instances booted with those extra features enabled.
(you can also see feature when grepping for the qemu process in the processlist, you will see the configured cpu in the qemu commandline)
The Westmere CPU does not support avx/avx2 etc.
 
>How can I avoid this error, without rebooting any VMs?
You cannot do that for any instance which is already running with the extra cpu flags. 
Basically, you would need to remove cpu-features on the fly when live-migrating. 
That’s not possible.
    
> Host B with the Broadwell CPU has the libvirt CPU masking enabled in its nova.conf;
>     [libvirt]
>   cpu_mode = custom
>    cpu_model = Westmere
>  
>    Host A with the Westmere CPU, never had the cpu_mode set in nova.conf but as its KVM it should default to the following;
>    [libvirt]
>    cpu_mode = host-model
>

According the docs :"host-model" - this causes libvirt to identify the named CPU model which most closely matches the host from the above list, and then request additional CPU flags to complete the match.   
If you want full live migration compatibility across different cpu generations I would recommend to set the cpu mode and model everywhere to the same values.

> Interestingly once a host is booted on Host B and has the above CPU config, it can be migrated back to Host A without any error.
I would expect that, as long as the destination CPU has all the cpu features where the qemu process was started with (in this case Westmere profile) it should work.

There have been a couple of bugs in the past with the comparison of cpu features but I think what you are seeing is expected behavior.

Note that there is quite a bit of change between Westmere and Broadwell and you might really want those extra cpu features.
e.g. avx2 can seriously improve SSL offloading (200% improvement with the correct cypher suites)

Cheers,
Robert van Leeuwen