<div dir="ltr"><div class="gmail_default" style="font-family:'courier new',monospace">Ok, so here is what I have done. Using "virsh capabilities" on the old CPUs I got:</div><div class="gmail_default"><div class="gmail_default">
<font face="courier new, monospace"> <cpu></font></div><div class="gmail_default"><font face="courier new, monospace"> <arch>x86_64</arch></font></div><div class="gmail_default"><font face="courier new, monospace"> <model>Opteron_G3</model></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <vendor>AMD</vendor></font></div><div class="gmail_default"><font face="courier new, monospace"> <topology sockets='1' cores='4' threads='1'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='nodeid_msr'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='wdt'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='skinit'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='ibs'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='osvw'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='3dnowprefetch'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='cr8legacy'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='extapic'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='cmp_legacy'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='3dnow'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='3dnowext'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='pdpe1gb'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='fxsr_opt'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='mmxext'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='ht'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='vme'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> </cpu></font></div><div class="gmail_default"><font face="courier new, monospace"><br></font></div><div class="gmail_default"><span style="font-family:'courier new',monospace">Using "virsh capabilities" on the new CPUs I got:</span><font face="courier new, monospace"><br>
</font></div><div class="gmail_default"><div class="gmail_default"><font face="courier new, monospace"> <cpu></font></div><div class="gmail_default"><font face="courier new, monospace"> <arch>x86_64</arch></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <model>Opteron_G5</model></font></div><div class="gmail_default"><font face="courier new, monospace"> <vendor>AMD</vendor></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <topology sockets='1' cores='64' threads='1'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='bmi1'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='perfctr_nb'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='perfctr_core'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='topoext'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='nodeid_msr'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='tce'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='lwp'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='wdt'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='skinit'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='ibs'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='osvw'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='cr8legacy'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='extapic'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='cmp_legacy'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='fxsr_opt'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='mmxext'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='osxsave'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='monitor'/></font></div><div class="gmail_default"><font face="courier new, monospace"> <feature name='ht'/></font></div>
<div class="gmail_default"><font face="courier new, monospace"> <feature name='vme'/></font></div><div class="gmail_default"><font face="courier new, monospace"> </cpu></font></div><div class="gmail_default">
<font face="courier new, monospace"><br></font></div><div class="gmail_default"><font face="courier new, monospace">So I came up with this CPU as a common feature set:</font></div><div class="gmail_default"><font face="courier new, monospace"><div class="gmail_default">
<cpu></div><div class="gmail_default"> <arch>x86_64</arch></div><div class="gmail_default"> <model>Opteron_G3</model></div><div class="gmail_default"> <vendor>AMD</vendor></div>
<div class="gmail_default"> <topology sockets='1' cores='4' threads='1'/></div><div class="gmail_default"> <feature name='nodeid_msr'/></div><div class="gmail_default">
<feature name='wdt'/></div><div class="gmail_default"> <feature name='skinit'/></div><div class="gmail_default"> <feature name='ibs'/></div><div class="gmail_default">
<feature name='osvw'/></div><div class="gmail_default"> <feature name='3dnowprefetch'/></div><div class="gmail_default"> <feature name='cr8legacy'/></div><div class="gmail_default">
<feature name='extapic'/></div><div class="gmail_default"> <feature name='cmp_legacy'/></div><div class="gmail_default"> <feature name='pdpe1gb'/></div><div class="gmail_default">
<feature name='fxsr_opt'/></div><div class="gmail_default"> <feature name='mmxext'/></div><div class="gmail_default"> <feature name='ht'/></div><div class="gmail_default">
<feature name='vme'/></div><div class="gmail_default"> </cpu></div><div class="gmail_default"><br></div><div class="gmail_default"><br></div><div class="gmail_default">using "virsh cpu-compare" I got the following:</div>
<div class="gmail_default"><br></div><div class="gmail_default"><div class="gmail_default">root@old:~# virsh cpu-compare model </div><div class="gmail_default">Host CPU is a superset of CPU described in model</div><div class="gmail_default">
<br></div><div class="gmail_default"><div class="gmail_default">root@new:~# virsh cpu-compare model </div><div class="gmail_default">Host CPU is a superset of CPU described in model</div><div class="gmail_default"><br></div>
<div class="gmail_default">Excellent!</div><div class="gmail_default"><br></div><div class="gmail_default">So I convert this over to the cpu_map.xml format:</div><div class="gmail_default"><div class="gmail_default"> <model name='ACG_COMMON'></div>
<div class="gmail_default"> <model name='Opteron_G3'/></div><div class="gmail_default"> <feature name='nodeid_msr'/></div><div class="gmail_default"> <feature name='wdt'/></div>
<div class="gmail_default"> <feature name='skinit'/></div><div class="gmail_default"> <feature name='ibs'/></div><div class="gmail_default"> <feature name='osvw'/></div>
<div class="gmail_default"> <feature name='3dnowprefetch'/></div><div class="gmail_default"> <feature name='cr8legacy'/></div><div class="gmail_default"> <feature name='extapic'/></div>
<div class="gmail_default"> <feature name='cmp_legacy'/></div><div class="gmail_default"> <feature name='pdpe1gb'/></div><div class="gmail_default"> <feature name='fxsr_opt'/></div>
<div class="gmail_default"> <feature name='mmxext'/></div><div class="gmail_default"> <feature name='ht'/></div><div class="gmail_default"> <feature name='vme'/></div>
<div class="gmail_default"> </model></div><div class="gmail_default"><br></div><div class="gmail_default"><br></div><div class="gmail_default">And launch a VM on the new node. Here are the CPU flags:</div><div class="gmail_default">
-cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme (from NEW processor)<br></div><div class="gmail_default"><br></div><div class="gmail_default">
Here are the cpu flags from a vm running on the old processors:</div><div class="gmail_default">-cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme (from the OLD processor)<br>
</div><div class="gmail_default"><br></div><div class="gmail_default">Again this looks great right? The new processor does not appear to support 3dnow or 3dnowext.</div><div class="gmail_default"><br></div><div class="gmail_default">
So when I try to migrate from NEW -> OLD I get this error on the OLD node: <a href="http://paste.openstack.org/show/53203/">http://paste.openstack.org/show/53203/</a></div><div class="gmail_default"><br></div><div class="gmail_default">
Any ideas what I should try next? It seems like this should have worked. Do I need to set the CPU model in the OLD nova.conf to the same custom cpu as in the nova.conf on the NEW node?</div><div class="gmail_default"><br>
</div><div class="gmail_default">Thanks,</div><div class="gmail_default">Sam</div><div class="gmail_default"><br></div></div></div></div></font></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Wed, Nov 13, 2013 at 4:47 PM, Samuel Winchenbach <span dir="ltr"><<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_default" style="font-family:'courier new',monospace">Thanks Steve,</div><div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">
I actually found which extensions the two CPUs had in common and created my own in cpu_map.xml (<a href="http://pastie.org/pastes/8478554/text" target="_blank">http://pastie.org/pastes/8478554/text</a>). The extensions commented out are ones that I could not find referenced in the cpu_map... I may try un-commenting them to see what happens :P</div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default"><span style="font-family:'courier new',monospace">It is strange... I start up a VM on the new compute node and it is running fine, when I try to do a live migration I get: "</span><font face="courier new, monospace">2013-11-13 16:43:18.063 ERROR nova.virt.libvirt.driver [req-7591a8a0-17ad-4539-b969-6b4a7d156963 43b2e3d7bed64bacab2f69b0cb7ce387 417de5251c804c45baabd0fa2873b43f] CPU doesn't have compatibility." which is odd because I only specified extensions the two CPUs have in common. Hrmmmm.</font></div>
<div class="gmail_default" style="font-family:'courier new',monospace"><br></div><div class="gmail_default" style="font-family:'courier new',monospace">Sam</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra">
<br><br><div class="gmail_quote">
On Wed, Nov 13, 2013 at 3:47 PM, Steve Gordon <span dir="ltr"><<a href="mailto:sgordon@redhat.com" target="_blank">sgordon@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>----- Original Message -----<br>
> From: "Samuel Winchenbach" <<a href="mailto:swinchen@gmail.com" target="_blank">swinchen@gmail.com</a>><br>
> To: <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a><br>
> Sent: Wednesday, November 13, 2013 2:57:32 PM<br>
> Subject: [Openstack-operators] New compute node, new libvirt_cpu_mode<br>
><br>
> Hi All.<br>
><br>
> I was wondering what the implications of changing libvirt_cpu_mode from the<br>
> default (host-passthrough) to none might be? I setup a new compute node<br>
> that has a slightly newer processor model than the other nodes and live<br>
> migration is NOT happy about that. I have done some reading and<br>
> discovered setting libvirt_cpu_mode=none should help solve the problem.<br>
> What can I expect if I do this? I am assuming a slight performance hit<br>
> as the VM will not have access to certain processor extensions. Here is a<br>
> list of /proce/cpuinfo for two vms (one with =none and the other default).<br>
><br>
> <a href="http://paste.openstack.org/show/52519/" target="_blank">http://paste.openstack.org/show/52519/</a><br>
><br>
> It is clear that the default mode enables more processor extensions.<br>
><br>
> Is there any other way around this problem?<br>
><br>
> Thanks,<br>
> Sam<br>
<br>
</div></div>If you can identify the Libvirt CPU model that best maps to the lowest common denominator for your cluster (that is includes the CPU flags that are common between the CPUs in your compute nodes) then you can set libvirt_cpu_mode to "custom" and set libvirt_cpu_model to the specific model [1]. This should be Opteron_G3 for the CPU exposed when you used "host-passthrough" in your output.<br>
<br>
This will give you better performance than the QEMU baseline CPU and more importantly working live migration but obviously means some of the flags exposed by the newest CPU in the cluster aren't made available to guests.<br>
<br>
-Steve<br>
<br>
[1] <a href="https://wiki.openstack.org/wiki/LibvirtXMLCPUModel" target="_blank">https://wiki.openstack.org/wiki/LibvirtXMLCPUModel</a><br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>