Live migration error
Hello Team,
I am trying to migrate instances from one host to another but I am getting this error.
*Error: *Failed to live migrate instance to host "compute1". Details http://10.10.13.31/admin/instances/#message_details Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da)
Any idea on how to fix this?
Regards
Tony Karera
It means that you have CPUs with incompatible flags or you've got differing kernel versions that expose different flags or you've got differing libvirt versions that expose different flags. You either need to use a lowest common denominator cpu_model or do a cold migration. ________________________________ From: Karera Tony tonykarera@gmail.com Sent: 01 September 2023 10:13 To: openstack-discuss openstack-discuss@lists.openstack.org Subject: Live migration error
CAUTION: This email originates from outside THG
________________________________ Hello Team,
I am trying to migrate instances from one host to another but I am getting this error.
Error: Failed to live migrate instance to host "compute1". Detailshttp://10.10.13.31/admin/instances/#message_details
Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResulthttp://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da)
Any idea on how to fix this?
Regards
Tony Karera
Hello Danny,
Thanks for the feedback.
use a lowest common denominator cpu_model : Does this mean changing the servers ?
Regards
Tony Karera
On Fri, Sep 1, 2023 at 12:37 PM Danny Webb Danny.Webb@thehutgroup.com wrote:
It means that you have CPUs with incompatible flags or you've got differing kernel versions that expose different flags or you've got differing libvirt versions that expose different flags. You either need to use a lowest common denominator cpu_model or do a cold migration.
*From:* Karera Tony tonykarera@gmail.com *Sent:* 01 September 2023 10:13 *To:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Live migration error
- CAUTION: This email originates from outside THG *
Hello Team,
I am trying to migrate instances from one host to another but I am getting this error.
*Error: *Failed to live migrate instance to host "compute1". Details http://10.10.13.31/admin/instances/#message_details Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da)
Any idea on how to fix this?
Regards
Tony Karera
On Fri, 2023-09-01 at 14:56 +0200, Karera Tony wrote:
Hello Danny,
Thanks for the feedback.
use a lowest common denominator cpu_model : Does this mean changing the servers ?
it means tha tif you have a mix of cpus in the deployment you shoudl hardcod to the older of the diffent cpu models i.e skylake or whatever it may be in your case.
https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu...
[libvirt] cpu_mode=custom cpu_models=skylake
in more recent release we replaced the older cpu_model with cpu_models with is a comma sperated list of models in prefernce order. i.e. nova will use the first cpu_modle in the list that work on the current host. not that while this give extra flexiblity it vs just using the oldest cpu model it does limit the set of hosts you can migrate too but means you can have better perfromance so its a tradeoff. performance vs portablity.
the error you mentioned can also be caused by micorocode updates. intel remove TSX and MPX i beilve in the last 2 years form come of there cpus that breaks live migration if a vm was create with access to that cpu instruction
the only way to resolve that is to cold migrate the guest. i.e. if a vm is currently booted with cpu_model X it cannot be modifed while the guest is running so you either need to update the config option and reboot all the vms or more particlaly update the config and then cold migrate them which will allow you to move the instnace(your orginal goal) while allowing the new cpu model to take effect.
novas default for the cpu model when not set is with our default cpu_mode is host_model meaning whatever model best matches the current hosts cpu.
that is the correct default in general but if you have mixed cpu generatiosn in your cloud its not ideal.
hopefuly that helps
Regards
Tony Karera
On Fri, Sep 1, 2023 at 12:37 PM Danny Webb Danny.Webb@thehutgroup.com wrote:
It means that you have CPUs with incompatible flags or you've got differing kernel versions that expose different flags or you've got differing libvirt versions that expose different flags. You either need to use a lowest common denominator cpu_model or do a cold migration.
*From:* Karera Tony tonykarera@gmail.com *Sent:* 01 September 2023 10:13 *To:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Live migration error
- CAUTION: This email originates from outside THG *
Hello Team,
I am trying to migrate instances from one host to another but I am getting this error.
*Error: *Failed to live migrate instance to host "compute1". Details http://10.10.13.31/admin/instances/#message_details Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da)
Any idea on how to fix this?
Regards
Tony Karera
Hello Danny,
I am actually using Broadwell-noTSX-IBRS on all the servers but I have seen that some flags are not there in the older servers.
The flags below to be specific.
hwp hwp_act_window hwp_pkg_req
Regards
Tony Karera
On Fri, Sep 1, 2023 at 4:05 PM smooney@redhat.com wrote:
On Fri, 2023-09-01 at 14:56 +0200, Karera Tony wrote:
Hello Danny,
Thanks for the feedback.
use a lowest common denominator cpu_model : Does this mean changing the servers ?
it means tha tif you have a mix of cpus in the deployment you shoudl hardcod to the older of the diffent cpu models i.e skylake or whatever it may be in your case.
https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu...
[libvirt] cpu_mode=custom cpu_models=skylake
in more recent release we replaced the older cpu_model with cpu_models with is a comma sperated list of models in prefernce order. i.e. nova will use the first cpu_modle in the list that work on the current host. not that while this give extra flexiblity it vs just using the oldest cpu model it does limit the set of hosts you can migrate too but means you can have better perfromance so its a tradeoff. performance vs portablity.
the error you mentioned can also be caused by micorocode updates. intel remove TSX and MPX i beilve in the last 2 years form come of there cpus that breaks live migration if a vm was create with access to that cpu instruction
the only way to resolve that is to cold migrate the guest. i.e. if a vm is currently booted with cpu_model X it cannot be modifed while the guest is running so you either need to update the config option and reboot all the vms or more particlaly update the config and then cold migrate them which will allow you to move the instnace(your orginal goal) while allowing the new cpu model to take effect.
novas default for the cpu model when not set is with our default cpu_mode is host_model meaning whatever model best matches the current hosts cpu.
that is the correct default in general but if you have mixed cpu generatiosn in your cloud its not ideal.
hopefuly that helps
Regards
Tony Karera
On Fri, Sep 1, 2023 at 12:37 PM Danny Webb Danny.Webb@thehutgroup.com wrote:
It means that you have CPUs with incompatible flags or you've got differing kernel versions that expose different flags or you've got differing libvirt versions that expose different flags. You either
need to
use a lowest common denominator cpu_model or do a cold migration.
*From:* Karera Tony tonykarera@gmail.com *Sent:* 01 September 2023 10:13 *To:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Live migration error
- CAUTION: This email originates from outside THG *
Hello Team,
I am trying to migrate instances from one host to another but I am
getting
this error.
*Error: *Failed to live migrate instance to host "compute1". Details http://10.10.13.31/admin/instances/#message_details Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da)
Any idea on how to fix this?
Regards
Tony Karera
On Mon, 2023-09-04 at 12:59 +0200, Karera Tony wrote:
Hello Danny,
I am actually using Broadwell-noTSX-IBRS on all the servers but I have seen that some flags are not there in the older servers.
The flags below to be specific.
hwp hwp_act_window hwp_pkg_req
so hwp is apprentlly intels pstate contol based on https://unix.stackexchange.com/a/43540
i dont belive this is normally exposed to a guest but if you are seeign differnce between hosts then that likely means you have disabled pstates also know as "intel speedstep" on some of the host and not others.
Regards
Tony Karera
On Fri, Sep 1, 2023 at 4:05 PM smooney@redhat.com wrote:
On Fri, 2023-09-01 at 14:56 +0200, Karera Tony wrote:
Hello Danny,
Thanks for the feedback.
use a lowest common denominator cpu_model : Does this mean changing the servers ?
it means tha tif you have a mix of cpus in the deployment you shoudl hardcod to the older of the diffent cpu models i.e skylake or whatever it may be in your case.
https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu...
[libvirt] cpu_mode=custom cpu_models=skylake
in more recent release we replaced the older cpu_model with cpu_models with is a comma sperated list of models in prefernce order. i.e. nova will use the first cpu_modle in the list that work on the current host. not that while this give extra flexiblity it vs just using the oldest cpu model it does limit the set of hosts you can migrate too but means you can have better perfromance so its a tradeoff. performance vs portablity.
the error you mentioned can also be caused by micorocode updates. intel remove TSX and MPX i beilve in the last 2 years form come of there cpus that breaks live migration if a vm was create with access to that cpu instruction
the only way to resolve that is to cold migrate the guest. i.e. if a vm is currently booted with cpu_model X it cannot be modifed while the guest is running so you either need to update the config option and reboot all the vms or more particlaly update the config and then cold migrate them which will allow you to move the instnace(your orginal goal) while allowing the new cpu model to take effect.
novas default for the cpu model when not set is with our default cpu_mode is host_model meaning whatever model best matches the current hosts cpu.
that is the correct default in general but if you have mixed cpu generatiosn in your cloud its not ideal.
hopefuly that helps
Regards
Tony Karera
On Fri, Sep 1, 2023 at 12:37 PM Danny Webb Danny.Webb@thehutgroup.com wrote:
It means that you have CPUs with incompatible flags or you've got differing kernel versions that expose different flags or you've got differing libvirt versions that expose different flags. You either
need to
use a lowest common denominator cpu_model or do a cold migration.
*From:* Karera Tony tonykarera@gmail.com *Sent:* 01 September 2023 10:13 *To:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Live migration error
- CAUTION: This email originates from outside THG *
Hello Team,
I am trying to migrate instances from one host to another but I am
getting
this error.
*Error: *Failed to live migrate instance to host "compute1". Details http://10.10.13.31/admin/instances/#message_details Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult (HTTP 400) (Request-ID: req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da)
Any idea on how to fix this?
Regards
Tony Karera
Hello Smooney,
I can confirm that it is disabled on all the servers.
Regards
Tony Karera
On Mon, Sep 4, 2023 at 1:25 PM smooney@redhat.com wrote:
On Mon, 2023-09-04 at 12:59 +0200, Karera Tony wrote:
Hello Danny,
I am actually using Broadwell-noTSX-IBRS on all the servers but I have
seen
that some flags are not there in the older servers.
The flags below to be specific.
hwp hwp_act_window hwp_pkg_req
so hwp is apprentlly intels pstate contol based on https://unix.stackexchange.com/a/43540
i dont belive this is normally exposed to a guest but if you are seeign differnce between hosts then that likely means you have disabled pstates also know as "intel speedstep" on some of the host and not others.
Regards
Tony Karera
On Fri, Sep 1, 2023 at 4:05 PM smooney@redhat.com wrote:
On Fri, 2023-09-01 at 14:56 +0200, Karera Tony wrote:
Hello Danny,
Thanks for the feedback.
use a lowest common denominator cpu_model : Does this mean changing
the
servers ?
it means tha tif you have a mix of cpus in the deployment you shoudl hardcod to the older of the diffent cpu models i.e skylake or whatever it may
be
in your case.
https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu...
[libvirt] cpu_mode=custom cpu_models=skylake
in more recent release we replaced the older cpu_model with cpu_models with is a comma sperated list of models in prefernce order. i.e. nova will use the first
cpu_modle
in the list that work on the current host. not that while this give extra flexiblity it vs
just
using the oldest cpu model it does limit the set of hosts you can migrate too but means you can
have
better perfromance so its a tradeoff. performance vs portablity.
the error you mentioned can also be caused by micorocode updates. intel remove TSX and MPX i beilve in the last 2 years form come of
there
cpus that breaks live migration if a vm was create with access to that cpu instruction
the only way to resolve that is to cold migrate the guest. i.e. if a vm is currently booted with cpu_model X it cannot be modifed while the guest is running so you either need to update the config option and reboot all the vms
or
more particlaly update the config and then cold migrate them which will allow you to move the
instnace(your
orginal goal) while allowing the new cpu model to take effect.
novas default for the cpu model when not set is with our default
cpu_mode
is host_model meaning whatever model best matches the current hosts cpu.
that is the correct default in general but if you have mixed cpu generatiosn in your cloud its not ideal.
hopefuly that helps
Regards
Tony Karera
On Fri, Sep 1, 2023 at 12:37 PM Danny Webb <
Danny.Webb@thehutgroup.com>
wrote:
It means that you have CPUs with incompatible flags or you've got differing kernel versions that expose different flags or you've got differing libvirt versions that expose different flags. You either
need to
use a lowest common denominator cpu_model or do a cold migration.
*From:* Karera Tony tonykarera@gmail.com *Sent:* 01 September 2023 10:13 *To:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Live migration error
- CAUTION: This email originates from outside THG *
Hello Team,
I am trying to migrate instances from one host to another but I am
getting
this error.
*Error: *Failed to live migrate instance to host "compute1".
Details
http://10.10.13.31/admin/instances/#message_details Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to
http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
(HTTP 400) (Request-ID: req-9dfdfe2e-70fa-481a-bd3d-c2c76a6e93da)
Any idea on how to fix this?
Regards
Tony Karera
participants (3)
-
Danny Webb
-
Karera Tony
-
smooney＠redhat.com