[ops][nova][victoria] QEMU/KVM libvirt Authentication Failed
All; I feel like I've dealt with this issue before, but I can't find any records of it. I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again. I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1]. When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com 1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#s...
Just a heads up even if you get things working you’re not going to be able to live migrate from centos to ubuntu and vice versa since there’s going to be things like apparmor and SELinux issues On Fri, Nov 19, 2021 at 11:35 AM <DHilsbos@performair.com> wrote:
All;
I feel like I've dealt with this issue before, but I can't find any records of it.
I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again.
I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1].
When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed
At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for?
Thank you,
Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#s...
-- Mohammed Naser VEXXHOST, Inc.
Which version of Openstack are you running? It seems to try to connect over qemu with auth over tcp. Without ssh? Is the cold migration working now? On Fri, Nov 19, 2021 at 8:35 PM Mohammed Naser <mnaser@vexxhost.com> wrote:
Just a heads up even if you get things working you’re not going to be able to live migrate from centos to ubuntu and vice versa since there’s going to be things like apparmor and SELinux issues
On Fri, Nov 19, 2021 at 11:35 AM <DHilsbos@performair.com> wrote:
All;
I feel like I've dealt with this issue before, but I can't find any records of it.
I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again.
I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1].
When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed
At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for?
Thank you,
Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#s...
-- Mohammed Naser VEXXHOST, Inc.
Laurent; We're running Victoria. Here are specific package versions: Ubuntu: 20.10 nova-compute: 22.2.1-0ubuntu1 (both) nova-compute-kvm: 22.2.1-0ubuntu1 (both) qemu-kvm: 5.0-5unbuntu9.9 (both) libvirt-daemon: 6.6.0-1ubuntu3.5 (both) As I said, this has come up for me before, but I can't find records of how it was addressed. I don't remember an issue of authentication from before, however. From before, I do remember that after the ssh connection to setup the new host, qemu/kvm on the old host makes a connection to qemu/kvm on the new host, in order to coordinate the transfer of memory contents, and other dynamic elements. Yes, I can cold migrate between all 3 servers, which makes this a non-critical issue. While I have a CentOS Nova host, I'm not going to attempt to get live-migration working between the Ubuntu Servers Changing the configuration of libvirt from system sockets to native listeners got me past a connection refused error (it appears that lbvirt also connects from one server to another?) Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com From: Laurent Dumont [mailto:laurentfdumont@gmail.com] Sent: Friday, November 19, 2021 6:45 PM To: Mohammed Naser Cc: Dominic Hilsbos; openstack-discuss Subject: Re: [ops][nova][victoria] QEMU/KVM libvirt Authentication Failed Which version of Openstack are you running? It seems to try to connect over qemu with auth over tcp. Without ssh? Is the cold migration working now? On Fri, Nov 19, 2021 at 8:35 PM Mohammed Naser <mnaser@vexxhost.com> wrote: Just a heads up even if you get things working you’re not going to be able to live migrate from centos to ubuntu and vice versa since there’s going to be things like apparmor and SELinux issues On Fri, Nov 19, 2021 at 11:35 AM <DHilsbos@performair.com> wrote: All; I feel like I've dealt with this issue before, but I can't find any records of it. I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again. I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1]. When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com 1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#s... -- Mohammed Naser VEXXHOST, Inc.
Laurent; Your message ended up pointing me in the right direction. I started asking myself why libvirtd came from Ubuntu configured incorrectly for live migrations. The obvious answer is: it didn't. That suggested that I ad configured something incorrectly. That realization, together with the discussion from [1] led me to set libvirt.live_migration_scheme="ssh" in nova.conf. After restarting nova-compute, I can now live migrate instances between Ubuntu servers. Thank you for your assistance. Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com [1] https://bugzilla.redhat.com/show_bug.cgi?id=1254307 -----Original Message----- From: DHilsbos@performair.com [mailto:DHilsbos@performair.com] Sent: Monday, November 22, 2021 9:09 AM To: laurentfdumont@gmail.com; mnaser@vexxhost.com Cc: openstack-discuss@lists.openstack.org Subject: RE: [ops][nova][victoria] QEMU/KVM libvirt Authentication Failed Laurent; We're running Victoria. Here are specific package versions: Ubuntu: 20.10 nova-compute: 22.2.1-0ubuntu1 (both) nova-compute-kvm: 22.2.1-0ubuntu1 (both) qemu-kvm: 5.0-5unbuntu9.9 (both) libvirt-daemon: 6.6.0-1ubuntu3.5 (both) As I said, this has come up for me before, but I can't find records of how it was addressed. I don't remember an issue of authentication from before, however. From before, I do remember that after the ssh connection to setup the new host, qemu/kvm on the old host makes a connection to qemu/kvm on the new host, in order to coordinate the transfer of memory contents, and other dynamic elements. Yes, I can cold migrate between all 3 servers, which makes this a non-critical issue. While I have a CentOS Nova host, I'm not going to attempt to get live-migration working between the Ubuntu Servers Changing the configuration of libvirt from system sockets to native listeners got me past a connection refused error (it appears that lbvirt also connects from one server to another?) Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com From: Laurent Dumont [mailto:laurentfdumont@gmail.com] Sent: Friday, November 19, 2021 6:45 PM To: Mohammed Naser Cc: Dominic Hilsbos; openstack-discuss Subject: Re: [ops][nova][victoria] QEMU/KVM libvirt Authentication Failed Which version of Openstack are you running? It seems to try to connect over qemu with auth over tcp. Without ssh? Is the cold migration working now? On Fri, Nov 19, 2021 at 8:35 PM Mohammed Naser <mnaser@vexxhost.com> wrote: Just a heads up even if you get things working you’re not going to be able to live migrate from centos to ubuntu and vice versa since there’s going to be things like apparmor and SELinux issues On Fri, Nov 19, 2021 at 11:35 AM <DHilsbos@performair.com> wrote: All; I feel like I've dealt with this issue before, but I can't find any records of it. I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again. I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1]. When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com 1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#s... -- Mohammed Naser VEXXHOST, Inc.
Fantastic! Glad you were able to find a solution. On Mon, Nov 22, 2021 at 12:02 PM <DHilsbos@performair.com> wrote:
Laurent;
Your message ended up pointing me in the right direction. I started asking myself why libvirtd came from Ubuntu configured incorrectly for live migrations. The obvious answer is: it didn't. That suggested that I ad configured something incorrectly. That realization, together with the discussion from [1] led me to set libvirt.live_migration_scheme="ssh" in nova.conf. After restarting nova-compute, I can now live migrate instances between Ubuntu servers.
Thank you for your assistance.
Thank you,
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1254307
-----Original Message----- From: DHilsbos@performair.com [mailto:DHilsbos@performair.com] Sent: Monday, November 22, 2021 9:09 AM To: laurentfdumont@gmail.com; mnaser@vexxhost.com Cc: openstack-discuss@lists.openstack.org Subject: RE: [ops][nova][victoria] QEMU/KVM libvirt Authentication Failed
Laurent;
We're running Victoria. Here are specific package versions: Ubuntu: 20.10 nova-compute: 22.2.1-0ubuntu1 (both) nova-compute-kvm: 22.2.1-0ubuntu1 (both) qemu-kvm: 5.0-5unbuntu9.9 (both) libvirt-daemon: 6.6.0-1ubuntu3.5 (both)
As I said, this has come up for me before, but I can't find records of how it was addressed. I don't remember an issue of authentication from before, however. From before, I do remember that after the ssh connection to setup the new host, qemu/kvm on the old host makes a connection to qemu/kvm on the new host, in order to coordinate the transfer of memory contents, and other dynamic elements.
Yes, I can cold migrate between all 3 servers, which makes this a non-critical issue.
While I have a CentOS Nova host, I'm not going to attempt to get live-migration working between the Ubuntu Servers
Changing the configuration of libvirt from system sockets to native listeners got me past a connection refused error (it appears that lbvirt also connects from one server to another?)
Thank you,
Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
From: Laurent Dumont [mailto:laurentfdumont@gmail.com] Sent: Friday, November 19, 2021 6:45 PM To: Mohammed Naser Cc: Dominic Hilsbos; openstack-discuss Subject: Re: [ops][nova][victoria] QEMU/KVM libvirt Authentication Failed
Which version of Openstack are you running?
It seems to try to connect over qemu with auth over tcp. Without ssh?
Is the cold migration working now?
On Fri, Nov 19, 2021 at 8:35 PM Mohammed Naser <mnaser@vexxhost.com> wrote: Just a heads up even if you get things working you’re not going to be able to live migrate from centos to ubuntu and vice versa since there’s going to be things like apparmor and SELinux issues
On Fri, Nov 19, 2021 at 11:35 AM <DHilsbos@performair.com> wrote: All;
I feel like I've dealt with this issue before, but I can't find any records of it.
I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again.
I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1].
When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed
At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for?
Thank you,
Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#s... -- Mohammed Naser VEXXHOST, Inc.
Mohammed; Yep, I'm aware. I have 3 Nova servers, 1 runs CentOS, 2 run Ubuntu. I'm only trying to live migrate between the 2 Ubuntu servers. Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com From: Mohammed Naser [mailto:mnaser@vexxhost.com] Sent: Friday, November 19, 2021 6:33 PM To: Dominic Hilsbos Cc: openstack-discuss@lists.openstack.org Subject: Re: [ops][nova][victoria] QEMU/KVM libvirt Authentication Failed Just a heads up even if you get things working you’re not going to be able to live migrate from centos to ubuntu and vice versa since there’s going to be things like apparmor and SELinux issues On Fri, Nov 19, 2021 at 11:35 AM <DHilsbos@performair.com> wrote: All; I feel like I've dealt with this issue before, but I can't find any records of it. I've been swapping out the compute nodes in my cluster for newer and better hardware. We also decided to abandon CentOS. All the differences mean that we haven't been able to do live migrations. I now have 2 servers with the same CPUs, OS (Ubuntu), OS Version (20.10), etc., and would like to get live migration working again. I configured passwordless ssh access between the servers for the nova users to get cold migration working. I have also configured passwordless ssh for the root users in accordance with [1]. When I try to do a live migration, the origin server generates this error, in the nova-compute log: 2021-11-19 15:52:31.130 15610 ERROR nova.virt.libvirt.driver [-] [instance: 5935c07d-0c7f-48cc-a4b9-674504fc6005] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://<remote_server>/system: authentication failed: authentication failed At one point, I came across a tutorial on configuring live-migration for libvirt, which included a bunch of user configuration. I don't remember having to do that before, but is that what I need to be looking for? Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com 1: https://docs.openstack.org/nova/victoria/admin/configuring-migrations.html#s... -- Mohammed Naser VEXXHOST, Inc.
participants (3)
-
DHilsbos＠performair.com
-
Laurent Dumont
-
Mohammed Naser