I am testing vm migration manually using virsh from one compute node to another: --- nova@ccompute:~$ virsh migrate instance-0003457f0 qemu+ssh://<remote-ip>/system --live --verbose --unsafe error: Unable to open file: /var/lib/nova/instances/f4d4r423r4d/console.log: No such file or directory nova@ccompute:~$ ll /var/lib/nova/instances/f4d4r423r4d/ drwxr-xr-x 2 nova nova 4096 Feb 17 15:43 ./ drwxr-xr-x 14 nova nova 4096 Dec 16 22:31 ../ -rw------- 1 nova root 56864 Feb 17 15:43 console.log -rw-r--r-- 1 libvirt-qemu kvm 1154624 Feb 17 15:35 disk -rw-r--r-- 1 nova nova 79 Dec 16 22:30 disk.info ------ I am doing the above as 'nova' user. Why is it failing even though it has r/w access to the console file ? Thanks elinux
Since you're doing it with virsh directly, I would assume that the console.log should be owned by libvirt-qemu user (like the disk) and/or group kvm? I haven't tried it this way myself though, but probably worth a shot. Zitat von engineer2024 <engineerlinux2024@gmail.com>:
I am testing vm migration manually using virsh from one compute node to another:
--- nova@ccompute:~$ virsh migrate instance-0003457f0 qemu+ssh://<remote-ip>/system --live --verbose --unsafe error: Unable to open file: /var/lib/nova/instances/f4d4r423r4d/console.log: No such file or directory
nova@ccompute:~$ ll /var/lib/nova/instances/f4d4r423r4d/ drwxr-xr-x 2 nova nova 4096 Feb 17 15:43 ./ drwxr-xr-x 14 nova nova 4096 Dec 16 22:31 ../ -rw------- 1 nova root 56864 Feb 17 15:43 console.log -rw-r--r-- 1 libvirt-qemu kvm 1154624 Feb 17 15:35 disk -rw-r--r-- 1 nova nova 79 Dec 16 22:30 disk.info ------ I am doing the above as 'nova' user. Why is it failing even though it has r/w access to the console file ?
Thanks elinux
I would be very surprised for this to work, as I would expect nova-compute to destroy the VM which is not supposed to be there once it's started. On Mon, 17 Feb 2025, 18:23 Eugen Block, <eblock@nde.ag> wrote:
Since you're doing it with virsh directly, I would assume that the console.log should be owned by libvirt-qemu user (like the disk) and/or group kvm? I haven't tried it this way myself though, but probably worth a shot.
Zitat von engineer2024 <engineerlinux2024@gmail.com>:
I am testing vm migration manually using virsh from one compute node to another:
--- nova@ccompute:~$ virsh migrate instance-0003457f0 qemu+ssh://<remote-ip>/system --live --verbose --unsafe error: Unable to open file: /var/lib/nova/instances/f4d4r423r4d/console.log: No such file or directory
nova@ccompute:~$ ll /var/lib/nova/instances/f4d4r423r4d/ drwxr-xr-x 2 nova nova 4096 Feb 17 15:43 ./ drwxr-xr-x 14 nova nova 4096 Dec 16 22:31 ../ -rw------- 1 nova root 56864 Feb 17 15:43 console.log -rw-r--r-- 1 libvirt-qemu kvm 1154624 Feb 17 15:35 disk -rw-r--r-- 1 nova nova 79 Dec 16 22:30 disk.info ------ I am doing the above as 'nova' user. Why is it failing even though it has r/w access to the console file ?
Thanks elinux
So should the nova api and compute services be stopped ? On Mon, 17 Feb 2025, 22:55 Dmitriy Rabotyagov, <noonedeadpunk@gmail.com> wrote:
I would be very surprised for this to work, as I would expect nova-compute to destroy the VM which is not supposed to be there once it's started.
On Mon, 17 Feb 2025, 18:23 Eugen Block, <eblock@nde.ag> wrote:
Since you're doing it with virsh directly, I would assume that the console.log should be owned by libvirt-qemu user (like the disk) and/or group kvm? I haven't tried it this way myself though, but probably worth a shot.
Zitat von engineer2024 <engineerlinux2024@gmail.com>:
I am testing vm migration manually using virsh from one compute node to another:
--- nova@ccompute:~$ virsh migrate instance-0003457f0 qemu+ssh://<remote-ip>/system --live --verbose --unsafe error: Unable to open file: /var/lib/nova/instances/f4d4r423r4d/console.log: No such file or directory
nova@ccompute:~$ ll /var/lib/nova/instances/f4d4r423r4d/ drwxr-xr-x 2 nova nova 4096 Feb 17 15:43 ./ drwxr-xr-x 14 nova nova 4096 Dec 16 22:31 ../ -rw------- 1 nova root 56864 Feb 17 15:43 console.log -rw-r--r-- 1 libvirt-qemu kvm 1154624 Feb 17 15:35 disk -rw-r--r-- 1 nova nova 79 Dec 16 22:30 disk.info ------ I am doing the above as 'nova' user. Why is it failing even though it has r/w access to the console file ?
Thanks elinux
Can you please elaborate a bit on why you are even doing this instead of relying on the nova API to execute live migrations? On Mon, 17 Feb 2025, 18:33 engineer2024, <engineerlinux2024@gmail.com> wrote:
So should the nova api and compute services be stopped ?
On Mon, 17 Feb 2025, 22:55 Dmitriy Rabotyagov, <noonedeadpunk@gmail.com> wrote:
I would be very surprised for this to work, as I would expect nova-compute to destroy the VM which is not supposed to be there once it's started.
On Mon, 17 Feb 2025, 18:23 Eugen Block, <eblock@nde.ag> wrote:
Since you're doing it with virsh directly, I would assume that the console.log should be owned by libvirt-qemu user (like the disk) and/or group kvm? I haven't tried it this way myself though, but probably worth a shot.
Zitat von engineer2024 <engineerlinux2024@gmail.com>:
I am testing vm migration manually using virsh from one compute node to another:
--- nova@ccompute:~$ virsh migrate instance-0003457f0 qemu+ssh://<remote-ip>/system --live --verbose --unsafe error: Unable to open file: /var/lib/nova/instances/f4d4r423r4d/console.log: No such file or directory
nova@ccompute:~$ ll /var/lib/nova/instances/f4d4r423r4d/ drwxr-xr-x 2 nova nova 4096 Feb 17 15:43 ./ drwxr-xr-x 14 nova nova 4096 Dec 16 22:31 ../ -rw------- 1 nova root 56864 Feb 17 15:43 console.log -rw-r--r-- 1 libvirt-qemu kvm 1154624 Feb 17 15:35 disk -rw-r--r-- 1 nova nova 79 Dec 16 22:30 disk.info ------ I am doing the above as 'nova' user. Why is it failing even though it has r/w access to the console file ?
Thanks elinux
Good question.. want to study the performance of live migration from the libvirt level. Trying to use guestperf.py script written by Dan Berrange. But I am not sure if it works with a running vm. Pinpointing the bottlenecks during live migration. Apart from post copy, rdma nics, have to explore multifds for quicker device state transfers. The goal is to minimize the LM time On Mon, 17 Feb 2025, 23:21 Dmitriy Rabotyagov, <noonedeadpunk@gmail.com> wrote:
Can you please elaborate a bit on why you are even doing this instead of relying on the nova API to execute live migrations?
On Mon, 17 Feb 2025, 18:33 engineer2024, <engineerlinux2024@gmail.com> wrote:
So should the nova api and compute services be stopped ?
On Mon, 17 Feb 2025, 22:55 Dmitriy Rabotyagov, <noonedeadpunk@gmail.com> wrote:
I would be very surprised for this to work, as I would expect nova-compute to destroy the VM which is not supposed to be there once it's started.
On Mon, 17 Feb 2025, 18:23 Eugen Block, <eblock@nde.ag> wrote:
Since you're doing it with virsh directly, I would assume that the console.log should be owned by libvirt-qemu user (like the disk) and/or group kvm? I haven't tried it this way myself though, but probably worth a shot.
Zitat von engineer2024 <engineerlinux2024@gmail.com>:
I am testing vm migration manually using virsh from one compute node to another:
--- nova@ccompute:~$ virsh migrate instance-0003457f0 qemu+ssh://<remote-ip>/system --live --verbose --unsafe error: Unable to open file: /var/lib/nova/instances/f4d4r423r4d/console.log: No such file or directory
nova@ccompute:~$ ll /var/lib/nova/instances/f4d4r423r4d/ drwxr-xr-x 2 nova nova 4096 Feb 17 15:43 ./ drwxr-xr-x 14 nova nova 4096 Dec 16 22:31 ../ -rw------- 1 nova root 56864 Feb 17 15:43 console.log -rw-r--r-- 1 libvirt-qemu kvm 1154624 Feb 17 15:35 disk -rw-r--r-- 1 nova nova 79 Dec 16 22:30 disk.info ------ I am doing the above as 'nova' user. Why is it failing even though it has r/w access to the console file ?
Thanks elinux
For this kind of exploratory work, it's likely easier to use QEMU and Libvirt directly, outside of OpenStack. You will have more control over the live migration process. You can always work on transferring what you learned into OpenStack later. On Mon, 17 Feb 2025 at 19:01, engineer2024 <engineerlinux2024@gmail.com> wrote:
Good question.. want to study the performance of live migration from the libvirt level. Trying to use guestperf.py script written by Dan Berrange. But I am not sure if it works with a running vm. Pinpointing the bottlenecks during live migration. Apart from post copy, rdma nics, have to explore multifds for quicker device state transfers. The goal is to minimize the LM time
On Mon, 17 Feb 2025, 23:21 Dmitriy Rabotyagov, <noonedeadpunk@gmail.com> wrote:
Can you please elaborate a bit on why you are even doing this instead of relying on the nova API to execute live migrations?
On Mon, 17 Feb 2025, 18:33 engineer2024, <engineerlinux2024@gmail.com> wrote:
So should the nova api and compute services be stopped ?
On Mon, 17 Feb 2025, 22:55 Dmitriy Rabotyagov, <noonedeadpunk@gmail.com> wrote:
I would be very surprised for this to work, as I would expect nova-compute to destroy the VM which is not supposed to be there once it's started.
On Mon, 17 Feb 2025, 18:23 Eugen Block, <eblock@nde.ag> wrote:
Since you're doing it with virsh directly, I would assume that the console.log should be owned by libvirt-qemu user (like the disk) and/or group kvm? I haven't tried it this way myself though, but probably worth a shot.
Zitat von engineer2024 <engineerlinux2024@gmail.com>:
I am testing vm migration manually using virsh from one compute node to another:
--- nova@ccompute:~$ virsh migrate instance-0003457f0 qemu+ssh://<remote-ip>/system --live --verbose --unsafe error: Unable to open file: /var/lib/nova/instances/f4d4r423r4d/console.log: No such file or directory
nova@ccompute:~$ ll /var/lib/nova/instances/f4d4r423r4d/ drwxr-xr-x 2 nova nova 4096 Feb 17 15:43 ./ drwxr-xr-x 14 nova nova 4096 Dec 16 22:31 ../ -rw------- 1 nova root 56864 Feb 17 15:43 console.log -rw-r--r-- 1 libvirt-qemu kvm 1154624 Feb 17 15:35 disk -rw-r--r-- 1 nova nova 79 Dec 16 22:30 disk.info ------ I am doing the above as 'nova' user. Why is it failing even though it has r/w access to the console file ?
Thanks elinux
for the record if you do any write operation i.e. modifying a domain xml or live migrating an instnace form a nova perscitive the vm is entirely unsupported in a downstream context that is highly likely to violate your support agreement with a vendor upstream if you modify the domain in way and it results in an issue we would not consider it valid bug. as far as operators should be concerned interacting with libvirt is off limits. it can be done in a read only mode for debugging but otherwise its breaking the contract between nova and libvirt of you make direct modification via the qemu monitor or the libvirt api. it sound like you are trying to debug a performance issue in your environment? or at least characterize the performance? qemu+ssh:// is itself a bottleneck as you have the overhead of tunneling the guest traffic over ssh using native tls is faster, as is raw tcp but that is not encrypted so its not recommended. enable ssh compression can also introduce a CPU bottleneck. the main limitation tends to be the overall network bandwidth available to the migration but not in all cases. On 17/02/2025 18:15, Pierre Riteau wrote:
For this kind of exploratory work, it's likely easier to use QEMU and Libvirt directly, outside of OpenStack. You will have more control over the live migration process. You can always work on transferring what you learned into OpenStack later.
On Mon, 17 Feb 2025 at 19:01, engineer2024 <engineerlinux2024@gmail.com> wrote:
Good question.. want to study the performance of live migration from the libvirt level. Trying to use guestperf.py script written by Dan Berrange. But I am not sure if it works with a running vm. Pinpointing the bottlenecks during live migration. Apart from post copy, rdma nics, have to explore multifds for quicker device state transfers. The goal is to minimize the LM time
On Mon, 17 Feb 2025, 23:21 Dmitriy Rabotyagov, <noonedeadpunk@gmail.com> wrote:
Can you please elaborate a bit on why you are even doing this instead of relying on the nova API to execute live migrations?
On Mon, 17 Feb 2025, 18:33 engineer2024, <engineerlinux2024@gmail.com> wrote:
So should the nova api and compute services be stopped ?
On Mon, 17 Feb 2025, 22:55 Dmitriy Rabotyagov, <noonedeadpunk@gmail.com> wrote:
I would be very surprised for this to work, as I would expect nova-compute to destroy the VM which is not supposed to be there once it's started.
On Mon, 17 Feb 2025, 18:23 Eugen Block, <eblock@nde.ag> wrote:
Since you're doing it with virsh directly, I would assume that the console.log should be owned by libvirt-qemu user (like the disk) and/or group kvm? I haven't tried it this way myself though, but probably worth a shot.
Zitat von engineer2024 <engineerlinux2024@gmail.com>:
> I am testing vm migration manually using virsh from one compute node to > another: > > --- > nova@ccompute:~$ virsh migrate instance-0003457f0 > qemu+ssh://<remote-ip>/system --live --verbose --unsafe > error: Unable to open file: > /var/lib/nova/instances/f4d4r423r4d/console.log: No such file or directory > > > nova@ccompute:~$ ll /var/lib/nova/instances/f4d4r423r4d/ > drwxr-xr-x 2 nova nova 4096 Feb 17 15:43 ./ > drwxr-xr-x 14 nova nova 4096 Dec 16 22:31 ../ > -rw------- 1 nova root 56864 Feb 17 15:43 console.log > -rw-r--r-- 1 libvirt-qemu kvm 1154624 Feb 17 15:35 disk > -rw-r--r-- 1 nova nova 79 Dec 16 22:30 disk.info > ------ > I am doing the above as 'nova' user. Why is it failing even though it has > r/w access to the console file ? > > Thanks > elinux
participants (5)
-
Dmitriy Rabotyagov
-
engineer2024
-
Eugen Block
-
Pierre Riteau
-
Sean Mooney