Hi Oliver,

I just pushed up a PR that I tested which lists all VMs if it doesn’t find any VMs:

https://github.com/vexxhost/migratekit/pull/10

I’ve merged it and this pushed up a new image that has gotten promoted, so I suggest pulling the latest image (aka `docker pull …`) and trying again. It’ll help you find the right path. 😊

Thanks, Mohammed

From: Oliver Weinmann oliver.weinmann@me.com Date: Monday, August 26, 2024 at 9:02 AM To: Mohammed Naser mnaser@vexxhost.com Cc: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: Migratekit: near-live migrations from VMware to OpenStack Hi Mohammed,

thanks four your quick reply.

Unfortunately this also doesn't work.

Cheers, Oliver

On 26 Aug 2024, at 14:49, Mohammed Naser mnaser@vexxhost.com wrote:

Hi Oliver:

Can you add quotes around the path and replace the %20 with spaces and see if that works?

A frequent ask is being able to list VMs.. we may just add that feature to make it easier to use.

Thanks Mohammed

GetOutlook for iOShttps://aka.ms/o0ukef ________________________________

From: Oliver Weinmann oliver.weinmann@me.com Sent: Monday, August 26, 2024 8:30:32 AM To: openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: Migratekit: near-live migrations from VMware to OpenStack

Hi Mohammed,

We struggled using virt-v2v directly with OpenStack. Our approach was using ovftool to export the VM as OVA and then create a qcow2 image that we can import in OpenStack. So many manual steps involved.

So I tested your tool aganst vsphere 7.x vcenter. But I just can't manage to get it working.

It keeps complaining about the path of the source VM. I can use the following just fine with virt-v2v:

virt-v2v -v -x --password-file pass.txt -ic vpx://user123@vcenter.somedomain.localmailto:user123@vcenter.somedomain.local/GE%Office%20One/Projects/esx01.somedomain.local?no_verify=1 test-vm-01

Using the same in your tool:

docker run -it --rm --privileged \ -v /dev:/devhttp://dev:/dev \ -v /usr/lib64/vmware-vix-disklib/:/usr/lib64/vmware-vix-disklib:ro \ --env-file <(env | grep OS_) registry.atmosphere.dev/library/migratekit:latesthttp://registry.atmosphere.dev/library/migratekit:latest migrate \ --vmware-endpoint vcenter.somedomain.local \ --vmware-username user123 \ --vmware-password XXXXXX \ --vmware-path /GE%Office%20One/Projects/esx01.somedomain.local/test-vm-01

Results in:

Error: vm '/GE%Office%20One/Projects/esx01.somedomain.local/test-vm-01http://esx01.somedomain.local/test-vm-01' not found

I tried many different variants of the path none has worked.

Is there a way to lookup the path of a VM?

I can also use:

virsh -c 'vpx://user123@vcenter.somedomain.localmailto:user123@vcenter.somedomain.local/GE%Office%20One/Projects/esx01.somedomain.local?no_verify=1 list --all

which shows me all VMs running on that particular ESXi host.

Best Regards, Oliver

Hi Mohammend, today we invested quite some time to get it working and we noted a few things. We use kolla-ansible and we figured out that the migratekit needs to connect to the metadata service in openstack. So we deployed a fresh instance of Rocky9 and got stuck again with some nbdcopy error and so we deployed ubuntu 24 with docker and this time we got a lot further. What is strange is that nbdcopy is very slow. On our first try we only had around 40MB/s. Both source and destination are on SSDs with a 10Gb network in between. Something we need to investigate on. We managed to get a copy done and initiated the cutover but got stuck again at: INFO[0000] Setting Disk Bus: virtio INFO[0000] Ensuring OpenStack resources exist INFO[0000] Flavor exists, ensuring network resources exist flavor=flavor_2C_4GB_120GB Error: no network mapping found for MAC address Strange was that after we did the cutover the migrate would again start with a full copy instead of incremental. Cheers, Oliver On 29 Aug 2024, at 17:19, Oliver Weinmann oliver.weinmann@me.com wrote: Hi Mohammed, Many thanks for looking into this so quickly. A colleague has tested your new container, but it would not offer any suggestions for the VM name or path. He looked in your code and figured that you use govc. So he used it to figure out the correct path to the VM: GOVMOMI: (to get the VM-PATH) Download govmomi Binaries or directly install in linux using this executed by root: curl -L -o - "https://github.com/vmware/govmomi/releases/latest/download/govc_$(uname -s)_$(uname -m).tar.gz" | tar -C /usr/local/bin -xvzf - govc logout and login Afterwards set variables accordingly to your credentials and execute this to find the VM-Path for your vm (replace password and <testvm*> accordingly. * is Wildcard) export GOVC_USERNAME="<username>@<domain>" export GOVC_PASSWORD="**********" export GOVC_URL= https://vcenter.local/sdk export GOVC_INSECURE=true export VMPATH="$(govc find . -type m -name <vmname*>)" ##Output: /Datacenter Germany/vm/Projects Cluster/Linux/testvm.local ##Output: /<Datacenter>/vm/<Cluster>/<Folder>/<VM> Feel free to add this to your Readme if you want. It is a life saver. :) We will do some more tests tomorrow and migrate a couple of VMs. Cheers, Oliver Am 26.08.2024 um 18:16 schrieb Mohammed Naser: Hi Oliver, I just pushed up a PR that I tested which lists all VMs if it doesn’t find any VMs: https://github.com/vexxhost/migratekit/pull/10 I’ve merged it and this pushed up a new image that has gotten promoted, so I suggest pulling the latest image (aka `docker pull …`) and trying again. It’ll help you find the right path. 😊 Thanks, Mohammed From: Oliver Weinmann oliver.weinmann@me.com Date: Monday, August 26, 2024 at 9:02 AM To: Mohammed Naser mnaser@vexxhost.com Cc: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: Migratekit: near-live migrations from VMware to OpenStack Hi Mohammed, thanks four your quick reply. Unfortunately this also doesn't work. Cheers, Oliver On 26 Aug 2024, at 14:49, Mohammed Naser mnaser@vexxhost.com wrote: Hi Oliver: Can you add quotes around the path and replace the %20 with spaces and see if that works? A frequent ask is being able to list VMs.. we may just add that feature to make it easier to use. Thanks Mohammed Get Outlook for iOS From: Oliver Weinmann oliver.weinmann@me.com Sent: Monday, August 26, 2024 8:30:32 AM To: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: Migratekit: near-live migrations from VMware to OpenStack Hi Mohammed, We struggled using virt-v2v directly with OpenStack. Our approach was using ovftool to export the VM as OVA and then create a qcow2 image that we can import in OpenStack. So many manual steps involved. So I tested your tool aganst vsphere 7.x vcenter. But I just can't manage to get it working. It keeps complaining about the path of the source VM. I can use the following just fine with virt-v2v: virt-v2v -v -x --password-file pass.txt -ic vpx:// user123@vcenter.somedomain.local /GE%Office%20One/Projects/esx01.somedomain.local?no_verify=1 test-vm-01 Using the same in your tool: docker run -it --rm --privileged \ -v / dev:/dev \ -v /usr/lib64/vmware-vix-disklib/:/usr/lib64/vmware-vix-disklib:ro \ --env-file <(env | grep OS_) registry.atmosphere.dev/library/migratekit:latest migrate \ --vmware-endpoint vcenter.somedomain.local \ --vmware-username user123 \ --vmware-password XXXXXX \ --vmware-path /GE%Office%20One/Projects/esx01.somedomain.local/test-vm-01 Results in: Error: vm '/GE%Office%20One/Projects/ esx01.somedomain.local/test-vm-01 ' not found I tried many different variants of the path none has worked. Is there a way to lookup the path of a VM? I can also use: virsh -c ' vpx:// user123@vcenter.somedomain.local /GE%Office%20One/Projects/esx01.somedomain.local?no_verify=1 list --all which shows me all VMs running on that particular ESXi host. Best Regards, Oliver

Hi Mohammed,

looks like you have added a couple of fixes in the past days. It is working a lot better now. Thanks for that.

We still have some problems with the cutover but that is not a big issue. We simply create an instance from the cloned volume.

Cheers,

Oliver

Am 02.09.2024 um 17:46 schrieb Mohammed Naser:

Hi Oliver,

We’re hoping to try and use `diskimage-builder` to build an appliance instead of a Docker image, because it seems there’s a lot of issues around compatibility that we’re seeing right now.

For nbdcopy, I’m not sure there’s much we can do in there, we’re relying on an external tool to run this, it would be good to double check, I think part of this is because it uses VDDK which pipes all the traffic through vCenter (I think).

For the cutover, I think it restarts with a full copy because VMware CBT resets when the VM restarts (once again, not too familiar, but I think), but the error you’re seeing is because you need to specify the network mapping (which essentially tells what VMware MAC address of interface maps to which OpenStack network):

You can see that documented in the README that explains how to specify the network mapping, with the MAC, OpenStack network ID, etc.  IS it possible you have multiple NICs and you forgot one of them in the defn?

Thanks for all the feedback

Regards,

Mohammed

*From: *Oliver Weinmann oliver.weinmann@me.com *Date: *Monday, September 2, 2024 at 11:43 AM *To: *Mohammed Naser mnaser@vexxhost.com *Cc: *openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org *Subject: *Re: Migratekit: near-live migrations from VMware to OpenStack

Hi Mohammend,

today we invested quite some time to get it working and we noted a few things.

We use kolla-ansible and we figured out that the migratekit needs to connect to the metadata service in openstack. So we deployed a fresh instance of Rocky9 and got stuck again with some nbdcopy error and so we deployed ubuntu 24 with docker and this time we got a lot further.

What is strange is that nbdcopy is very slow. On our first try we only had around 40MB/s. Both source and destination are on SSDs with a 10Gb network in between. Something we need to investigate on.

We managed to get a copy done and initiated the cutover but got stuck again at:

INFO[0000] Setting Disk Bus: virtio

INFO[0000] Ensuring OpenStack resources exist

INFO[0000] Flavor exists, ensuring network resources exist  flavor=flavor_2C_4GB_120GB

Error: no network mapping found for MAC address

Strange was that after we did the cutover the migrate would again start with a full copy instead of incremental.

Cheers,

Oliver

On 29 Aug 2024, at 17:19, Oliver Weinmann <oliver.weinmann@me.com>
wrote:

Hi Mohammed,

Many thanks for looking into this so quickly. A colleague has
tested your new container, but it would not offer any suggestions
for the VM name or path.

He looked in your code and figured that you use govc.

So he used it to figure out the correct path to the VM:

GOVMOMI:

(to get the VM-PATH)

    Download govmomi Binaries or directly install in linux using
this executed by root:

curl -L -o -
"https://github.com/vmware/govmomi/releases/latest/download/govc_$(uname
-s)_$(uname -m).tar.gz"
<https://github.com/vmware/govmomi/releases/latest/download/govc_$(uname-s)_$(uname-m).tar.gz>
| tar -C /usr/local/bin -xvzf - govc

    logout and login

    Afterwards set variables accordingly to your credentials and
execute this to find the VM-Path for your vm (replace password and
<testvm*> accordingly. * is Wildcard)

export GOVC_USERNAME="<username>@<domain>"

export GOVC_PASSWORD="**********"

export GOVC_URL=https://vcenter.local/sdk

export GOVC_INSECURE=true

export VMPATH="$(govc find . -type m -name <vmname*>)"

##Output: /Datacenter Germany/vm/Projects Cluster/Linux/testvm.local

##Output: /<Datacenter>/vm/<Cluster>/<Folder>/<VM>

Feel free to add this to your Readme if you want. It is a life
saver. :)

We will do some more tests tomorrow and migrate a couple of VMs.

Cheers,

Oliver

Am 26.08.2024 um 18:16 schrieb Mohammed Naser:

    Hi Oliver,

    I just pushed up a PR that I tested which lists all VMs if it
    doesn’t find any VMs:

    https://github.com/vexxhost/migratekit/pull/10

    I’ve merged it and this pushed up a new image that has gotten
    promoted, so I suggest pulling the latest image (aka `docker
    pull …`) and trying again.  It’ll help you find the right path. 😊

    Thanks,

    Mohammed

    *From: *Oliver Weinmann <oliver.weinmann@me.com>
    <mailto:oliver.weinmann@me.com>
    *Date: *Monday, August 26, 2024 at 9:02 AM
    *To: *Mohammed Naser <mnaser@vexxhost.com>
    <mailto:mnaser@vexxhost.com>
    *Cc: *openstack-discuss@lists.openstack.org
    <openstack-discuss@lists.openstack.org>
    <mailto:openstack-discuss@lists.openstack.org>
    *Subject: *Re: Migratekit: near-live migrations from VMware to
    OpenStack

    Hi Mohammed,

    thanks four your quick reply.

    Unfortunately this also doesn't work.

    Cheers,

    Oliver

        On 26 Aug 2024, at 14:49, Mohammed Naser
        <mnaser@vexxhost.com> <mailto:mnaser@vexxhost.com> wrote:

        Hi Oliver:

        Can you add quotes around the path and replace the %20
        with spaces and see if that works?

        A frequent ask is being able to list VMs.. we may just add
        that feature to make it easier to use.

        Thanks

        Mohammed

        GetOutlook for iOS <https://aka.ms/o0ukef>

        ------------------------------------------------------------------------

        *From:*Oliver Weinmann <oliver.weinmann@me.com>
        <mailto:oliver.weinmann@me.com>
        *Sent:* Monday, August 26, 2024 8:30:32 AM
        *To:* openstack-discuss@lists.openstack.org
        <openstack-discuss@lists.openstack.org>
        <mailto:openstack-discuss@lists.openstack.org>
        *Subject:* Re: Migratekit: near-live migrations from
        VMware to OpenStack

        Hi Mohammed,

        We struggled using virt-v2v directly with OpenStack. Our
        approach was using ovftool to export the VM as OVA and
        then create a qcow2 image that we can import in OpenStack.
        So many manual steps involved.

        So I tested your tool aganst vsphere 7.x vcenter. But I
        just can't manage to get it working.

        It keeps complaining about the path of the source VM. I
        can use the following just fine with virt-v2v:

        virt-v2v -v -x --password-file pass.txt -ic
        vpx://user123@vcenter.somedomain.local/GE%Office%20One/Projects/esx01.somedomain.local?no_verify=1
        test-vm-01

        Using the same in your tool:

        docker run -it --rm --privileged   \

        -v /dev:/dev <http://dev:/dev> \

        -v
        /usr/lib64/vmware-vix-disklib/:/usr/lib64/vmware-vix-disklib:ro
        \

        --env-file <(env | grep OS_)
        registry.atmosphere.dev/library/migratekit:latest
        <http://registry.atmosphere.dev/library/migratekit:latest>
        migrate   \

        --vmware-endpoint vcenter.somedomain.local   \

        --vmware-username user123   \

        --vmware-password XXXXXX   \

        --vmware-path
        /GE%Office%20One/Projects/esx01.somedomain.local/test-vm-01

        Results in:

        Error: vm
        '/GE%Office%20One/Projects/esx01.somedomain.local/test-vm-01
        <http://esx01.somedomain.local/test-vm-01>' not found

        I tried many different variants of the path none has worked.

        Is there a way to lookup the path of a VM?

        I can also use:

        virsh -c
        'vpx://user123@vcenter.somedomain.local/GE%Office%20One/Projects/esx01.somedomain.local?no_verify=1
        list --all

        which shows me all VMs running on that particular ESXi host.

        Best Regards,

        Oliver

Hi Mohammed,

my colleague is the one who is mostly testing it. I will check with him when I'm back in the office this week.

Have a nice weekend

Cheers,

Oliver

Am 10.09.2024 um 17:22 schrieb Mohammed Naser:

Hi Oliver,

I’d love to have those issues locked down.  Do you know why it’s failing and why you’re having to create a cloned volume?

Thanks,

Mohammed

*From: *Oliver Weinmann oliver.weinmann@me.com *Date: *Tuesday, September 10, 2024 at 11:10 AM *To: *Mohammed Naser mnaser@vexxhost.com *Cc: *openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org *Subject: *Re: Migratekit: near-live migrations from VMware to OpenStack

Hi Mohammed,

looks like you have added a couple of fixes in the past days. It is working a lot better now. Thanks for that.

We still have some problems with the cutover but that is not a big issue. We simply create an instance from the cloned volume.

Cheers,

Oliver

<snip>