Hi all, I have a question. If we have an old openstack deployment (Queens) that have VM's (with & without volumes attached) and we want to migrate those to a new deploy (Victoria) can this be achieved by the owners of the VM's through horizon in some way? We can take snapshots of the vm's that don't have volumes attached, copy them to the new controller and re instantiate them but we haven't found a solution for the volumes yet. This give us more work though as it all needs to be done over cli so the user is taken out of the loop and it falls on us 3 admins to move over 300 VM's. Any advice or info would be greatly appreciated. Thanks in advance. Derek
On Thu, 2021-07-29 at 09:04 +0000, Derek O keeffe wrote:
Hi all, I have a question. If we have an old openstack deployment (Queens) that have VM's (with & without volumes attached) and we want to migrate those to a new deploy (Victoria) can this be achieved by the owners of the VM's through horizon in some way?
no for two reasons. first cold and live migration are admin only so normal teants cannot use it. second whild its technially possibel to mix greate then n+1 version deltas in nova up to victoria or wallaby it may or may not work.
we only test n to n+1 queens to victoria is 5 releases.
you are very lucking in that you queens used version 5.0 of the comput erpc and victoria is 5.12 which means in theory you can pin to victoria to sue 5.0 and it might work.
https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L416-L4...
i not so long ago tested train -> wallaby pinning to 5.3 and it appeared to work but this is well in the depts of undefiend behavior land.
if you upgrade your nova contoler service to victoria and pin the RCP you __might__ be able to manage the queens compute suffently enought to have your admin manually migrate the vms form teh queens host to victoria host but i would not try to do it the other direction.
We can take snapshots of the vm's that don't have volumes attached, copy them to the new controller and re instantiate them but we haven't found a solution for the volumes yet.
i assume you care not willing or are unable to jsut do rolling update between the 5 release instead fo trying to skip strait form queens to victoria. that is the tested and supported way to do this.
This give us more work though as it all needs to be done over cli so the user is taken out of the loop and it falls on us 3 admins to move over 300 VM's.
well your admins will have to move them whatever you decided to do since live migration is admin only unless you have used custom policy to alter that.
Any advice or info would be greatly appreciated. Thanks in advance.
so my main advice would be if you cant jus tdo back to back rulling upgrades keeping your vms active and in place then you need to do a lot of testing. on the neutron/cinder side you need to ensure all nodes use a singel release. nova can handel talking to newer or older neutron/cinder serivces but it cant handle talking to a mixed neutron or cinder well.
in queens neutron does not supprot multiple port binding in victoria it does. if you upgrade neutorn on the contoler to victoria but have the neutron l2 agent on the compute nodes still running queens the neutorn api will report support for multiple port bindings and nova will use the new live migration code path but the queens host wont have supprot for that. in this case you actully can live migrate to a victoria node but if anythign goes wrong it wont be able to revert as reactivignt the port binding on the souce node will fail.
so if your going to try and mix version like this you shoudl avoid mixing neutron verison. the same is likely ture of cinder, all of the openstack serice only offcially support n to n+1 upgrades so you want to avoid breaking that in all cases where you can avoid it. if you must do it for nova for some reason you should only upgrade the nova serices and leave the rest of the cloud on queens or upgrade everythin but nova to victoria fully.
even followign that advice its not guarentted to work
as i have said i have tested walably contolers with trian computes recently and i was able to live migrate from train to wallaby with a minor code chagne but that was with the simplest possibe vm with no cinder no cpu pinning or hugepages or any other complex features. i did break part of my advice by keeping the train version of neutorn l2 agent on the train comptue node but that was due to devstack limitations. you might be forced to do the same but the multiple port bidnings freature was added in rocky so in my case but side had it and it was not an issue for you your queens ndoes will not support ti but your victoria nodes will so if you upgrade neutorn on the contoler it will make live migraiton more risky as you wont be able to revert. the vm will have to be anually fix if the migration fails for any once it starts.
Derek
(re-sent from a correct address for openstack-discuss :) )
Hello Derek,
On Thu, Jul 29, 2021 at 9:08 AM Derek O keeffe derekokeeffe85@yahoo.ie wrote:
Hi all,
I have a question. If we have an old openstack deployment (Queens) that have VM's (with & without volumes attached) and we want to migrate those to a new deploy (Victoria) can this be achieved by the owners of the VM's through horizon in some way?
We can take snapshots of the vm's that don't have volumes attached, copy them to the new controller and re instantiate them but we haven't found a solution for the volumes yet. This give us more work though as it all needs to be done over cli so the user is taken out of the loop and it falls on us 3 admins to move over 300 VM's.
Any advice or info would be greatly appreciated. Thanks in advance.
OS Migrate might be worth a look, it's an Ansible collection. It does cold migration that does not require admin privileges (is runnable by tenants who are sufficiently technically savvy). It can copy VMs with attached volumes. If boot disk copy is desired, then the destination instance will be created as boot-from-volume (the alternative option being re-imaging the instance from a pre-existing Glance image in the destination cloud, which is good for instances which store important data only in volumes, and can be rebuilt as in `openstack server rebuild`). Aside from migrating VMs, it should also help you migrate prerequisites for VMs, e.g. tenant networking resources or security groups.
Perhaps even if you end up doing the migration on behalf of tenants, at least you wouldn't have to do it fully by hand, and you'd have something for the migration of attached volumes.
Here's an "executive summary" of how it works:
https://www.jistr.com/blog/2021-07-12-introduction-to-os-migrate/
And docs:
https://os-migrate.github.io/os-migrate/
Have a good day,
Jirka
On Thu, 2021-07-29 at 15:28 +0000, Jiri Stransky wrote:
(re-sent from a correct address for openstack-discuss :) )
Hello Derek,
On Thu, Jul 29, 2021 at 9:08 AM Derek O keeffe derekokeeffe85@yahoo.ie wrote:
Hi all,
I have a question. If we have an old openstack deployment (Queens) that have VM's (with & without volumes attached) and we want to migrate those to a new deploy (Victoria) can this be achieved by the owners of the VM's through horizon in some way?
We can take snapshots of the vm's that don't have volumes attached, copy them to the new controller and re instantiate them but we haven't found a solution for the volumes yet. This give us more work though as it all needs to be done over cli so the user is taken out of the loop and it falls on us 3 admins to move over 300 VM's.
Any advice or info would be greatly appreciated. Thanks in advance.
OS Migrate might be worth a look, it's an Ansible collection. It does cold migration that does not require admin privileges (is runnable by tenants who are sufficiently technically savvy).
default nova policy requrie admin right for cold migration https://github.com/openstack/nova/blob/master/nova/policies/migrate_server.p...
os-migrate is not useing nova migration apis its migrating the data and vms behind novas back.
It can copy VMs with attached volumes. If boot disk copy is desired, then the destination instance will be created as boot-from-volume (the alternative option being re-imaging the instance from a pre-existing Glance image in the destination cloud, which is good for instances which store important data only in volumes, and can be rebuilt as in `openstack server rebuild`). Aside from migrating VMs, it should also help you migrate prerequisites for VMs, e.g. tenant networking resources or security groups.
Perhaps even if you end up doing the migration on behalf of tenants, at least you wouldn't have to do it fully by hand, and you'd have something for the migration of attached volumes.
os-migrate is a tool for moveing betweeen different cloud deployemnets isntead of doing a upgrade you deploy a parralel cloud and then you use os-migrate to move workloads.
while that sound nice in practic it will break any automatin that site on top of openstack ans the workload will not be moved to a new cloud so so if you have deployed yoru vms with heat or use an external VNFM you willl nolonger be abel to manage the vms.
os migrate will not be transparent to the enduser for example there keystone project id and user id will change. this is because keysotne does not allow you to choose a uuid when create user or projects https://docs.openstack.org/api-ref/identity/v3/index.html?expanded=create-us... https://docs.openstack.org/api-ref/identity/v3/index.html?expanded=create-pr...
while some resouces can be recarted with the same external ids resouces like flaover obviously need admin privladges. nova lso caches the version of the imave and flavor the server was booted with.
if you have modifed the flaovr or image extra spces since the vms were booted the vms that are create on the destination may be diffeente then the souce and that may or may not break the newly created vm.
so while it might be useful in limited cases you shoudl be aware the way it works is generally not supprotable in public cloud or by normal users its a tool that a cloud operator can use.
as a reulst it is portentially more dangourous then just a normal upgades as its more like that the vm abi and resouce uuids will change. im also not sure if it will work with ironic i suspect not.
it also cannot handel all resouces for all service for example it cannot tansfer octaivia loadblanceser, barbical secreats, swift objecst or trove db instance or heat stacks to name but a few service that are not supported.
if you can live with those limitation then os-migrate might eb the tool for you but i would personally upgrade in place.
Here's an "executive summary" of how it works:
https://www.jistr.com/blog/2021-07-12-introduction-to-os-migrate/
And docs:
https://os-migrate.github.io/os-migrate/
Have a good day,
Jirka
Hello folks,
On Thu, Jul 29, 2021 at 6:24 PM Sean Mooney smooney@redhat.com wrote:
On Thu, 2021-07-29 at 15:28 +0000, Jiri Stransky wrote:
OS Migrate might be worth a look, it's an Ansible collection. It does cold migration that does not require admin privileges (is runnable by tenants who are sufficiently technically savvy).
default nova policy requrie admin right for cold migration https://github.com/openstack/nova/blob/master/nova/policies/migrate_server.p...
os-migrate is not useing nova migration apis its migrating the data and vms behind novas back.
I used the term "cold migration" in the generic sense (migration of VMs while they're stopped), not in the particular "Nova cold migration" sense. While we're not using any migration-specific APIs (not sure if these would work *between* clouds, incl. copying Cinder volumes), we do still interact with OpenStack only through the normal REST APIs, we do not touch the backing MariaDB database directly. (So i do not see our approach as migrating "behind Nova's back".)
It can copy VMs with attached volumes. If boot disk copy is desired, then the destination instance will be created as boot-from-volume (the alternative option being re-imaging the instance from a pre-existing Glance image in the destination cloud, which is good for instances which store important data only in volumes, and can be rebuilt as in `openstack server rebuild`). Aside from migrating VMs, it should also help you migrate prerequisites for VMs, e.g. tenant networking resources or security groups.
Perhaps even if you end up doing the migration on behalf of tenants, at least you wouldn't have to do it fully by hand, and you'd have something for the migration of attached volumes.
os-migrate is a tool for moveing betweeen different cloud deployemnets isntead of doing a upgrade you deploy a parralel cloud and then you use os-migrate to move workloads.
+1.
while that sound nice in practic it will break any automatin that site on top of openstack ans the workload will not be moved to a new cloud so so if you have deployed yoru vms with heat or use an external VNFM you willl nolonger be abel to manage the vms.
The data is copied to destination and a new VM is created using the data copy (assuming the two different migration modes i described in previous e-mails and in the blog). Floating IP addresses will also differ in the destination since generally different clouds have different FIP ranges. Whether this is a suitable approach or not depends per use case expectations and constraints.
os migrate will not be transparent to the enduser for example there keystone project id and user id will change. this is because keysotne does not allow you to choose a uuid when create user or projects https://docs.openstack.org/api-ref/identity/v3/index.html?expanded=create-us... https://docs.openstack.org/api-ref/identity/v3/index.html?expanded=create-pr...
Indeed as we're recreating the resources using APIs rather than a direct DB access, IDs in the destination are different, but on the other hand going trough the normal REST APIs (rather than trying to force IDs by DB editing for example) considerably lowers the risk of creating inconsistent state of OpenStack services. Again, whether this is suitable or not depends per use case. I would say the migration procedure is transparent to the user (e.g. the resources are serialized into readable and editable YAML files, which are then used when creating resources in destination).
while some resouces can be recarted with the same external ids resouces like flaover obviously need admin privladges. nova lso caches the version of the imave and flavor the server was booted with.
if you have modifed the flaovr or image extra spces since the vms were booted the vms that are create on the destination may be diffeente then the souce and that may or may not break the newly created vm.
Indeed this is something to watch out for if the src/dst flavors or images happen to differ somehow. I would not say differences are entirely disqualifying the migration, but care needs to be taken that workloads' requirements are still satisfied.
so while it might be useful in limited cases you shoudl be aware the way it works is generally not supprotable in public cloud or by normal users its a tool that a cloud operator can use.
As i wrote earlier, the user needs to be somewhat technically savvy (most importantly, be able to work with Ansible and have at least high-level understanding of OpenStack), but admin privileges are not required unless you are trying to migrate resources that only admins can create (public images, users, projects, flavors, ...). We tested it with a cloud where we didn't have admin access.
as a reulst it is portentially more dangourous then just a normal upgades as its more like that the vm abi and resouce uuids will change.
Upgrade is less invasive on the level of individual resources, but migration allows you to scope risk better. In other words: when upgrading, the entire cloud is at some level of risk. When migrating tenant content with tenant credentials, only one tenant in the source and one tenant in the destination are being affected at a given time. Which approach is considered less dangerous will depend on use cases.
im also not sure if it will work with ironic i suspect not.
I would hope at least the re-imaging mode of migration would work, but we did not test migrating instances deployed on Ironic nodes, so i'm also not sure.
it also cannot handel all resouces for all service for example it cannot tansfer octaivia loadblanceser, barbical secreats, swift objecst or trove db instance or heat stacks to name but a few service that are not supported.
The supported resources are: users, projects, networks, subnets, routers, security groups, images, nova flavors and key pairs, VMs + attached volumes + FIP creation.
Have a good day,
Jirka
On Tue, 2021-08-03 at 16:49 +0000, Jiri Stransky wrote:
Hello folks,
On Thu, Jul 29, 2021 at 6:24 PM Sean Mooney smooney@redhat.com wrote:
On Thu, 2021-07-29 at 15:28 +0000, Jiri Stransky wrote:
OS Migrate might be worth a look, it's an Ansible collection. It does cold migration that does not require admin privileges (is runnable by tenants who are sufficiently technically savvy).
default nova policy requrie admin right for cold migration https://github.com/openstack/nova/blob/master/nova/policies/migrate_server.p...
os-migrate is not useing nova migration apis its migrating the data and vms behind novas back.
I used the term "cold migration" in the generic sense (migration of VMs while they're stopped), not in the particular "Nova cold migration" sense. While we're not using any migration-specific APIs (not sure if these would work *between* clouds, incl. copying Cinder volumes), we do still interact with OpenStack only through the normal REST APIs, we do not touch the backing MariaDB database directly. (So i do not see our approach as migrating "behind Nova's back".)
do you run any process on the host or interact with any files on the host or libvirt/qemu in any way with the data migrator? if you are just using snapshots and grabign datat form glance or doing data copeis with tools in the vms i retract my statement but otherwise any interaction with the nova instance directly outside of the nova api would be out of band and behind novas back.
It can copy VMs with attached volumes. If boot disk copy is desired, then the destination instance will be created as boot-from-volume (the alternative option being re-imaging the instance from a pre-existing Glance image in the destination cloud, which is good for instances which store important data only in volumes, and can be rebuilt as in `openstack server rebuild`). Aside from migrating VMs, it should also help you migrate prerequisites for VMs, e.g. tenant networking resources or security groups.
Perhaps even if you end up doing the migration on behalf of tenants, at least you wouldn't have to do it fully by hand, and you'd have something for the migration of attached volumes.
os-migrate is a tool for moveing betweeen different cloud deployemnets isntead of doing a upgrade you deploy a parralel cloud and then you use os-migrate to move workloads.
+1.
while that sound nice in practic it will break any automatin that site on top of openstack ans the workload will not be moved to a new cloud so so if you have deployed yoru vms with heat or use an external VNFM you willl nolonger be abel to manage the vms.
The data is copied to destination and a new VM is created using the data copy (assuming the two different migration modes i described in previous e-mails and in the blog). Floating IP addresses will also differ in the destination since generally different clouds have different FIP ranges. Whether this is a suitable approach or not depends per use case expectations and constraints.
os migrate will not be transparent to the enduser for example there keystone project id and user id will change. this is because keysotne does not allow you to choose a uuid when create user or projects https://docs.openstack.org/api-ref/identity/v3/index.html?expanded=create-us... https://docs.openstack.org/api-ref/identity/v3/index.html?expanded=create-pr...
Indeed as we're recreating the resources using APIs rather than a direct DB access, IDs in the destination are different, but on the other hand going trough the normal REST APIs (rather than trying to force IDs by DB editing for example) considerably lowers the risk of creating inconsistent state of OpenStack services. Again, whether this is suitable or not depends per use case. I would say the migration procedure is transparent to the user (e.g. the resources are serialized into readable and editable YAML files, which are then used when creating resources in destination).
while some resouces can be recarted with the same external ids resouces like flaover obviously need admin privladges. nova lso caches the version of the imave and flavor the server was booted with.
if you have modifed the flaovr or image extra spces since the vms were booted the vms that are create on the destination may be diffeente then the souce and that may or may not break the newly created vm.
Indeed this is something to watch out for if the src/dst flavors or images happen to differ somehow. I would not say differences are entirely disqualifying the migration, but care needs to be taken that workloads' requirements are still satisfied.
so while it might be useful in limited cases you shoudl be aware the way it works is generally not supprotable in public cloud or by normal users its a tool that a cloud operator can use.
As i wrote earlier, the user needs to be somewhat technically savvy (most importantly, be able to work with Ansible and have at least high-level understanding of OpenStack), but admin privileges are not required unless you are trying to migrate resources that only admins can create (public images, users, projects, flavors, ...). We tested it with a cloud where we didn't have admin access.
as a reulst it is portentially more dangourous then just a normal upgades as its more like that the vm abi and resouce uuids will change.
Upgrade is less invasive on the level of individual resources, but migration allows you to scope risk better. In other words: when upgrading, the entire cloud is at some level of risk. When migrating tenant content with tenant credentials, only one tenant in the source and one tenant in the destination are being affected at a given time. Which approach is considered less dangerous will depend on use cases.
im also not sure if it will work with ironic i suspect not.
I would hope at least the re-imaging mode of migration would work, but we did not test migrating instances deployed on Ironic nodes, so i'm also not sure.
it also cannot handel all resouces for all service for example it cannot tansfer octaivia loadblanceser, barbical secreats, swift objecst or trove db instance or heat stacks to name but a few service that are not supported.
The supported resources are: users, projects, networks, subnets, routers, security groups, images, nova flavors and key pairs, VMs + attached volumes + FIP creation.
Have a good day,
Jirka
On Tue, Aug 3, 2021 at 5:04 PM Sean Mooney smooney@redhat.com wrote:
On Tue, 2021-08-03 at 16:49 +0000, Jiri Stransky wrote:
Hello folks,
On Thu, Jul 29, 2021 at 6:24 PM Sean Mooney smooney@redhat.com wrote:
On Thu, 2021-07-29 at 15:28 +0000, Jiri Stransky wrote:
OS Migrate might be worth a look, it's an Ansible collection. It does cold migration that does not require admin privileges (is runnable by tenants who are sufficiently technically savvy).
default nova policy requrie admin right for cold migration https://github.com/openstack/nova/blob/master/nova/policies/migrate_server.p...
os-migrate is not useing nova migration apis its migrating the data and vms behind novas back.
I used the term "cold migration" in the generic sense (migration of VMs while they're stopped), not in the particular "Nova cold migration" sense. While we're not using any migration-specific APIs (not sure if these would work *between* clouds, incl. copying Cinder volumes), we do still interact with OpenStack only through the normal REST APIs, we do not touch the backing MariaDB database directly. (So i do not see our approach as migrating "behind Nova's back".)
do you run any process on the host or interact with any files on the host or libvirt/qemu in any way with the data migrator? if you are just using snapshots and grabign datat form glance or doing data copeis with tools in the vms i retract my statement but otherwise any interaction with the nova instance directly outside of the nova api would be out of band and behind novas back.
We do not run any process on the compute hosts. OS Migrate workload migration must work without admin access, which also means no access to the compute hosts.
There are tenant-owned VMs which facilitate direct data copying between the clouds, which we call "conversion hosts", and they do one thing on top of a pure copy: sparsification using virt-sparsify before the copying starts. This significantly speeds up copying of "empty space", and is only done on partitions where virt-sparsify can recognize a filesystem that can be sparsified. (We should probably have an option to disable this behavior, in case the user would be interested in a pure copy including e.g. leftovers of deleted files which are considered inaccessible empty space by the filesystem.)
Have a good day,
Jirka
On Wed, 2021-08-04 at 08:45 +0000, Jiri Stransky wrote:
On Tue, Aug 3, 2021 at 5:04 PM Sean Mooney smooney@redhat.com wrote:
On Tue, 2021-08-03 at 16:49 +0000, Jiri Stransky wrote:
Hello folks,
On Thu, Jul 29, 2021 at 6:24 PM Sean Mooney smooney@redhat.com wrote:
On Thu, 2021-07-29 at 15:28 +0000, Jiri Stransky wrote:
OS Migrate might be worth a look, it's an Ansible collection. It does cold migration that does not require admin privileges (is runnable by tenants who are sufficiently technically savvy).
default nova policy requrie admin right for cold migration https://github.com/openstack/nova/blob/master/nova/policies/migrate_server.p...
os-migrate is not useing nova migration apis its migrating the data and vms behind novas back.
I used the term "cold migration" in the generic sense (migration of VMs while they're stopped), not in the particular "Nova cold migration" sense. While we're not using any migration-specific APIs (not sure if these would work *between* clouds, incl. copying Cinder volumes), we do still interact with OpenStack only through the normal REST APIs, we do not touch the backing MariaDB database directly. (So i do not see our approach as migrating "behind Nova's back".)
do you run any process on the host or interact with any files on the host or libvirt/qemu in any way with the data migrator? if you are just using snapshots and grabign datat form glance or doing data copeis with tools in the vms i retract my statement but otherwise any interaction with the nova instance directly outside of the nova api would be out of band and behind novas back.
We do not run any process on the compute hosts. OS Migrate workload migration must work without admin access, which also means no access to the compute hosts.
There are tenant-owned VMs which facilitate direct data copying between the clouds, which we call "conversion hosts", and they do one thing on top of a pure copy: sparsification using virt-sparsify before the copying starts. This significantly speeds up copying of "empty space", and is only done on partitions where virt-sparsify can recognize a filesystem that can be sparsified. (We should probably have an option to disable this behavior, in case the user would be interested in a pure copy including e.g. leftovers of deleted files which are considered inaccessible empty space by the filesystem.)
ah ok that is much more supportable then in a production cloud. thanks for explaining. itought the data mover vm was basically connecting to the host to copy the qcows directly im glad to hear it is not.
Have a good day,
Jirka
participants (3)
-
Derek O keeffe
-
Jiri Stransky
-
Sean Mooney