[ops][nova][victoria] Power State = Suspended?
All; I had something unusual happen this morning; one of my VMs was showing "Suspended" under the Power State in the Horizon dashboard. I've never seen that. What does it mean? Any search that I do points me to a bunch of resources for Status Suspended. Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
All;
I had something unusual happen this morning; one of my VMs was showing "Suspended" under the Power State in the Horizon dashboard.
I've never seen that. What does it mean?
Any search that I do points me to a bunch of resources for Status Suspended. suspened is like hibernate in windows. in the libvirt driver we call libvirt managed_save api
On Wed, 2021-08-04 at 16:39 +0000, DHilsbos@performair.com wrote: this pauses the guests, snapshots the guest ram and saves it to disk then stops the instance. so this frees the guest ram on the host and save it to a file so that we can recreate the vm and resume it as if nothing happened. nova also supports pause. pause is similar to suspend but much less invasive. pause jsut stops the execution of the guest cpus but the guest qemu process is still running. if you have pci passtough device when you use suspend the device are detach before its suspended and then reattached when its resumed. if you use pause we dont need to do this since the dma regoins for the passthough devices are never touched. different virt dirver may or may not supprot pause/suspend and they may work differently. suspeedn is bascally supend the guest execution to disk where as pause is like you laptos sleep or suspend to ram. in a could env i dont really think either are that useful if im being honest and im not sure we woudl add them today if they did not really exits, generally if the vm is not running you should shleve it. from a schduler point of view pausign, suspending and stoping an insntce all use the same amount of resouces as a runnign instnace so form a billing perpsective public cloud will change you the same for all 4 states. shelve instance are only using disk space and floating ips so genrealy that will be reflected in the cost and it will be much cheaper on a public cloud to shelve an instnace.
Thank you,
Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
All;
I had something unusual happen this morning; one of my VMs was showing "Suspended" under the Power State in the Horizon dashboard.
I've never seen that. What does it mean?
Any search that I do points me to a bunch of resources for Status Suspended. suspened is like hibernate in windows. in the libvirt driver we call libvirt managed_save api
Sean; Thank you. Again, this was not Status. The VM was not suspended through OpenStack. Suspended showed up in the Power State. Normally, if I suspend a VM through OpenStack, Status = Suspended, Power State = Shut Down. In this case Status = Active, Power State = Supsended. Is it possible that the Windows OS inside the VM went into a Suspended state, and libvirt / kvm / qemu recognized that? Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Wednesday, August 4, 2021 10:37 AM To: Dominic Hilsbos; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova][victoria] Power State = Suspended? On Wed, 2021-08-04 at 16:39 +0000, DHilsbos@performair.com wrote: this pauses the guests, snapshots the guest ram and saves it to disk then stops the instance. so this frees the guest ram on the host and save it to a file so that we can recreate the vm and resume it as if nothing happened. nova also supports pause. pause is similar to suspend but much less invasive. pause jsut stops the execution of the guest cpus but the guest qemu process is still running. if you have pci passtough device when you use suspend the device are detach before its suspended and then reattached when its resumed. if you use pause we dont need to do this since the dma regoins for the passthough devices are never touched. different virt dirver may or may not supprot pause/suspend and they may work differently. suspeedn is bascally supend the guest execution to disk where as pause is like you laptos sleep or suspend to ram. in a could env i dont really think either are that useful if im being honest and im not sure we woudl add them today if they did not really exits, generally if the vm is not running you should shleve it. from a schduler point of view pausign, suspending and stoping an insntce all use the same amount of resouces as a runnign instnace so form a billing perpsective public cloud will change you the same for all 4 states. shelve instance are only using disk space and floating ips so genrealy that will be reflected in the cost and it will be much cheaper on a public cloud to shelve an instnace.
Thank you,
Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
All;
I had something unusual happen this morning; one of my VMs was showing "Suspended" under the Power State in the Horizon dashboard.
I've never seen that. What does it mean?
Any search that I do points me to a bunch of resources for Status Suspended. suspened is like hibernate in windows. in the libvirt driver we call libvirt managed_save api
All; Like with the network issue conversation that I disappeared from, I apologize for disappearing from this conversation. I've pretty much confirmed that this issue is caused by the Windows 10 guest operating system going into / trying to go into a sleep state. Using the exact same image, if I leave sleep enabled, this un-recoverable state is guaranteed. If I disable sleep, I haven't yet run into this problem. Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: DHilsbos@performair.com [mailto:DHilsbos@performair.com] Sent: Wednesday, August 4, 2021 10:48 AM To: openstack-discuss@lists.openstack.org Cc: smooney@redhat.com Subject: RE: [ops][nova][victoria] Power State = Suspended? Sean; Thank you. Again, this was not Status. The VM was not suspended through OpenStack. Suspended showed up in the Power State. Normally, if I suspend a VM through OpenStack, Status = Suspended, Power State = Shut Down. In this case Status = Active, Power State = Supsended. Is it possible that the Windows OS inside the VM went into a Suspended state, and libvirt / kvm / qemu recognized that? Thank you, Dominic L. Hilsbos, MBA Vice President – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Wednesday, August 4, 2021 10:37 AM To: Dominic Hilsbos; openstack-discuss@lists.openstack.org Subject: Re: [ops][nova][victoria] Power State = Suspended? On Wed, 2021-08-04 at 16:39 +0000, DHilsbos@performair.com wrote: this pauses the guests, snapshots the guest ram and saves it to disk then stops the instance. so this frees the guest ram on the host and save it to a file so that we can recreate the vm and resume it as if nothing happened. nova also supports pause. pause is similar to suspend but much less invasive. pause jsut stops the execution of the guest cpus but the guest qemu process is still running. if you have pci passtough device when you use suspend the device are detach before its suspended and then reattached when its resumed. if you use pause we dont need to do this since the dma regoins for the passthough devices are never touched. different virt dirver may or may not supprot pause/suspend and they may work differently. suspeedn is bascally supend the guest execution to disk where as pause is like you laptos sleep or suspend to ram. in a could env i dont really think either are that useful if im being honest and im not sure we woudl add them today if they did not really exits, generally if the vm is not running you should shleve it. from a schduler point of view pausign, suspending and stoping an insntce all use the same amount of resouces as a runnign instnace so form a billing perpsective public cloud will change you the same for all 4 states. shelve instance are only using disk space and floating ips so genrealy that will be reflected in the cost and it will be much cheaper on a public cloud to shelve an instnace.
Thank you,
Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
Hey there, On 04/08/2021 19:37, Sean Mooney wrote:
I had something unusual happen this morning; one of my VMs was showing "Suspended" under the Power State in the Horizon dashboard.
I've never seen that. What does it mean?
Any search that I do points me to a bunch of resources for Status Suspended. suspened is like hibernate in windows. in the libvirt driver we call libvirt managed_save api this pauses the guests, snapshots the guest ram and saves it to disk then stops the instance. so this frees the guest ram on the host and save it to a file so that we can recreate the vm and resume it as if nothing happened.
Sorry to hijack such an old thread. Looking into these features, I was just wondering if it was possible to: 1) Disable the support for pause / suspend altogether and not allow anyone to place instances in such states? 2) Change the storage location of the saved guest RAM to a shared storage to allow the instance to be migrated while being suspended/paused. As far as I can see currently this data is saved on the host disk. Regards Christian
On Thu, Jan 20, 2022 at 8:20 AM Christian Rohmann < christian.rohmann@inovex.de> wrote:
Hey there,
On 04/08/2021 19:37, Sean Mooney wrote:
I had something unusual happen this morning; one of my VMs was showing "Suspended" under the Power State in the Horizon dashboard.
I've never seen that. What does it mean?
Any search that I do points me to a bunch of resources for Status Suspended.
suspened is like hibernate in windows. in the libvirt driver we call libvirt managed_save api this pauses the guests, snapshots the guest ram and saves it to disk then stops the instance. so this frees the guest ram on the host and save it to a file so that we can recreate the vm and resume it as if nothing happened.
Sorry to hijack such an old thread. Looking into these features, I was just wondering if it was possible to:
1) Disable the support for pause / suspend altogether and not allow anyone to place instances in such states?
you can use policy to disable suspending vms via the api
2) Change the storage location of the saved guest RAM to a shared storage to allow the instance to be migrated while being suspended/paused. As far as I can see currently this data is saved on the host disk.
you can mount the path where things get saved at where ever you want (I think it’s somewhere inside /var/lib/nova/instances)
Regards
Christian
--
Mohammed Naser VEXXHOST, Inc.
Hey Mohammed, thanks for the input! On 21/01/2022 12:48, Mohammed Naser wrote:
Sorry to hijack such an old thread. Looking into these features, I was just wondering if it was possible to:
1) Disable the support for pause / suspend altogether and not allow anyone to place instances in such states?
you can use policy to disable suspending vms via the api
Good point, thanks.
2) Change the storage location of the saved guest RAM to a shared storage to allow the instance to be migrated while being suspended/paused. As far as I can see currently this data is saved on the host disk.
you can mount the path where things get saved at where ever you want (I think it’s somewhere inside /var/lib/nova/instances)
That's true, but this would require some multi-mountable shared storage like CephFS or some NFS to remove the dependency from a single node. It's not like Nova would store this data as e.g. a RBD image in Ceph via some config option, right? Regards Christian
On Mon, 2022-01-24 at 10:59 +0100, Christian Rohmann wrote:
Hey Mohammed,
thanks for the input!
On 21/01/2022 12:48, Mohammed Naser wrote:
Sorry to hijack such an old thread. Looking into these features, I was just wondering if it was possible to:
1) Disable the support for pause / suspend altogether and not allow anyone to place instances in such states?
you can use policy to disable suspending vms via the api
Good point, thanks.
2) Change the storage location of the saved guest RAM to a shared storage to allow the instance to be migrated while being suspended/paused. As far as I can see currently this data is saved on the host disk.
you can mount the path where things get saved at where ever you want (I think it’s somewhere inside /var/lib/nova/instances)
That's true, but this would require some multi-mountable shared storage like CephFS or some NFS to remove the dependency from a single node. It's not like Nova would store this data as e.g. a RBD image in Ceph via some config option, right?
the instance state dir contence no. so wehn you use the rbd images backend and you suspend the ram is saved to disk on the local system i could see adding a feature to nova to poteally upload that as an addtional rbd image to ceph. or maybe store it itn swift or something like that but nova cannot do it today. you could certenly put the instance state dir on a cephfs share or nfs we dont realy like dealing with shared storage for the state dir but the generic code we have for nfs should work with cephfs. just be aware de dont currently test that but i dont see why we could not extned the ceph job to mount /var/lib/nova on cephfs to get coverage in the ci. in the short term i agree that using a cephfs mount is likely the best way to avoid the guest ram from beign stored on the comptue node disk. in terms fo disabling suspend the simplelst way to do that is to alter the policy.json and make that either admin only or preferably require a new role then just dont give that role to your tenants.
Regards
Christian
On Mon, 2022-01-24 at 13:44 +0000, Sean Mooney wrote:
On Mon, 2022-01-24 at 10:59 +0100, Christian Rohmann wrote:
Hey Mohammed,
thanks for the input!
On 21/01/2022 12:48, Mohammed Naser wrote:
Sorry to hijack such an old thread. Looking into these features, I was just wondering if it was possible to:
1) Disable the support for pause / suspend altogether and not allow anyone to place instances in such states?
you can use policy to disable suspending vms via the api
Good point, thanks.
2) Change the storage location of the saved guest RAM to a shared storage to allow the instance to be migrated while being suspended/paused. As far as I can see currently this data is saved on the host disk.
you can mount the path where things get saved at where ever you want (I think it’s somewhere inside /var/lib/nova/instances)
That's true, but this would require some multi-mountable shared storage like CephFS or some NFS to remove the dependency from a single node. It's not like Nova would store this data as e.g. a RBD image in Ceph via some config option, right?
the instance state dir contence no. so wehn you use the rbd images backend and you suspend the ram is saved to disk on the local system i could see adding a feature to nova to poteally upload that as an addtional rbd image to ceph. or maybe store it itn swift or something like that but nova cannot do it today.
you could certenly put the instance state dir on a cephfs share or nfs we dont realy like dealing with shared storage for the state dir but the generic code we have for nfs should work with cephfs. just be aware de dont currently test that but i dont see why we could not extned the ceph job to mount /var/lib/nova on cephfs to get coverage in the ci.
in the short term i agree that using a cephfs mount is likely the best way to avoid the guest ram from beign stored on the comptue node disk. in terms fo disabling suspend the simplelst way to do that is to alter the policy.json and make that either admin only or preferably require a new role then just dont give that role to your tenants.
the polices in question are defined here https://github.com/openstack/nova/blob/master/nova/policies/suspend_server.p... so you woudl override os_compute_api:os-suspend-server:suspend with check_str='is_admin:True' or check_str='project_id:%(project_id)s and role:suspend' it really depends on if you want to still allow suspend for some user or just admins if you want to block even admins then set check_str='!' that will alwasys reject the request.
Regards
Christian
participants (4)
-
Christian Rohmann
-
DHilsbos＠performair.com
-
Mohammed Naser
-
Sean Mooney