[kolla] Clean shutdown for guest VMs when host shutdown
Hi, I'm about ready to go into production with kolla, but one small issue remains. On regular Openstack deployments, I would use the libvirt-guests service to ensure that VMs on hosts always shutdown cleanly when the host shuts down. This is to prevent database issues which may happen on guests during forced, unclean shutdowns. At first glance, Kolla doesn't seem to install this service or anything similar that would ensure clean shutdown of guests when the host shuts down. I have found in the github source, in the tools folder, a systemd script that is supposed to do that, but it tries to execute /usr/libexec/libvirt-guests.sh shutdown in the libvirt container, which doesn't exist there. Does Kolla already ensure clean shutdown automatically, or is there a script I'm missing that could make this work?
On Tue, 6 Jul 2021 at 23:18, J-P Methot <jp.methot@planethoster.info> wrote:
Hi,
I'm about ready to go into production with kolla, but one small issue remains. On regular Openstack deployments, I would use the libvirt-guests service to ensure that VMs on hosts always shutdown cleanly when the host shuts down. This is to prevent database issues which may happen on guests during forced, unclean shutdowns.
At first glance, Kolla doesn't seem to install this service or anything similar that would ensure clean shutdown of guests when the host shuts down. I have found in the github source, in the tools folder, a systemd script that is supposed to do that, but it tries to execute /usr/libexec/libvirt-guests.sh shutdown in the libvirt container, which doesn't exist there.
Does Kolla already ensure clean shutdown automatically, or is there a script I'm missing that could make this work?
Hi J-P Methot, I'm aware of the scripts you've mentioned, but haven't tried them. I think in general we would migrate workloads away from compute nodes before rebooting. Thanks, Mark
On Tue, 6 Jul 2021 at 23:18, J-P Methot <jp.methot@planethoster.info> wrote:
Hi,
I'm about ready to go into production with kolla, but one small issue remains. On regular Openstack deployments, I would use the libvirt-guests service to ensure that VMs on hosts always shutdown cleanly when the host shuts down. This is to prevent database issues which may happen on guests during forced, unclean shutdowns.
At first glance, Kolla doesn't seem to install this service or anything similar that would ensure clean shutdown of guests when the host shuts down. I have found in the github source, in the tools folder, a systemd script that is supposed to do that, but it tries to execute /usr/libexec/libvirt-guests.sh shutdown in the libvirt container, which doesn't exist there.
Does Kolla already ensure clean shutdown automatically, or is there a script I'm missing that could make this work?
Hi J-P Methot,
I'm aware of the scripts you've mentioned, but haven't tried them. I think in general we would migrate workloads away from compute nodes before rebooting.
On Thu, 2021-07-08 at 09:18 +0100, Mark Goddard wrote: thats pretty hevey handed. live migration is not an operation that should be used lightly. there are 0 SLAs for geust performance, packet loss or similar side effect of live migration. live migration should almost never be used for hevily loaded guest. having the ablity to ues the libvirt-guest service would be quite useful when people can take the outage of the vm or where the vm cannot be live migrated. guests with vGPU for example cant be live migrated guest with 512GB of persitent memory can technically be migrated but it will take 10s or minutes unless you have a very fast network connection. i actully tough this was already shiped with kolla-ansible but maybe not? i had it on my home cloud but it may have com in when in from a different package i installed on the host.
Thanks, Mark
On 7/8/21 6:21 AM, Sean Mooney wrote:
i actully tough this was already shiped with kolla-ansible but maybe not? i had it on my home cloud but it may have com in when in from a different package i installed on the host.
There's definitely something to make libvirt-guests work with Kolla on github, but the needed script file appears to be missing from the kolla container image, at least on Ubuntu. Could it be an oversight or a regression? -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
On Thu, 2021-07-08 at 13:58 -0400, J-P Methot wrote:
On 7/8/21 6:21 AM, Sean Mooney wrote:
i actully tough this was already shiped with kolla-ansible but maybe not? i had it on my home cloud but it may have com in when in from a different package i installed on the host.
There's definitely something to make libvirt-guests work with Kolla on github, but the needed script file appears to be missing from the kolla container image, at least on Ubuntu. Could it be an oversight or a regression?
the script and systemd service file need to be installed on the host outside the containers really for it to work so i would not expect it to be in the contianer it self.
On 7/8/21 2:31 PM, Sean Mooney wrote:
the script and systemd service file need to be installed on the host outside the containers really for it to work so i would not expect it to be in the contianer it self.
The systemd script does call another script in the container. Here, lets look at this snippet from the systemd script at https://github.com/openstack/kolla-ansible/blob/master/tools/kolla-libvirt-g... : ExecStart=-/usr/bin/docker exec nova_libvirt /bin/rm -f /var/lib/libvirt/libvirt-guests ExecStart=-/usr/bin/docker start nova_compute ExecStop=/usr/bin/docker stop nova_compute ExecStop=/usr/bin/docker exec nova_libvirt /bin/sh -x /usr/libexec/libvirt-guests.sh shutdown It looks like it removes the libvirt-guests regular script if it's present when it starts and starts a different script with the shutdown argument when it stops. This second script is the one missing from my installed container image, hence why I am confused. -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
On Thu, 8 Jul 2021 at 21:47, J-P Methot <jp.methot@planethoster.info> wrote:
On 7/8/21 2:31 PM, Sean Mooney wrote:
the script and systemd service file need to be installed on the host outside the containers really for it to work so i would not expect it to be in the contianer it self.
The systemd script does call another script in the container. Here, lets look at this snippet from the systemd script at
https://github.com/openstack/kolla-ansible/blob/master/tools/kolla-libvirt-g... :
ExecStart=-/usr/bin/docker exec nova_libvirt /bin/rm -f /var/lib/libvirt/libvirt-guests ExecStart=-/usr/bin/docker start nova_compute ExecStop=/usr/bin/docker stop nova_compute
ExecStop=/usr/bin/docker exec nova_libvirt /bin/sh -x /usr/libexec/libvirt-guests.sh shutdown
It looks like it removes the libvirt-guests regular script if it's present when it starts and starts a different script with the shutdown argument when it stops. This second script is the one missing from my installed container image, hence why I am confused.
The /usr/libexec/libvirt-guests.sh script is available on CentOS images, at least those that I checked: train (centos7) and ussuri (centos8). However, in Ubuntu the script lives at /usr/lib/libvirt/libvirt-guests.sh in both bionic [1] and focal [2]. It is possible that this has simply never been tested on Ubuntu. [1] https://packages.ubuntu.com/bionic/amd64/libvirt-daemon/filelist [2] https://packages.ubuntu.com/focal/amd64/libvirt-daemon/filelist
participants (4)
-
J-P Methot
-
Mark Goddard
-
Pierre Riteau
-
Sean Mooney