Open Stack

Thank you Sean. We've scheduled a change to fix this in our prod cluster, but we have a mess to clean up in the meantime; this is very helpful!

-----Original Message-----
From: Sean Mooney <smooney at redhat.com> 
Sent: Wednesday, December 4, 2019 4:30 AM
To: Albert Braden <albertb at synopsys.com>; openstack-discuss at lists.openstack.org
Subject: Re: Rebooted VM disappears when HV lacks memory

On Wed, 2019-12-04 at 00:51 +0000, Albert Braden wrote:
> We are running Rocky and we have the allocation bug because we set cpu_allocation_ratio and ram_allocation_ratio to 1
> on the controllers but not on the hypervisors, so the hypervisors are oversubscribed. When we try to reboot a VM when
> the HV is short memory, the VM is deleted from the HV. It still exists in Openstack but not on the HV. Is it possible
> to recover a VM from this state?
so the vm is likely not deleted the harddisk and the libvirt domain will still be defiend. my guess is starting the
vm cause the oom reaper to kill the qemu process when it was relaunched.

if you are deploying with memory over subsription enabled you should configre enough swap space equal to total
memory* ram_allocation_ratio. 

if you do that you you should be able to start the vm. now the other option is while the vm is stopped you can
cold migrate it to another host and then start it there.
> 
> Logs: https://urldefense.proofpoint.com/v2/url?u=http-3A__dpaste.com_1E8VQ4V&d=DwICaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=7fgYsjA7VtYl3Zo7ZOo08R0HaR3R7RhHMwgCUqpKA0I&s=Frt7z6L65bkZegYdEev8ZM6gYH1UblhS0dziKFbB1zY&e= 
> 
> root at us01-p02-hv128:~# virsh list --all --uuid|grep 19ad13a4-f974-43ec-88a5-a6dbc13667a2
> root at us01-p02-hv128:~#
> 
> root at us01odc-p02-ctrl1:~# oss 19ad13a4-f974-43ec-88a5-a6dbc13667a2
> +-------------------------------------+-------------------------------------------------------------------------------
> ----+
> > Field                               |
> > Value                                                                             |
> 
> +-------------------------------------+-------------------------------------------------------------------------------
> ----+
> > OS-DCF:diskConfig                   |
> > MANUAL                                                                            |
> > OS-EXT-AZ:availability_zone         |
> > nova                                                                              |
> > OS-EXT-SRV-ATTR:host                | us01-p02-
> > hv128                                                                    |
> > OS-EXT-SRV-ATTR:hypervisor_hostname | us01-p02-
> > hv128.internal.synopsys.com                                              |
> > OS-EXT-SRV-ATTR:instance_name       | instance-
> > 00000c85                                                                 |
> > OS-EXT-STS:power_state              |
> > NOSTATE                                                                           |
> > OS-EXT-STS:task_state               |
> > None                                                                              |
> > OS-EXT-STS:vm_state                 |
> > stopped                                                                           |
> > OS-SRV-USG:launched_at              | 2019-11-
> > 11T20:11:30.000000                                                        |
> > OS-SRV-USG:terminated_at            |
> > None                                                                              |
> > accessIPv4                          |                                                                               
> >     |
> > accessIPv6                          |                                                                               
> >     |
> > addresses                           | vg-
> > network=10.195.72.215                                                          |
> > config_drive                        |                                                                               
> >     |
> > created                             | 2019-11-
> > 11T20:10:40Z                                                              |
> > flavor                              | s1.16cx120g (3cfd8fbc-55da-4c1d-a717-
> > ed86d3a0219f)                                |
> > hostId                              |
> > fa4acc9f16b333592492f385f41bf15d593a269e72e03d24f752a611                          |
> > id                                  | 19ad13a4-f974-43ec-88a5-
> > a6dbc13667a2                                              |
> > image                               | QSC-P-CentOS6.6-19P1-v4 (91497018-7b42-4448-aa2a-
> > 4c08fc9621ac)                    |
> > key_name                            |
> > None                                                                              |
> > name                                | vctrain2-qscp-cs66-849-
> > 104                                                        |
> > project_id                          |
> > 0b591e0864d04e6b8b6afa18a5a4a638                                                  |
> > properties                          | BU='VC-Product', Farm='odc_vctrain', Owner='jvrp', Workspace='vctrain2-qscp-
> > cs66' |
> > security_groups                     |
> > name='default'                                                                    |
> > status                              |
> > SHUTOFF                                                                           |
> > updated                             | 2019-12-
> > 03T23:57:49Z                                                              |
> > user_id                             |
> > 0df75a295783f2f7c3843642f3a170f500bbeb4c69806bfd53e166b772617bf4                  |
> > volumes_attached                    |                                                                               
> >     |
> 
> +-------------------------------------+-------------------------------------------------------------------------------
> ----+