On Wed, 2019-12-04 at 00:51 +0000, Albert Braden wrote:
We are running Rocky and we have the allocation bug because we set cpu_allocation_ratio and ram_allocation_ratio to 1 on the controllers but not on the hypervisors, so the hypervisors are oversubscribed. When we try to reboot a VM when the HV is short memory, the VM is deleted from the HV. It still exists in Openstack but not on the HV. Is it possible to recover a VM from this state? so the vm is likely not deleted the harddisk and the libvirt domain will still be defiend. my guess is starting the vm cause the oom reaper to kill the qemu process when it was relaunched.
if you are deploying with memory over subsription enabled you should configre enough swap space equal to total memory* ram_allocation_ratio. if you do that you you should be able to start the vm. now the other option is while the vm is stopped you can cold migrate it to another host and then start it there.
Logs: http://dpaste.com/1E8VQ4V
root@us01-p02-hv128:~# virsh list --all --uuid|grep 19ad13a4-f974-43ec-88a5-a6dbc13667a2 root@us01-p02-hv128:~#
root@us01odc-p02-ctrl1:~# oss 19ad13a4-f974-43ec-88a5-a6dbc13667a2 +-------------------------------------+------------------------------------------------------------------------------- ----+
Field | Value |
+-------------------------------------+------------------------------------------------------------------------------- ----+
OS-DCF:diskConfig | MANUAL | OS-EXT-AZ:availability_zone | nova | OS-EXT-SRV-ATTR:host | us01-p02- hv128 | OS-EXT-SRV-ATTR:hypervisor_hostname | us01-p02- hv128.internal.synopsys.com | OS-EXT-SRV-ATTR:instance_name | instance- 00000c85 | OS-EXT-STS:power_state | NOSTATE | OS-EXT-STS:task_state | None | OS-EXT-STS:vm_state | stopped | OS-SRV-USG:launched_at | 2019-11- 11T20:11:30.000000 | OS-SRV-USG:terminated_at | None | accessIPv4 | | accessIPv6 | | addresses | vg- network=10.195.72.215 | config_drive | | created | 2019-11- 11T20:10:40Z | flavor | s1.16cx120g (3cfd8fbc-55da-4c1d-a717- ed86d3a0219f) | hostId | fa4acc9f16b333592492f385f41bf15d593a269e72e03d24f752a611 | id | 19ad13a4-f974-43ec-88a5- a6dbc13667a2 | image | QSC-P-CentOS6.6-19P1-v4 (91497018-7b42-4448-aa2a- 4c08fc9621ac) | key_name | None | name | vctrain2-qscp-cs66-849- 104 | project_id | 0b591e0864d04e6b8b6afa18a5a4a638 | properties | BU='VC-Product', Farm='odc_vctrain', Owner='jvrp', Workspace='vctrain2-qscp- cs66' | security_groups | name='default' | status | SHUTOFF | updated | 2019-12- 03T23:57:49Z | user_id | 0df75a295783f2f7c3843642f3a170f500bbeb4c69806bfd53e166b772617bf4 | volumes_attached | |
+-------------------------------------+------------------------------------------------------------------------------- ----+