Hi All We have a Openstack setup with the Ussuri Version and I am regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly . If I restart then the instance is back . From logs I was able to see the messages below . WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None. INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already powered off in the hypervisor when stop is called. nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already shutdown. nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. I searched a few blogs and forums but couldn't find a solution to it . Few mentioned to add sync_power_state_interval=-1 in /etc/nova/nova.conf .But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (Instance shutdown by itself. Calling the stop API) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade . It would be great if anyone can shed light to this issue. Regards, Deepa K R Sent from my iPhone
On Tue, Nov 17, 2020 at 12:46 PM Deepa KR <deepa.kr@fingent.com> wrote:
Hi All
We have a Openstack setup with the Ussuri Version and I am regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly . If I restart then the instance is back .
From logs I was able to see the messages below .
WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None. INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already powered off in the hypervisor when stop is called. nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already shutdown. nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully.
I searched a few blogs and forums but couldn't find a solution to it .
Few mentioned to add sync_power_state_interval=-1 in /etc/nova/nova.conf .But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (Instance shutdown by itself. Calling the stop API) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade .
It would be great if anyone can shed light to this issue.
You should check and see if there is anything inside `dmesg` that shows the VM dying (any segfaults?). Also, it's possible that the VM itself is shutting off so maybe you should check ni its logs.
Regards, Deepa K R
Sent from my iPhone
-- Mohammed Naser VEXXHOST, Inc.
Hello Mohammed Thanks for the response. No error message inside vm. Have checked dmesg, syslog etc . I mentioned vm is shutting down itself because of error messages Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 from hypervisor. Sent from my iPhone
On 17-Nov-2020, at 11:35 PM, Mohammed Naser <mnaser@vexxhost.com> wrote:
On Tue, Nov 17, 2020 at 12:46 PM Deepa KR <deepa.kr@fingent.com> wrote:
Hi All
We have a Openstack setup with the Ussuri Version and I am regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly . If I restart then the instance is back .
From logs I was able to see the messages below .
WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None. INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already powered off in the hypervisor when stop is called. nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already shutdown. nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully.
I searched a few blogs and forums but couldn't find a solution to it .
Few mentioned to add sync_power_state_interval=-1 in /etc/nova/nova.conf .But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (Instance shutdown by itself. Calling the stop API) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade .
It would be great if anyone can shed light to this issue.
You should check and see if there is anything inside `dmesg` that shows the VM dying (any segfaults?). Also, it's possible that the VM itself is shutting off so maybe you should check ni its logs.
Regards, Deepa K R
Sent from my iPhone
-- Mohammed Naser VEXXHOST, Inc.
On Tue, Nov 17, 2020 at 9:43 PM Deepa KR <deepa.kr@fingent.com> wrote: > Hello Mohammed > > Thanks for the response. > No error message inside vm. Have checked dmesg, syslog etc . > > I mentioned vm is shutting down itself because of error messages Instance > shutdown by itself. Calling the stop API. Current vm_state: active, current > task_state: None, original DB power_state: 1, current VM power_state: 4 > from hypervisor. This really doesn’t mean that the VM is shutting itself down. This just means that Nova has noticed that the power state of the VM doesn’t match the database one. You would also see this error message in any scenario where the VM disappeared from libvirt > > Sent from my iPhone > > > On 17-Nov-2020, at 11:35 PM, Mohammed Naser <mnaser@vexxhost.com> wrote: > > > > On Tue, Nov 17, 2020 at 12:46 PM Deepa KR <deepa.kr@fingent.com> wrote: > >> > >> Hi All > >> > >> We have a Openstack setup with the Ussuri Version and I am regularly > facing auto shutdown of a few VMs (ubuntu16.04) randomly . > >> If I restart then the instance is back . > >> > >> From logs I was able to see the messages below . > >> > >> WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 > 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default > default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received > unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 > for instance with vm_state active and task_state None. > >> INFO nova.compute.manager [-] [instance: > 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) > >> INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - > - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During > _sync_instance_power_state the DB power_state (1) does not match the > vm_power_state from the hypervisor (4). Updating power_state in the DB to > match the hypervisor. > >> syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 > 07:01:07.684 2680204 WARNING nova.compute.manager > [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: > 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling > the stop API. Current vm_state: active, current task_state: None, original > DB power_state: 1, current VM power_state: 4 > >> nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - > -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already > powered off in the hypervisor when stop is called. > >> nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - > - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already > shutdown. > >> nova.virt.libvirt.driver [-] [instance: > 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. > >> nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 > d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default > default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output > >> nova.virt.libvirt.driver [-] [instance: > 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. > >> > >> I searched a few blogs and forums but couldn't find a solution to it . > >> > >> Few mentioned to add sync_power_state_interval=-1 in > /etc/nova/nova.conf .But understood that this will help only when nova > stops vm. > >> But in this case vm itself is shutting down (Instance shutdown by > itself. Calling the stop API) > >> Also no memory issue in VM nor the hypervisor. > >> Also did apt-get upgrade . > >> > >> It would be great if anyone can shed light to this issue. > > > > You should check and see if there is anything inside `dmesg` that > > shows the VM dying (any segfaults?). Also, it's possible that the VM > > itself is shutting off so maybe you should check ni its logs. > > > >> Regards, > >> Deepa K R > >> > >> Sent from my iPhone > > > > > > > > -- > > Mohammed Naser > > VEXXHOST, Inc. > -- Mohammed Naser VEXXHOST, Inc.
Thanks for the response. Any idea what would have caused power state to mismatch in DB. Hypervisors qemu versions are upto date. We have many VMs and seeing this issue with only few VMs(say 3) Sent from my iPhone
On 18-Nov-2020, at 8:57 AM, Mohammed Naser <mnaser@vexxhost.com> wrote:
This really doesn’t mean that the VM is shutting itself down. This just means that Nova has noticed that the power state of the VM doesn’t match the database one.
You would also see this error message in any scenario where the VM disappeared from libvirt
Hi, from my own experience the mismatch happened when someone was playing with virt-manager (or other tools) to reboot/shutdown VMs through libvirt and not from inside the VM or via openstack. And since nova has to catch up at some point this led to a cycle of unwanted shutdown events. In my case I know that users were trying to do that via virt-manager, does that apply to your scenario, too? Or does libvirt show any error logs that would explain unexpected shutdowns? Regards, Eugen Zitat von Deepa KR <deepa.kr@fingent.com>:
Thanks for the response. Any idea what would have caused power state to mismatch in DB. Hypervisors qemu versions are upto date. We have many VMs and seeing this issue with only few VMs(say 3)
Sent from my iPhone
On 18-Nov-2020, at 8:57 AM, Mohammed Naser <mnaser@vexxhost.com> wrote:
This really doesn’t mean that the VM is shutting itself down. This just means that Nova has noticed that the power state of the VM doesn’t match the database one.
You would also see this error message in any scenario where the VM disappeared from libvirt
Hello Eugen Thanks for the response. I am pretty sure none was playing with virt-manager. Wil check libvirt logs and update back. Even enabled crashdump in vm which didn’t generate any at the time of vm shutdown. In qemu logs there was just one message stating “ 2020-11-13 07:00:52.164+0000: shutting down, reason=crashed” Sent from my iPhone
On 18-Nov-2020, at 11:49 AM, Eugen Block <eblock@nde.ag> wrote:
Hi,
from my own experience the mismatch happened when someone was playing with virt-manager (or other tools) to reboot/shutdown VMs through libvirt and not from inside the VM or via openstack. And since nova has to catch up at some point this led to a cycle of unwanted shutdown events. In my case I know that users were trying to do that via virt-manager, does that apply to your scenario, too? Or does libvirt show any error logs that would explain unexpected shutdowns?
Regards, Eugen
Zitat von Deepa KR <deepa.kr@fingent.com>:
Thanks for the response. Any idea what would have caused power state to mismatch in DB. Hypervisors qemu versions are upto date. We have many VMs and seeing this issue with only few VMs(say 3)
Sent from my iPhone
On 18-Nov-2020, at 8:57 AM, Mohammed Naser <mnaser@vexxhost.com> wrote:
This really doesn’t mean that the VM is shutting itself down. This just means that Nova has noticed that the power state of the VM doesn’t match the database one.
You would also see this error message in any scenario where the VM disappeared from libvirt
Hi Deepa, You mention checking dmesg *inside* the VM. But have you checked dmesg on the hypervisor? It's possible your qemu-kvm processes are terminated by the kernel out-of-memory (OOM) killer because they try to allocate more memory than available. Best wishes, Pierre Riteau (priteau) On Wed, 18 Nov 2020 at 03:44, Deepa KR <deepa.kr@fingent.com> wrote:
Hello Mohammed
Thanks for the response. No error message inside vm. Have checked dmesg, syslog etc .
I mentioned vm is shutting down itself because of error messages Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 from hypervisor.
Sent from my iPhone
On 17-Nov-2020, at 11:35 PM, Mohammed Naser <mnaser@vexxhost.com> wrote:
On Tue, Nov 17, 2020 at 12:46 PM Deepa KR <deepa.kr@fingent.com> wrote:
Hi All
We have a Openstack setup with the Ussuri Version and I am regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly . If I restart then the instance is back .
From logs I was able to see the messages below .
WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None. INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already powered off in the hypervisor when stop is called. nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already shutdown. nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully.
I searched a few blogs and forums but couldn't find a solution to it .
Few mentioned to add sync_power_state_interval=-1 in /etc/nova/nova.conf .But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (Instance shutdown by itself. Calling the stop API) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade .
It would be great if anyone can shed light to this issue.
You should check and see if there is anything inside `dmesg` that shows the VM dying (any segfaults?). Also, it's possible that the VM itself is shutting off so maybe you should check ni its logs.
Regards, Deepa K R
Sent from my iPhone
-- Mohammed Naser VEXXHOST, Inc.
Hello Pierre Yeah have checked hypervisor too ..No error related to kernel out-of-memory (OOM) in hypervisor On Mon, Jan 4, 2021 at 3:45 PM Pierre Riteau <pierre@stackhpc.com> wrote: > Hi Deepa, > > You mention checking dmesg *inside* the VM. But have you checked dmesg > on the hypervisor? It's possible your qemu-kvm processes are > terminated by the kernel out-of-memory (OOM) killer because they try > to allocate more memory than available. > > Best wishes, > Pierre Riteau (priteau) > > > On Wed, 18 Nov 2020 at 03:44, Deepa KR <deepa.kr@fingent.com> wrote: > > > > Hello Mohammed > > > > Thanks for the response. > > No error message inside vm. Have checked dmesg, syslog etc . > > > > I mentioned vm is shutting down itself because of error messages > Instance shutdown by itself. Calling the stop API. Current vm_state: > active, current task_state: None, original DB power_state: 1, current VM > power_state: 4 from hypervisor. > > > > Sent from my iPhone > > > > > On 17-Nov-2020, at 11:35 PM, Mohammed Naser <mnaser@vexxhost.com> > wrote: > > > > > > On Tue, Nov 17, 2020 at 12:46 PM Deepa KR <deepa.kr@fingent.com> > wrote: > > >> > > >> Hi All > > >> > > >> We have a Openstack setup with the Ussuri Version and I am regularly > facing auto shutdown of a few VMs (ubuntu16.04) randomly . > > >> If I restart then the instance is back . > > >> > > >> From logs I was able to see the messages below . > > >> > > >> WARNING nova.compute.manager > [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 > 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: > 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event > network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance > with vm_state active and task_state None. > > >> INFO nova.compute.manager [-] [instance: > 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) > > >> INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - > - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During > _sync_instance_power_state the DB power_state (1) does not match the > vm_power_state from the hypervisor (4). Updating power_state in the DB to > match the hypervisor. > > >> syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: > 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager > [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: > 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling > the stop API. Current vm_state: active, current task_state: None, original > DB power_state: 1, current VM power_state: 4 > > >> nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - > - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already > powered off in the hypervisor when stop is called. > > >> nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - > - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already > shutdown. > > >> nova.virt.libvirt.driver [-] [instance: > 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. > > >> nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 > d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default > default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output > > >> nova.virt.libvirt.driver [-] [instance: > 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. > > >> > > >> I searched a few blogs and forums but couldn't find a solution to it . > > >> > > >> Few mentioned to add sync_power_state_interval=-1 in > /etc/nova/nova.conf .But understood that this will help only when nova > stops vm. > > >> But in this case vm itself is shutting down (Instance shutdown by > itself. Calling the stop API) > > >> Also no memory issue in VM nor the hypervisor. > > >> Also did apt-get upgrade . > > >> > > >> It would be great if anyone can shed light to this issue. > > > > > > You should check and see if there is anything inside `dmesg` that > > > shows the VM dying (any segfaults?). Also, it's possible that the VM > > > itself is shutting off so maybe you should check ni its logs. > > > > > >> Regards, > > >> Deepa K R > > >> > > >> Sent from my iPhone > > > > > > > > > > > > -- > > > Mohammed Naser > > > VEXXHOST, Inc. > > > -- Regards, Deepa K R | DevOps Team Lead USA | UAE | INDIA | AUSTRALIA
Thanks for pointing out. Have 70 + vms and has issue with just 3 vms so i am really confused Sent from my iPhone
On 18-Nov-2020, at 1:56 PM, rui zang <rui.zang@yandex.com> wrote:
[instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None.
Clearly the network virtual interface was somehow removed or unplugged. What you should look into is OVS or whatever the network solution you are using.
18.11.2020, 01:44, "Deepa KR" <deepa.kr@fingent.com>: Hi All
We have a Openstack setup with the Ussuri Version and I am regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly . If I restart then the instance is back .
From logs I was able to see the messages below .
WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None. INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already powered off in the hypervisor when stop is called. nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already shutdown. nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully.
I searched a few blogs and forums but couldn't find a solution to it .
Few mentioned to add sync_power_state_interval=-1 in /etc/nova/nova.conf .But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (Instance shutdown by itself. Calling the stop API) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade .
It would be great if anyone can shed light to this issue.
Regards, Deepa K R
Sent from my iPhone
Hi Can see only shutting down, reason=crashed in libvirt/qemu logs .Nothing else . Couldn't find anything else in neutron logs as well On Wed, Nov 18, 2020 at 5:23 PM Deepa KR <deepa.kr@fingent.com> wrote:
Thanks for pointing out. Have 70 + vms and has issue with just 3 vms so i am really confused
Sent from my iPhone
On 18-Nov-2020, at 1:56 PM, rui zang <rui.zang@yandex.com> wrote:
[instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None.*
Clearly the network virtual interface was somehow removed or unplugged. What you should look into is OVS or whatever the network solution you are using.
18.11.2020, 01:44, "Deepa KR" <deepa.kr@fingent.com>:
Hi All
We have a Openstack setup with the *Ussuri Version* and I am *regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly* . If I restart then the instance is back .
From logs I was able to see the messages below .
WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None.* INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.* syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4* nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance is already powered off in the hypervisor when stop is called.* nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance already shutdown.* nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed successfully.* nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed successfully.*
I searched a few blogs and forums but couldn't find a solution to it .
Few mentioned to add s*ync_power_state_interval=-1 in * */etc/nova/nova.conf *.But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (*Instance shutdown by itself. Calling the stop API*) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade .
It would be great if anyone can shed light to this issue.
Regards, Deepa K R
Sent from my iPhone
-- Regards, Deepa K R | DevOps Team Lead USA | UAE | INDIA | AUSTRALIA
Can see only shutting down, reason=crashed in libvirt/qemu logs .Nothing else . Couldn't find anything else in neutron logs as well
Just to be sure, this isn't due to a NUMA scheduling issue by chance? I had this topic up a little bit ago, where it is possible to have Nova over-schedule a NUMA node when the flavor's parameters are not set correctly: http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018045.h... Eric
Hi All Any suggestions highly appreciated. We are facing these issues very frequently now . On Mon, Nov 23, 2020 at 10:03 AM Deepa KR <deepa.kr@fingent.com> wrote:
Hi
Can see only shutting down, reason=crashed in libvirt/qemu logs .Nothing else . Couldn't find anything else in neutron logs as well
On Wed, Nov 18, 2020 at 5:23 PM Deepa KR <deepa.kr@fingent.com> wrote:
Thanks for pointing out. Have 70 + vms and has issue with just 3 vms so i am really confused
Sent from my iPhone
On 18-Nov-2020, at 1:56 PM, rui zang <rui.zang@yandex.com> wrote:
[instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None.*
Clearly the network virtual interface was somehow removed or unplugged. What you should look into is OVS or whatever the network solution you are using.
18.11.2020, 01:44, "Deepa KR" <deepa.kr@fingent.com>:
Hi All
We have a Openstack setup with the *Ussuri Version* and I am *regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly* . If I restart then the instance is back .
From logs I was able to see the messages below .
WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None.* INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.* syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4* nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance is already powered off in the hypervisor when stop is called.* nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance already shutdown.* nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed successfully.* nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed successfully.*
I searched a few blogs and forums but couldn't find a solution to it .
Few mentioned to add s*ync_power_state_interval=-1 in * */etc/nova/nova.conf *.But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (*Instance shutdown by itself. Calling the stop API*) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade .
It would be great if anyone can shed light to this issue.
Regards, Deepa K R
Sent from my iPhone
--
Regards,
Deepa K R | DevOps Team Lead
USA | UAE | INDIA | AUSTRALIA
-- Regards, Deepa K R | DevOps Team Lead USA | UAE | INDIA | AUSTRALIA
On 04-01-21 14:53:59, Deepa KR wrote:
Hi All
Any suggestions highly appreciated. We are facing these issues very frequently now .
Can you pastebin the domain QEMU log from /var/log/libvirt/qemu/$domain.log? That should detail why the domain is crashing. I'd also recommend reviewing the following docs from libvirt on how to enable debug logs etc: https://libvirt.org/kbase/debuglogs.html
On Mon, Nov 23, 2020 at 10:03 AM Deepa KR <deepa.kr@fingent.com> wrote:
Hi
Can see only shutting down, reason=crashed in libvirt/qemu logs .Nothing else . Couldn't find anything else in neutron logs as well
On Wed, Nov 18, 2020 at 5:23 PM Deepa KR <deepa.kr@fingent.com> wrote:
Thanks for pointing out. Have 70 + vms and has issue with just 3 vms so i am really confused
Sent from my iPhone
On 18-Nov-2020, at 1:56 PM, rui zang <rui.zang@yandex.com> wrote:
[instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None.*
Clearly the network virtual interface was somehow removed or unplugged. What you should look into is OVS or whatever the network solution you are using.
18.11.2020, 01:44, "Deepa KR" <deepa.kr@fingent.com>:
Hi All
We have a Openstack setup with the *Ussuri Version* and I am *regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly* . If I restart then the instance is back .
From logs I was able to see the messages below .
WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None.* INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.* syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4* nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance is already powered off in the hypervisor when stop is called.* nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance already shutdown.* nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed successfully.* nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed successfully.*
I searched a few blogs and forums but couldn't find a solution to it .
Few mentioned to add s*ync_power_state_interval=-1 in * */etc/nova/nova.conf *.But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (*Instance shutdown by itself. Calling the stop API*) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade .
It would be great if anyone can shed light to this issue.
Regards, Deepa K R
Sent from my iPhone
--
Regards,
Deepa K R | DevOps Team Lead
USA | UAE | INDIA | AUSTRALIA
--
Regards,
Deepa K R | DevOps Team Lead
USA | UAE | INDIA | AUSTRALIA
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
Hello Lee Below is the only information from qemu logs. 2020-12-22T07:20:10.251883Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-invept-single-context -noglobals [bit 43] 2020-12-22T07:20:10.251887Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(480H).vmx-ins-outs [bit 54] 2020-12-22T07:20:10.251890Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(480H).vmx-true-ctls [bit 55] 2020-12-22T07:20:10.251894Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(491H).vmx-eptp-switching [bit 0 ] *2021-01-05 01:39:20.896+0000: shutting down, reason=crashed <<< no logs before or after this from Timestamp* 2021-01-05 06:21:37.682+0000: starting up libvirt version: 6.0.0, package: 0ubuntu8~cloud0 (Openstack Ubuntu Testing Bot <openstack- testing-bot@ubuntu.com> Mon, 20 Apr 2020 18:44:06 +0000), qemu version: 4.2.0Debian 1:4.2-3ubuntu6~cloud0, kernel: 4.15.0-106-generi c, hostname: fgshwbucehyp02.maas LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \ HOME=/var/lib/libvirt/qemu/domain-147-instance-0000057b \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-147-instance-0000057b/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-147-instance-0000057b/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-147-instance-0000057b/.config \ These VMs are migrated from VMware to Openstack (if this tip helps in someway) May VMs get auto shutdown randomly instance-0000032c.log:2021-01-03 17:30:32.980+0000: shutting down, reason=crashed instance-000004d3.log:2020-12-10 21:20:45.807+0000: shutting down, reason=crashed instance-000004d3.log:2020-12-13 21:06:43.683+0000: shutting down, reason=crashed instance-000004d3.log:2020-12-18 23:02:30.727+0000: shutting down, reason=crashed instance-000004d3.log:2020-12-23 16:39:22.194+0000: shutting down, reason=crashed instance-000004d3.log:2020-12-29 23:43:03.554+0000: shutting down, reason=crashed instance-000004d3.log:2021-01-03 19:13:08.850+0000: shutting down, reason=crashed instance-0000057b.log:2020-12-07 04:20:44.540+0000: shutting down, reason=crashed instance-0000057b.log:2020-12-09 18:22:08.652+0000: shutting down, reason=crashed instance-0000057b.log:2020-12-12 19:37:34.824+0000: shutting down, reason=crashed instance-0000057b.log:2020-12-15 16:38:50.268+0000: shutting down, reason=crashed instance-0000057b.log:2020-12-16 17:31:29.975+0000: shutting down, reason=crashed instance-0000057b.log:2020-12-21 18:55:58.644+0000: shutting down, reason=crashed instance-0000057b.log:2021-01-05 01:39:20.896+0000: shutting down, reason=crashed Not sure what could i do more to prevent the auto shutdown On Mon, Jan 4, 2021 at 3:14 PM Lee Yarwood <lyarwood@redhat.com> wrote: > On 04-01-21 14:53:59, Deepa KR wrote: > > Hi All > > > > Any suggestions highly appreciated. > > We are facing these issues very frequently now . > > Can you pastebin the domain QEMU log from > /var/log/libvirt/qemu/$domain.log? That should detail why the domain is > crashing. > > I'd also recommend reviewing the following docs from libvirt on how to > enable debug logs etc: > > https://libvirt.org/kbase/debuglogs.html > > > On Mon, Nov 23, 2020 at 10:03 AM Deepa KR <deepa.kr@fingent.com> wrote: > > > Hi > > > > > > Can see only shutting down, reason=crashed in libvirt/qemu logs > .Nothing > > > else . > > > Couldn't find anything else in neutron logs as well > > > > > > > > > On Wed, Nov 18, 2020 at 5:23 PM Deepa KR <deepa.kr@fingent.com> wrote: > > > > > >> Thanks for pointing out. Have 70 + vms and has issue with just 3 vms > so i > > >> am really confused > > >> > > >> Sent from my iPhone > > >> > > >> On 18-Nov-2020, at 1:56 PM, rui zang <rui.zang@yandex.com> wrote: > > >> > > >> > > >> [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received unexpected > > >> event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for > > >> instance with vm_state active and task_state None.* > > >> > > >> > > >> Clearly the network virtual interface was somehow removed or > unplugged. > > >> What you should look into is OVS or whatever the network solution you > are > > >> using. > > >> > > >> > > >> 18.11.2020, 01:44, "Deepa KR" <deepa.kr@fingent.com>: > > >> > > >> Hi All > > >> > > >> We have a Openstack setup with the *Ussuri Version* and I am > *regularly > > >> facing auto shutdown of a few VMs (ubuntu16.04) randomly* . > > >> If I restart then the instance is back . > > >> > > >> From logs I was able to see the messages below . > > >> > > >> WARNING nova.compute.manager > [req-2a21d455-ac04-44aa-b248-4776e5109013 > > >> 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - > default > > >> default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received > > >> unexpected event > network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 > > >> for instance with vm_state active and task_state None.* > > >> INFO nova.compute.manager [-] [instance: > > >> 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) > > >> INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 > - - > > >> - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *During > > >> _sync_instance_power_state the DB power_state (1) does not match the > > >> vm_power_state from the hypervisor (4). Updating power_state in the > DB to > > >> match the hypervisor.* > > >> syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: > 2020-11-13 > > >> 07:01:07.684 2680204 WARNING nova.compute.manager > > >> [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: > > >> 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance shutdown by itself. > > >> Calling the stop API. Current vm_state: active, current task_state: > None, > > >> original DB power_state: 1, current VM power_state: 4* > > >> nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - > - - > > >> -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance is > already > > >> powered off in the hypervisor when stop is called.* > > >> nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - > - - > > >> - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance > already > > >> shutdown.* > > >> nova.virt.libvirt.driver [-] [instance: > > >> 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed > successfully.* > > >> nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 > > >> d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - > default > > >> default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console > output > > >> nova.virt.libvirt.driver [-] [instance: > > >> 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed > successfully.* > > >> > > >> I searched a few blogs and forums but couldn't find a solution to it . > > >> > > >> Few mentioned to add s*ync_power_state_interval=-1 in * > > >> */etc/nova/nova.conf *.But understood that this will help only when > nova > > >> stops vm. > > >> But in this case vm itself is shutting down (*Instance shutdown by > > >> itself. Calling the stop API*) > > >> Also no memory issue in VM nor the hypervisor. > > >> Also did apt-get upgrade . > > >> > > >> It would be great if anyone can shed light to this issue. > > >> > > >> Regards, > > >> Deepa K R > > >> > > >> Sent from my iPhone > > >> > > >> > > > > > > -- > > > > > > > > > Regards, > > > > > > Deepa K R | DevOps Team Lead > > > > > > > > > > > > USA | UAE | INDIA | AUSTRALIA > > > > > > > > > > > > > -- > > > > > > Regards, > > > > Deepa K R | DevOps Team Lead > > > > > > > > USA | UAE | INDIA | AUSTRALIA > > > > > -- > Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 > 2D76 > -- Regards, Deepa K R | DevOps Team Lead USA | UAE | INDIA | AUSTRALIA
participants (7)
-
Deepa KR
-
Eric K. Miller
-
Eugen Block
-
Lee Yarwood
-
Mohammed Naser
-
Pierre Riteau
-
rui zang