Open Stack

Hello Pierre

Yeah have checked hypervisor too ..No error related to kernel out-of-memory
(OOM) in hypervisor

On Mon, Jan 4, 2021 at 3:45 PM Pierre Riteau <pierre at stackhpc.com> wrote:

> Hi Deepa,
>
> You mention checking dmesg *inside* the VM. But have you checked dmesg
> on the hypervisor? It's possible your qemu-kvm processes are
> terminated by the kernel out-of-memory (OOM) killer because they try
> to allocate more memory than available.
>
> Best wishes,
> Pierre Riteau (priteau)
>
>
> On Wed, 18 Nov 2020 at 03:44, Deepa KR <deepa.kr at fingent.com> wrote:
> >
> > Hello Mohammed
> >
> > Thanks for the response.
> > No error message inside vm. Have checked dmesg, syslog etc .
> >
> > I mentioned vm is shutting down itself because of error messages
> Instance shutdown by itself. Calling the stop API. Current vm_state:
> active, current task_state: None, original DB power_state: 1, current VM
> power_state: 4 from hypervisor.
> >
> > Sent from my iPhone
> >
> > > On 17-Nov-2020, at 11:35 PM, Mohammed Naser <mnaser at vexxhost.com>
> wrote:
> > >
> > > On Tue, Nov 17, 2020 at 12:46 PM Deepa KR <deepa.kr at fingent.com>
> wrote:
> > >>
> > >>  Hi All
> > >>
> > >> We have a Openstack setup with the Ussuri Version and I am regularly
> facing auto shutdown of a few VMs (ubuntu16.04) randomly .
> > >> If I restart then the instance is back .
> > >>
> > >> From logs I was able to see the messages below .
> > >>
> > >> WARNING nova.compute.manager
> [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541
> 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance:
> 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event
> network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance
> with vm_state active and task_state None.
> > >> INFO nova.compute.manager [-] [instance:
> 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event)
> > >> INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 -
> - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During
> _sync_instance_power_state the DB power_state (1) does not match the
> vm_power_state from the hypervisor (4). Updating power_state in the DB to
> match the hypervisor.
> > >> syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]:
> 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager
> [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance:
> 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling
> the stop API. Current vm_state: active, current task_state: None, original
> DB power_state: 1, current VM power_state: 4
> > >>  nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - -
> - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already
> powered off in the hypervisor when stop is called.
> > >> nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 -
> - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already
> shutdown.
> > >> nova.virt.libvirt.driver [-] [instance:
> 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully.
> > >> nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3
> d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default
> default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output
> > >> nova.virt.libvirt.driver [-] [instance:
> 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully.
> > >>
> > >> I searched a few blogs and forums but couldn't find a solution to it .
> > >>
> > >> Few mentioned to add sync_power_state_interval=-1 in
> /etc/nova/nova.conf .But understood that this will help only when nova
> stops vm.
> > >> But in this case vm itself is shutting down (Instance shutdown by
> itself. Calling the stop API)
> > >> Also no memory issue in VM nor the hypervisor.
> > >> Also did apt-get upgrade .
> > >>
> > >> It would be great if anyone can shed light to this issue.
> > >
> > > You should check and see if there is anything inside `dmesg` that
> > > shows the VM dying (any segfaults?).  Also, it's possible that the VM
> > > itself is shutting off so maybe you should check ni its logs.
> > >
> > >> Regards,
> > >> Deepa K R
> > >>
> > >> Sent from my iPhone
> > >
> > >
> > >
> > > --
> > > Mohammed Naser
> > > VEXXHOST, Inc.
> >
>


-- 


Regards,

Deepa K R | DevOps Team Lead



USA | UAE | INDIA | AUSTRALIA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210105/081e1f9f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature-1.gif
Type: image/gif
Size: 566 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210105/081e1f9f/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo_for_signature.png
Type: image/png
Size: 10509 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210105/081e1f9f/attachment.png>