Open Stack

Hello Lee

Below is the only information from qemu logs.

2020-12-22T07:20:10.251883Z qemu-system-x86_64: warning: host doesn't
support requested feature: MSR(48CH).vmx-invept-single-context
-noglobals [bit 43]
2020-12-22T07:20:10.251887Z qemu-system-x86_64: warning: host doesn't
support requested feature: MSR(480H).vmx-ins-outs [bit 54]
2020-12-22T07:20:10.251890Z qemu-system-x86_64: warning: host doesn't
support requested feature: MSR(480H).vmx-true-ctls [bit 55]
2020-12-22T07:20:10.251894Z qemu-system-x86_64: warning: host doesn't
support requested feature: MSR(491H).vmx-eptp-switching [bit 0
]
*2021-01-05 01:39:20.896+0000: shutting down, reason=crashed <<< no logs
before or after this from Timestamp*
2021-01-05 06:21:37.682+0000: starting up libvirt version: 6.0.0, package:
0ubuntu8~cloud0 (Openstack Ubuntu Testing Bot <openstack-
testing-bot at ubuntu.com> Mon, 20 Apr 2020 18:44:06 +0000), qemu version:
4.2.0Debian 1:4.2-3ubuntu6~cloud0, kernel: 4.15.0-106-generi
c, hostname: fgshwbucehyp02.maas
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
\
HOME=/var/lib/libvirt/qemu/domain-147-instance-0000057b \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-147-instance-0000057b/.local/share
\
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-147-instance-0000057b/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-147-instance-0000057b/.config \

These VMs are migrated from VMware to Openstack (if this tip helps in
someway)

May VMs get auto shutdown randomly

instance-0000032c.log:2021-01-03 17:30:32.980+0000: shutting down,
reason=crashed
instance-000004d3.log:2020-12-10 21:20:45.807+0000: shutting down,
reason=crashed
instance-000004d3.log:2020-12-13 21:06:43.683+0000: shutting down,
reason=crashed
instance-000004d3.log:2020-12-18 23:02:30.727+0000: shutting down,
reason=crashed
instance-000004d3.log:2020-12-23 16:39:22.194+0000: shutting down,
reason=crashed
instance-000004d3.log:2020-12-29 23:43:03.554+0000: shutting down,
reason=crashed
instance-000004d3.log:2021-01-03 19:13:08.850+0000: shutting down,
reason=crashed
instance-0000057b.log:2020-12-07 04:20:44.540+0000: shutting down,
reason=crashed
instance-0000057b.log:2020-12-09 18:22:08.652+0000: shutting down,
reason=crashed
instance-0000057b.log:2020-12-12 19:37:34.824+0000: shutting down,
reason=crashed
instance-0000057b.log:2020-12-15 16:38:50.268+0000: shutting down,
reason=crashed
instance-0000057b.log:2020-12-16 17:31:29.975+0000: shutting down,
reason=crashed
instance-0000057b.log:2020-12-21 18:55:58.644+0000: shutting down,
reason=crashed
instance-0000057b.log:2021-01-05 01:39:20.896+0000: shutting down,
reason=crashed

Not sure what could i do more to prevent the auto shutdown

On Mon, Jan 4, 2021 at 3:14 PM Lee Yarwood <lyarwood at redhat.com> wrote:

> On 04-01-21 14:53:59, Deepa KR wrote:
> > Hi All
> >
> > Any suggestions highly appreciated.
> > We are facing these issues very frequently now .
>
> Can you pastebin the domain QEMU log from
> /var/log/libvirt/qemu/$domain.log? That should detail why the domain is
> crashing.
>
> I'd also recommend reviewing the following docs from libvirt on how to
> enable debug logs etc:
>
> https://libvirt.org/kbase/debuglogs.html
>
> > On Mon, Nov 23, 2020 at 10:03 AM Deepa KR <deepa.kr at fingent.com> wrote:
> > > Hi
> > >
> > > Can see only  shutting down, reason=crashed in libvirt/qemu logs
> .Nothing
> > > else .
> > > Couldn't find anything else in neutron logs as well
> > >
> > >
> > > On Wed, Nov 18, 2020 at 5:23 PM Deepa KR <deepa.kr at fingent.com> wrote:
> > >
> > >> Thanks for pointing out. Have 70 + vms and has issue with just 3 vms
> so i
> > >> am really confused
> > >>
> > >> Sent from my iPhone
> > >>
> > >> On 18-Nov-2020, at 1:56 PM, rui zang <rui.zang at yandex.com> wrote:
> > >>
> > >> 
> > >> [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received unexpected
> > >> event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for
> > >> instance with vm_state active and task_state None.*
> > >>
> > >>
> > >> Clearly the network virtual interface was somehow removed or
> unplugged.
> > >> What you should look into is OVS or whatever the network solution you
> are
> > >> using.
> > >>
> > >>
> > >> 18.11.2020, 01:44, "Deepa KR" <deepa.kr at fingent.com>:
> > >>
> > >>  Hi All
> > >>
> > >> We have a Openstack setup with the *Ussuri Version* and I am
> *regularly
> > >> facing auto shutdown of a few VMs (ubuntu16.04) randomly* .
> > >> If I restart then the instance is back .
> > >>
> > >> From logs I was able to see the messages below .
> > >>
> > >>  WARNING nova.compute.manager
> [req-2a21d455-ac04-44aa-b248-4776e5109013
> > >> 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 -
> default
> > >> default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Received
> > >> unexpected event
> network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9
> > >> for instance with vm_state active and task_state None.*
> > >>  INFO nova.compute.manager [-] [instance:
> > >> 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event)
> > >>  INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477
> - -
> > >> - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *During
> > >> _sync_instance_power_state the DB power_state (1) does not match the
> > >> vm_power_state from the hypervisor (4). Updating power_state in the
> DB to
> > >> match the hypervisor.*
> > >> syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]:
> 2020-11-13
> > >> 07:01:07.684 2680204 WARNING nova.compute.manager
> > >> [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance:
> > >> 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance shutdown by itself.
> > >> Calling the stop API. Current vm_state: active, current task_state:
> None,
> > >> original DB power_state: 1, current VM power_state: 4*
> > >>   nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - -
> - -
> > >> -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance is
> already
> > >> powered off in the hypervisor when stop is called.*
> > >>  nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 -
> - -
> > >> - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance
> already
> > >> shutdown.*
> > >>  nova.virt.libvirt.driver [-] [instance:
> > >> 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed
> successfully.*
> > >>  nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3
> > >> d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c -
> default
> > >> default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console
> output
> > >>  nova.virt.libvirt.driver [-] [instance:
> > >> 28cd861c-ef15-444a-a902-9cac643c72b5] *Instance destroyed
> successfully.*
> > >>
> > >> I searched a few blogs and forums but couldn't find a solution to it .
> > >>
> > >> Few mentioned to add s*ync_power_state_interval=-1 in *
> > >> */etc/nova/nova.conf *.But understood that this will help only when
> nova
> > >> stops vm.
> > >> But in this case vm itself is shutting down (*Instance shutdown by
> > >> itself. Calling the stop API*)
> > >> Also no memory issue in VM nor the hypervisor.
> > >> Also did apt-get upgrade .
> > >>
> > >> It would be great if anyone can shed light to this issue.
> > >>
> > >> Regards,
> > >> Deepa K R
> > >>
> > >> Sent from my iPhone
> > >>
> > >>
> > >
> > > --
> > >
> > >
> > > Regards,
> > >
> > > Deepa K R | DevOps Team Lead
> > >
> > >
> > >
> > > USA | UAE | INDIA | AUSTRALIA
> > >
> > >
> > >
> >
> > --
> >
> >
> > Regards,
> >
> > Deepa K R | DevOps Team Lead
> >
> >
> >
> > USA | UAE | INDIA | AUSTRALIA
>
>
>
>
> --
> Lee Yarwood                 A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672
> 2D76
>


-- 


Regards,

Deepa K R | DevOps Team Lead



USA | UAE | INDIA | AUSTRALIA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210105/a9cf8374/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature-1.gif
Type: image/gif
Size: 566 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210105/a9cf8374/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logo_for_signature.png
Type: image/png
Size: 10509 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210105/a9cf8374/attachment-0001.png>