Open Stack

When I start and stop the giant VM I don't see any evidence of OOM errors. I suspect that the #centos guys may be correct when they say that the "Virtio memory balloon" device is not capable of addressing that much memory, and that I must disable it if I want to create VMs with 1.4T RAM. Setting "mem_stats_period_seconds = 0" doesn't seem to disable it.

How are others working around this? Is anyone else creating Centos 6 VMs with 1.4T or more RAM?

Console log: https://f.perl.bot/p/njvgbm

The error is at line 404: [   18.736435] BUG: unable to handle kernel paging request at ffff9ca8d9980000

Dmesg:
[Tue Feb  4 17:50:42 2020] brq49cbe55d-51: port 1(tap039191ba-25) entered disabled state
[Tue Feb  4 17:50:42 2020] device tap039191ba-25 left promiscuous mode
[Tue Feb  4 17:50:42 2020] brq49cbe55d-51: port 1(tap039191ba-25) entered disabled state
[Tue Feb  4 17:50:47 2020] brq49cbe55d-51: port 1(tap039191ba-25) entered blocking state
[Tue Feb  4 17:50:47 2020] brq49cbe55d-51: port 1(tap039191ba-25) entered disabled state
[Tue Feb  4 17:50:47 2020] device tap039191ba-25 entered promiscuous mode
[Tue Feb  4 17:50:47 2020] brq49cbe55d-51: port 1(tap039191ba-25) entered blocking state
[Tue Feb  4 17:50:47 2020] brq49cbe55d-51: port 1(tap039191ba-25) entered forwarding state

Syslog:

Feb  4 17:50:51 us01odc-p01-hv214 kernel: [2859840.751339] brq49cbe55d-51: port 1(tap039191ba-25) entered blocking state
Feb  4 17:50:51 us01odc-p01-hv214 kernel: [2859840.751342] brq49cbe55d-51: port 1(tap039191ba-25) entered disabled state
Feb  4 17:50:51 us01odc-p01-hv214 kernel: [2859840.751450] device tap039191ba-25 entered promiscuous mode
Feb  4 17:50:51 us01odc-p01-hv214 systemd-networkd[781]: tap039191ba-25: Gained carrier
Feb  4 17:50:51 us01odc-p01-hv214 libvirtd[37317]: 2020-02-05 01:50:51.386+0000: 37321: warning : qemuDomainObjTaint:5602 : Domain id=15 name='instance-00002164' uuid=33611060-887a-44c1-a3b8-1c36cb8f9984 is tainted: host-cpu
Feb  4 17:50:51 us01odc-p01-hv214 systemd-udevd[238052]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  4 17:50:51 us01odc-p01-hv214 networkd-dispatcher[1214]: WARNING:Unknown index 32 seen, reloading interface list
Feb  4 17:50:51 us01odc-p01-hv214 dnsmasq[28739]: reading /etc/resolv.conf
Feb  4 17:50:51 us01odc-p01-hv214 dnsmasq[28739]: using nameserver 127.0.0.53#53
Feb  4 17:50:51 us01odc-p01-hv214 kernel: [2859840.751683] brq49cbe55d-51: port 1(tap039191ba-25) entered blocking state
Feb  4 17:50:51 us01odc-p01-hv214 kernel: [2859840.751685] brq49cbe55d-51: port 1(tap039191ba-25) entered forwarding state
Feb  4 17:50:51 us01odc-p01-hv214 dnsmasq[28739]: reading /etc/resolv.conf
Feb  4 17:50:51 us01odc-p01-hv214 dnsmasq[28739]: using nameserver 127.0.0.53#53
Feb  4 17:50:52 us01odc-p01-hv214 systemd-networkd[781]: tap039191ba-25: Gained IPv6LL
Feb  4 17:50:52 us01odc-p01-hv214 dnsmasq[28739]: reading /etc/resolv.conf
Feb  4 17:50:52 us01odc-p01-hv214 dnsmasq[28739]: using nameserver 127.0.0.53#53


-----Original Message-----
From: Jeremy Stanley <fungi at yuggoth.org> 
Sent: Tuesday, February 4, 2020 4:01 AM
To: openstack-discuss at lists.openstack.org
Subject: Re: Virtio memory balloon driver

On 2020-02-03 23:57:28 +0000 (+0000), Albert Braden wrote:
> We are reserving 2 CPU and 16G RAM for the hypervisor. I haven't
> seen any OOM errors. Where should I look for those?
[...]

The `dmesg` utility on the hypervisor host should show you the
kernel's log ring buffer contents (the -T flag is useful to
translate its timestamps into something more readable than seconds
since boot too). If the ring buffer has overwritten the relevant
timeframe then look for signs of kernel OOM killer invocation in
your syslog or persistent journald storage.
-- 
Jeremy Stanley