Open Stack

I set mem_stats_period_seconds = 0 in nova.conf on controllers and hypervisors, and restarted nova services, and then built another VM, but it still has the balloon device:

albertb at alberttest4:~ $ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:04.0 SCSI storage controller: Red Hat, Inc. Virtio block device
00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device
00:06.0 SCSI storage controller: Red Hat, Inc. Virtio block device
00:07.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon

I'll try the q35 setting now.


-----Original Message-----
From: Sean Mooney <smooney at redhat.com> 
Sent: Monday, February 3, 2020 2:56 PM
To: Clark Boylan <cboylan at sapwetik.org>; openstack-discuss at lists.openstack.org
Subject: Re: Virtio memory balloon driver

On Mon, 2020-02-03 at 14:11 -0800, Clark Boylan wrote:
> On Mon, Feb 3, 2020, at 1:36 PM, Albert Braden wrote:
> >  
> > When we build a Centos 7 VM with 1.4T RAM it fails with “[ 17.797177] 
> > BUG: unable to handle kernel paging request at ffff988b19478000”
> > 
> > 
> > I asked in #centos and they asked me to show a list of devices from a 
> > working VM (if I use 720G RAM it works). This is the list:
> > 
> > 
> > [root at alberttest1 ~]# lspci
> > 
> > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
> > 
> > 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> > 
> > 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
> > 
> > 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB 
> > [Natoma/Triton II] (rev 01)
> > 
> > 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
> > 
> > 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
> > 
> > 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
> > 
> > 00:04.0 SCSI storage controller: Red Hat, Inc. Virtio block device
> > 
> > 00:05.0 SCSI storage controller: Red Hat, Inc. Virtio block device
> > 
> > 00:06.0 SCSI storage controller: Red Hat, Inc. Virtio block device
> > 
> > 00:07.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
> > 
> > [root at alberttest1 ~]# lsusb
> > 
> > Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
> > 
> > Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
> > 
> > 
> > They suspect that the “Virtio memory balloon” driver is causing the 
> > problem, and that we should disable it. I googled around and found this:
> > 
> > 
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linux-2Dkvm.org_page_Projects_auto-2Dballooning&d=DwIFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=CpgL1Yla6XeZXNt8yDzb1d1aB2MMUJ6P4cMQ5cpxSU4&s=GWJcQEqsYfdcWE_hfTBSzWDxJYvLhAtPWlFz1EmzJKY&e= 
> > 
> > 
> > It looks like memory ballooning is deprecated. How can I get rid of the driver?
> 
> Looking at Nova's code [0] the memballoon device is only set if mem_stats_period_seconds has a value greater than 0.
> The default [1] is 10 so you get it by default. I would try setting this config option to 0 and recreating the
> instance. Note I think this will apply to all VMs and was originally added so that tools could get memory usage
> statistics.
i forgot about that option. we had talked bout disableing the stats by default at one point.
downstream i think we do at least via config on realtime hosts as we found the stat collect causes
latency spikes.
> 
> > 
> > 
> > Also they complained about my host bridge device; they say that we 
> > should have a newer one:
> > 
> > 
> > 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
> > 
> > 
> > Where can I specify the host bridge?
> 
> For this I think you need to set hw_machine_type [2]. Looking at this bug [3] I think the value you may want is q35.
yes but if you enable q35 you neeed to also be aware that unlike the pc i440 machine type only 1 addtion pci slot
will be allocated so if you want to allow attaching more then one volume or nic after teh vm is booted you need to
adjust https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_nova_latest_configuration_config.html-23libvirt.num-5Fpcie-5Fports&d=DwIFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=CpgL1Yla6XeZXNt8yDzb1d1aB2MMUJ6P4cMQ5cpxSU4&s=y_xdeWDGbFM4fAr32iEVEER8d6hVZunyvwne1QJfOME&e= .
the more addtional pcie port you enable the more memory is required by qemu regardless of if you use them and by default
even without allocating more pcie port qemu uses more memory in q35 mode then when using the pc machine type.
you should also be aware that ide bus is not supported by default with q35 which causes issues for some older operating
systems if you use config drives.

with all that said we do want to eventully make q35 the default in nova but you just need to be aware that changing that
has lots of other side effects which is why we have not done it yet. q35 is required for many new feature and is
supported but its just not the default.
> 
> > 
> > 
> > <bugs_> ok ozzzo one of the devices is called "virtio memory balloon"
> > 
> > [13:18:12] <bugs_> do you see that?
> > 
> > [13:18:21] <ozzzo> yes
> > 
> > [13:18:47] <bugs_> i suggest you google that and read about what it 
> > does - i think it would
> > 
> > [13:19:02] <bugs_> be worth trying to disable that device on your 
> > larger vm to see what happens
> > 
> > [13:19:18] <ozzzo> ok I will try that, thank you
> > 
> > [13:19:30] * Altiare (~Altiare at unaffiliated/altiare) has quit IRC 
> > (Quit: Leaving)
> > 
> > [13:21:45] * Sheogorath[m] 
> > (sheogora1 at gateway/shell/matrix.org/x-uiiwpoddodtgrwwz) joins #centos
> > 
> > [13:22:06] <@TrevorH> I also notice that the VM seems to be using the 
> > very old 440FX and there's a newer model of hardware available that 
> > might be worth checking
> > 
> > [13:22:21] <@TrevorH> 440FX chipset is the old old pentium Pro chipset!
> > 
> > [13:22:32] <@TrevorH> I had one of those in about 1996
> > 
> 
> [0] https://urldefense.proofpoint.com/v2/url?u=https-3A__opendev.org_openstack_nova_src_branch_master_nova_virt_libvirt_driver.py-23L5840-2DL5852&d=DwIFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=CpgL1Yla6XeZXNt8yDzb1d1aB2MMUJ6P4cMQ5cpxSU4&s=H2dx7K2OyyHfGOjaQ51CGvU0308JWXFBp_80QuxCAPw&e= 
> [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_nova_train_configuration_config.html-23libvirt.mem-5Fstats-5Fperiod-5Fseconds&d=DwIFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=CpgL1Yla6XeZXNt8yDzb1d1aB2MMUJ6P4cMQ5cpxSU4&s=FWkMnQE0rIldStIjTeTrXlBoCR0Bb06TqNQpjQwwXuM&e= 
> [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_nova_train_configuration_config.html-23libvirt.hw-5Fmachine-5Ftype&d=DwIFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=CpgL1Yla6XeZXNt8yDzb1d1aB2MMUJ6P4cMQ5cpxSU4&s=p4N-kGlblu2E47dPo8qYHl2hv4BPROlLoM_-5YNbAzc&e= 
> [3] https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.launchpad.net_nova_-2Bbug_1780138&d=DwIFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=CpgL1Yla6XeZXNt8yDzb1d1aB2MMUJ6P4cMQ5cpxSU4&s=wZjiqcg-XvVbIVgTpHRsbPVuZd1K0mM4-BZ6P0JNMj8&e= 
>