[ops] [nova] Wrong reported memory hypervisor usage
Gaudenz Steinlin
gaudenz at durcheinandertal.ch
Wed Feb 27 14:49:54 UTC 2019
Hi
Massimo Sgaravatto <massimo.sgaravatto at gmail.com> writes:
> Thanks for the prompt feedback.
>
> This [*] is the output of "openstack hypervisor show
> cld-blu-01.cloud.pd.infn.it"
>
> Let me also add that if I restart openstack-nova-compute on the
> hypervisor, then "nova host-describe" shows the right
> information for a few seconds:
Could it be that you are hitting this bug:
https://bugs.launchpad.net/nova/+bug/1733034
If you are affected by this bug you will see the memory usage
change between the "sum of all instances + reserved memory" and
the actual usage on the hypervisor as reported by libvirt.
Just call "nova hypervisor-show " a few times to see the values
change. depending on which version you are running, you have to
create or destroy an instance to see the changing values.
Gaudenz
>
>
> [root at cld-ctrl-01 ~]# nova host-describe cld-blu-01.cloud.pd.infn.it
> +-----------------------------+----------------------------------+-----+-----------+---------+
> | HOST | PROJECT | cpu |
> memory_mb | disk_gb |
> +-----------------------------+----------------------------------+-----+-----------+---------+
> | cld-blu-01.cloud.pd.infn.it | (total) | 8 |
> 32722 | 241 |
> | cld-blu-01.cloud.pd.infn.it | (used_now) | 19 |
> 39424 | 95 |
> | cld-blu-01.cloud.pd.infn.it | (used_max) | 19 |
> 38912 | 95 |
> | cld-blu-01.cloud.pd.infn.it | b08eede75d5e4be4b0fe21e68fa9c688 | 1 |
> 2048 | 20 |
> | cld-blu-01.cloud.pd.infn.it | 7890b3e262264529a19f9743cf2f14bc | 16 |
> 32768 | 50 |
> | cld-blu-01.cloud.pd.infn.it | 4d8187cffa6a4085ad4a357b8a5afc03 | 2 |
> 4096 | 25 |
> +-----------------------------+----------------------------------+-----+-----------+---------+
>
>
> but after a while it goes back publishing 16256 MB as used_now
>
> Thanks, Massimo
>
> [*]
> # openstack hypervisor show cld-blu-01.cloud.pd.infn.it
> +----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | Field | Value
>
> |
> +----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> | aggregates | [u'Unipd-AdminTesting-Unipd',
> u'Unipd-HPC-Physics', u'Unipd-model-glasses---DISC', u'Unipd-DECAMP',
> u'Unipd-Biodiversity-Macro-biomes', u'Unipd- |
> | | CMB4PrimordialNonGaussianity',
> u'Unipd-Links-in-Channel', u'Unipd-TC4SAP', u'Unipd-Diabetes_Risk',
> u'Unipd-LabTrasporti', u'Unipd-Hydrological_DA', u'Unipd- |
> | | Smart_Enterprise_in_Cloud', u'Unipd-DEM_Sim_ICEA',
> u'Unipd-Plasmonics', u'Unipd-QuBiMM', u'Unipd-MedComp',
> u'Unipd-BigDataComputingCourse', u'Unipd-DSB---Sci.-Biomed', u |
> | | 'Unipd-Notion',
> u'Unipd-The_role_of_trade-offs_in_competitive_ecosystems',
> u'Unipd-MMS_Cloud', u'Unipd-SID2016', u'Unipd-QuantumFuture',
> u'Unipd-Link_Translocation-DFA', |
> | | u'Unipd-Hopping_Transport_in_Lithium_Niobate',
> u'Unipd-Negapedia', u'Unipd-SIGNET-ns3', u'Unipd-SIGNET-MATLAB',
> u'Unipd-QST', u'Unipd-AbinitioTransport', u'Unipd- |
> | | CalcStat', u'Unipd-PhysicsOfData-students',
> u'Unipd-DiSePaM', u'Unipd-Few-mode-optical-fibers', u'Unipd-cleva']
> |
> | cpu_info | {"vendor": "Intel", "model": "SandyBridge-IBRS",
> "arch": "x86_64", "features": ["pge", "avx", "xsaveopt", "clflush", "sep",
> "syscall", "tsc-deadline", "dtes64", "stibp", |
> | | "msr", "xsave", "vmx", "xtpr", "cmov", "ssse3",
> "est", "pat", "monitor", "smx", "pbe", "lm", "tsc", "nx", "fxsr", "tm",
> "sse4.1", "pae", "sse4.2", "pclmuldq", "cx16", |
> | | "pcid", "vme", "mmx", "osxsave", "cx8", "mce",
> "de", "rdtscp", "ht", "dca", "lahf_lm", "pdcm", "mca", "pdpe1gb", "apic",
> "sse", "pse", "ds", "invtsc", "pni", "tm2", |
> | | "aes", "sse2", "ss", "ds_cpl", "arat", "acpi",
> "spec-ctrl", "fpu", "ssbd", "pse36", "mtrr", "popcnt", "x2apic"],
> "topology": {"cores": 4, "cells": 2, "threads": 1, |
> | | "sockets": 1}}
>
> |
> | current_workload | 0
>
> |
> | disk_available_least | 124
>
> |
> | free_disk_gb | 146
>
> |
> | free_ram_mb | -6702
>
> |
> | host_ip | 192.168.60.150
>
> |
> | host_time | 11:56:28
>
> |
> | hypervisor_hostname | cld-blu-01.cloud.pd.infn.it
>
> |
> | hypervisor_type | QEMU
>
> |
> | hypervisor_version | 2010000
>
> |
> | id | 132
>
> |
> | load_average | 0.11, 0.19, 0.21
>
> |
> | local_gb | 241
>
> |
> | local_gb_used | 23
>
> |
> | memory_mb | 32722
>
> |
> | memory_mb_used | 16255
>
> |
> | running_vms | 4
>
> |
> | service_host | cld-blu-01.cloud.pd.infn.it
>
> |
> | service_id | 171
>
> |
> | state | up
>
> |
> | status | enabled
>
> |
> | uptime | 85 days, 34 min
>
> |
> | users | 1
>
> |
> | vcpus | 8
>
> |
> | vcpus_used | 19
>
> |
> +----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> [root at cld-ctrl-01 ~]#
>
>
>
>
>
> On Wed, Feb 27, 2019 at 11:47 AM Sean Mooney <smooney at redhat.com> wrote:
>
>> On Wed, 2019-02-27 at 09:33 +0100, Massimo Sgaravatto wrote:
>> > In the dashboard of my OpenStack Ocata cloud I see that the reported
>> memory usage is wrong for the hypervisors.
>> > As far as I understand that information should correspond to the
>> "used_now" field of the "nova host-describe"
>> nova host-describe has been removed in later version of nova.
>> > command. And indeed considering an hypervisor:
>> >
>> >
>> > # nova host-describe cld-blu-01.cloud.pd.infn.it
>> >
>> +-----------------------------+----------------------------------+-----+-----------+---------+
>> > | HOST | PROJECT | cpu |
>> memory_mb | disk_gb |
>> >
>> +-----------------------------+----------------------------------+-----+-----------+---------+
>> > | cld-blu-01.cloud.pd.infn.it | (total) | 8
>> | 32722 | 241 |
>> > | cld-blu-01.cloud.pd.infn.it | (used_now) | 19
>> | 16259 | 23 |
>> > | cld-blu-01.cloud.pd.infn.it | (used_max) | 19
>> | 38912 | 95 |
>> > | cld-blu-01.cloud.pd.infn.it | b08eede75d5e4be4b0fe21e68fa9c688 | 1
>> | 2048 | 20 |
>> > | cld-blu-01.cloud.pd.infn.it | 7890b3e262264529a19f9743cf2f14bc | 16
>> | 32768 | 50 |
>> > | cld-blu-01.cloud.pd.infn.it | 4d8187cffa6a4085ad4a357b8a5afc03 | 2
>> | 4096 | 25 |
>> >
>> +-----------------------------+----------------------------------+-----+-----------+---------+
>> >
>> >
>> > So for the memory it reports 16259 for used_now, while as far as far I
>> understand it should be 38912 + the memory used
>> > by the hypervisor
>> >
>> > Am I missing something ?
>> the hypervisors api's memory_mb_used is (reserved memory + total of all
>> memory associtate with instance on the host).
>> i would have expected used_now to reflect that.
>> what does "openstack hypervisor show cld-blu-01.cloud.pd.infn.it" print.
>>
>> >
>> > Thanks, Massimo
>> >
>>
>>
--
PGP: 836E 4F81 EFBB ADA7 0852 79BF A97A 7702 BAF9 1EF5
More information about the openstack-discuss
mailing list