<div dir="ltr">Seems indeed that bug, Or at least I can reproduce the behavior you described creating or destroying an instance <div><br></div><div>Thanks, Massimo</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 27, 2019 at 3:49 PM Gaudenz Steinlin <<a href="mailto:gaudenz@durcheinandertal.ch">gaudenz@durcheinandertal.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi<br>
<br>
Massimo Sgaravatto <<a href="mailto:massimo.sgaravatto@gmail.com" target="_blank">massimo.sgaravatto@gmail.com</a>> writes:<br>
<br>
> Thanks for the prompt feedback. <br>
> <br>
> This [*] is the output of "openstack hypervisor show <br>
> <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a>" <br>
> <br>
> Let me also add that if I restart openstack-nova-compute on the <br>
> hypervisor, then "nova host-describe" shows the right <br>
> information for a few seconds:<br>
<br>
Could it be that you are hitting this bug:<br>
<a href="https://bugs.launchpad.net/nova/+bug/1733034" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nova/+bug/1733034</a><br>
<br>
If you are affected by this bug you will see the memory usage <br>
change between the "sum of all instances + reserved memory" and <br>
the actual usage on the hypervisor as reported by libvirt.<br>
<br>
Just call "nova hypervisor-show " a few times to see the values <br>
change. depending on which version you are running, you have to <br>
create or destroy an instance to see the changing values.<br>
<br>
Gaudenz<br>
><br>
><br>
> [root@cld-ctrl-01 ~]# nova host-describe <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a><br>
> +-----------------------------+----------------------------------+-----+-----------+---------+<br>
> | HOST | PROJECT | cpu |<br>
> memory_mb | disk_gb |<br>
> +-----------------------------+----------------------------------+-----+-----------+---------+<br>
> | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | (total) | 8 |<br>
> 32722 | 241 |<br>
> | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | (used_now) | 19 |<br>
> 39424 | 95 |<br>
> | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | (used_max) | 19 |<br>
> 38912 | 95 |<br>
> | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | b08eede75d5e4be4b0fe21e68fa9c688 | 1 |<br>
> 2048 | 20 |<br>
> | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | 7890b3e262264529a19f9743cf2f14bc | 16 |<br>
> 32768 | 50 |<br>
> | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | 4d8187cffa6a4085ad4a357b8a5afc03 | 2 |<br>
> 4096 | 25 |<br>
> +-----------------------------+----------------------------------+-----+-----------+---------+<br>
><br>
><br>
> but after a while it goes back publishing 16256 MB as used_now<br>
><br>
> Thanks, Massimo<br>
><br>
> [*]<br>
> # openstack hypervisor show <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a><br>
> +----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>
> | Field | Value<br>
><br>
> |<br>
> +----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>
> | aggregates | [u'Unipd-AdminTesting-Unipd',<br>
> u'Unipd-HPC-Physics', u'Unipd-model-glasses---DISC', u'Unipd-DECAMP',<br>
> u'Unipd-Biodiversity-Macro-biomes', u'Unipd- |<br>
> | | CMB4PrimordialNonGaussianity',<br>
> u'Unipd-Links-in-Channel', u'Unipd-TC4SAP', u'Unipd-Diabetes_Risk',<br>
> u'Unipd-LabTrasporti', u'Unipd-Hydrological_DA', u'Unipd- |<br>
> | | Smart_Enterprise_in_Cloud', u'Unipd-DEM_Sim_ICEA',<br>
> u'Unipd-Plasmonics', u'Unipd-QuBiMM', u'Unipd-MedComp',<br>
> u'Unipd-BigDataComputingCourse', u'Unipd-DSB---Sci.-Biomed', u |<br>
> | | 'Unipd-Notion',<br>
> u'Unipd-The_role_of_trade-offs_in_competitive_ecosystems',<br>
> u'Unipd-MMS_Cloud', u'Unipd-SID2016', u'Unipd-QuantumFuture',<br>
> u'Unipd-Link_Translocation-DFA', |<br>
> | | u'Unipd-Hopping_Transport_in_Lithium_Niobate',<br>
> u'Unipd-Negapedia', u'Unipd-SIGNET-ns3', u'Unipd-SIGNET-MATLAB',<br>
> u'Unipd-QST', u'Unipd-AbinitioTransport', u'Unipd- |<br>
> | | CalcStat', u'Unipd-PhysicsOfData-students',<br>
> u'Unipd-DiSePaM', u'Unipd-Few-mode-optical-fibers', u'Unipd-cleva']<br>
> |<br>
> | cpu_info | {"vendor": "Intel", "model": "SandyBridge-IBRS",<br>
> "arch": "x86_64", "features": ["pge", "avx", "xsaveopt", "clflush", "sep",<br>
> "syscall", "tsc-deadline", "dtes64", "stibp", |<br>
> | | "msr", "xsave", "vmx", "xtpr", "cmov", "ssse3",<br>
> "est", "pat", "monitor", "smx", "pbe", "lm", "tsc", "nx", "fxsr", "tm",<br>
> "sse4.1", "pae", "sse4.2", "pclmuldq", "cx16", |<br>
> | | "pcid", "vme", "mmx", "osxsave", "cx8", "mce",<br>
> "de", "rdtscp", "ht", "dca", "lahf_lm", "pdcm", "mca", "pdpe1gb", "apic",<br>
> "sse", "pse", "ds", "invtsc", "pni", "tm2", |<br>
> | | "aes", "sse2", "ss", "ds_cpl", "arat", "acpi",<br>
> "spec-ctrl", "fpu", "ssbd", "pse36", "mtrr", "popcnt", "x2apic"],<br>
> "topology": {"cores": 4, "cells": 2, "threads": 1, |<br>
> | | "sockets": 1}}<br>
><br>
> |<br>
> | current_workload | 0<br>
><br>
> |<br>
> | disk_available_least | 124<br>
><br>
> |<br>
> | free_disk_gb | 146<br>
><br>
> |<br>
> | free_ram_mb | -6702<br>
><br>
> |<br>
> | host_ip | 192.168.60.150<br>
><br>
> |<br>
> | host_time | 11:56:28<br>
><br>
> |<br>
> | hypervisor_hostname | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a><br>
><br>
> |<br>
> | hypervisor_type | QEMU<br>
><br>
> |<br>
> | hypervisor_version | 2010000<br>
><br>
> |<br>
> | id | 132<br>
><br>
> |<br>
> | load_average | 0.11, 0.19, 0.21<br>
><br>
> |<br>
> | local_gb | 241<br>
><br>
> |<br>
> | local_gb_used | 23<br>
><br>
> |<br>
> | memory_mb | 32722<br>
><br>
> |<br>
> | memory_mb_used | 16255<br>
><br>
> |<br>
> | running_vms | 4<br>
><br>
> |<br>
> | service_host | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a><br>
><br>
> |<br>
> | service_id | 171<br>
><br>
> |<br>
> | state | up<br>
><br>
> |<br>
> | status | enabled<br>
><br>
> |<br>
> | uptime | 85 days, 34 min<br>
><br>
> |<br>
> | users | 1<br>
><br>
> |<br>
> | vcpus | 8<br>
><br>
> |<br>
> | vcpus_used | 19<br>
><br>
> |<br>
> +----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>
> [root@cld-ctrl-01 ~]#<br>
><br>
><br>
><br>
><br>
><br>
> On Wed, Feb 27, 2019 at 11:47 AM Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>> wrote:<br>
><br>
>> On Wed, 2019-02-27 at 09:33 +0100, Massimo Sgaravatto wrote:<br>
>> > In the dashboard of my OpenStack Ocata cloud I see that the reported<br>
>> memory usage is wrong for the hypervisors.<br>
>> > As far as I understand that information should correspond to the<br>
>> "used_now" field of the "nova host-describe"<br>
>> nova host-describe has been removed in later version of nova.<br>
>> > command. And indeed considering an hypervisor:<br>
>> ><br>
>> ><br>
>> > # nova host-describe <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a><br>
>> ><br>
>> +-----------------------------+----------------------------------+-----+-----------+---------+<br>
>> > | HOST | PROJECT | cpu |<br>
>> memory_mb | disk_gb |<br>
>> ><br>
>> +-----------------------------+----------------------------------+-----+-----------+---------+<br>
>> > | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | (total) | 8<br>
>> | 32722 | 241 |<br>
>> > | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | (used_now) | 19<br>
>> | 16259 | 23 |<br>
>> > | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | (used_max) | 19<br>
>> | 38912 | 95 |<br>
>> > | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | b08eede75d5e4be4b0fe21e68fa9c688 | 1<br>
>> | 2048 | 20 |<br>
>> > | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | 7890b3e262264529a19f9743cf2f14bc | 16<br>
>> | 32768 | 50 |<br>
>> > | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a> | 4d8187cffa6a4085ad4a357b8a5afc03 | 2<br>
>> | 4096 | 25 |<br>
>> ><br>
>> +-----------------------------+----------------------------------+-----+-----------+---------+<br>
>> ><br>
>> ><br>
>> > So for the memory it reports 16259 for used_now, while as far as far I<br>
>> understand it should be 38912 + the memory used<br>
>> > by the hypervisor<br>
>> ><br>
>> > Am I missing something ?<br>
>> the hypervisors api's memory_mb_used is (reserved memory + total of all<br>
>> memory associtate with instance on the host).<br>
>> i would have expected used_now to reflect that.<br>
>> what does "openstack hypervisor show <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a>" print.<br>
>><br>
>> ><br>
>> > Thanks, Massimo<br>
>> ><br>
>><br>
>><br>
-- <br>
PGP: 836E 4F81 EFBB ADA7 0852 79BF A97A 7702 BAF9 1EF5<br>
</blockquote></div>