Open Stack

 OK! I have applied the patch and now weights are shown!
 Furthermore as per your suggestion I have removed the "RamFilter" which 
 was the only one present.


 And here is the new log where spawning of 2 VMs can be seen with a few 
 seconds of difference: https://pastebin.com/Xy2FL2KL

 Initially both hosts are of weight 1.0 then the one with one VM already 
 running has negative weight but the new VM is placed on the other host.

 Really-really strange why this is happening...

 G.


> On Wed, 17 Apr 2019 12:55:45 -0700, Melanie Witt <melwittt at gmail.com> 
> wrote:
>> On Wed, 17 Apr 2019 22:45:45 +0300, Georgios Dimitrakakis
>> <giorgis at acmac.uoc.gr> wrote:
>>>    Hello again Menalie!
>>>
>>>    Exactly this is what I am thinking...something is not working
>>>    correctly!
>>>
>>>    To answer your questions there is one node acting as controller 
>>> where
>>>    the scheduler is running and I have pasted the nova.conf file 
>>> from
>>>    there.
>>>
>>>    I have also noticed that I have "ram_weight_multiplier" two 
>>> times (one
>>>    in [cells] and one in [filter_scheduler]) therefore I have 
>>> removed the
>>>    one in [cells] because I though it might give a problem but the 
>>> results
>>>    are still the same.
>>>
>>>    The log for the scheduler has this entry:
>>>
>>>    2019-04-17 22:04:50.045 131723 DEBUG oslo_service.service
>>>    [req-7e548ecb-f3ed-4a4d-835f-b3a996e32534 - - - - -]
>>>    filter_scheduler.ram_weight_multiplier = -1.0 log_opt_values
>>>    /usr/lib/python2.7/site-packages/oslo_config/cfg.py:3032
>>>
>>>    so it seems to be picked up correctly but without any influence.
>> Agreed, that log shows that the -1.0 value is being picked up 
>> properly
>> by the scheduler service.
>>
>>>    What also worries me from the scheduler log that I have send to 
>>> you
>>>    before is that in there I see an entry like this:
>>>
>>>    2019-04-17 19:53:07.298 98874 DEBUG nova.filters
>>>    [req-02fb5504-cbdb-4219-9509-d2be9da7bb0e
>>>    6a4c2e32919e4a6fa5c5d956beb68eef 
>>> 9f22e9bfa7974e14871d58bbb62242b2 -
>>>    default default] Filter RamFilter returned 2 host(s)
>>>    get_filtered_objects
>>>    /usr/lib/python2.7/site-packages/nova/filters.py:104
>>>
>>>    Shouldn't the RamFilter return 1host and the one with less RAM? 
>>> Why
>>>    does it return 2hosts??
>> No -- the RamFilter will return any hosts that meet the RAM 
>> requirement.
>> Filters do not weigh hosts. The RamFilter returns two hosts because 
>> both
>> hosts have enough RAM to fulfill the request. FYI though, as of Pike
>> [1], the (Core|Ram|Disk)Filter are redundant, as placement will do 
>> the
>> filtering for those resources before the nova scheduler filters run. 
>> So
>> you can safely remove (Core|Ram|Disk)Filter from your 
>> enabled_filters.
>> [1]
>> 
>> https://docs.openstack.org/releasenotes/nova/pike.html#relnotes-16-0-0-stable-pike-upgrade-notes
>>
>>>    If you have any other ideas or would like me to do some more 
>>> checking I
>>>    am all ears!
>> At this point, you could take Matt's suggestion from his latest 
>> reply on
>> this thread and patch in the logging regression fix he linked. That
>> would allow you to see in the debug log what weights nova is giving 
>> to
>> the hosts.
>
> OK, so I just searched open nova bugs for "weigh" and found this
> issue, which isn't necessarily a defect:
>
> https://bugs.launchpad.net/nova/+bug/1818239
>
> but something that could be affecting the host weighing in your
> environment. There's something called the BuildFailureWeigher which
> will apply a low weight multiplier to hosts that have had VMs fail to
> build on them. And that weight resets when a host experiences a
> successful VM build.
>
> If you apply the patch Matt suggested and take a look at the host
> weights, we should be able to see whether the BuildFailureWeigher is
> involved in the behavior you're seeing.
>
> -melanie
>
>> Aside from that, it's looking like we/I would need to reproduce this
>> issue locally with a devstack and try to figure out what's causing 
>> this
>> behavior.
>> -melanie
>>
>>>>> Thank you both Melanie and Matt for trying to assist me.
>>>>> I have double checked the nova.conf at the controller and here is
>>>>> what
>>>>>     I have (ignored hashed lines and obfuscating sensitive data):
>>>>>     https://pastebin.com/hW1PE4U7
>>>>> As you can see I have everything with default values as discussed
>>>>>     before with Melanie except the filters and the weight that I 
>>>>> have
>>>>>     applied that should lead to VM stacking instead of spreading.
>>>>> My case scenario is with two compute hosts (let's call them 
>>>>> "cpu1"
>>>>> and
>>>>>     "cpu2") and when an instance is already placed on "cpu2" I 
>>>>> expect
>>>>> the
>>>>>     next instance to be placed also there. But instead is placed 
>>>>> on
>>>>> "cpu1"
>>>>>     as you can see from the scheduler log that can find here:
>>>>>     https://pastebin.com/sCzB9L2e
>>>>> Do you see something strange that I fail to recognize?
>>>>
>>>> Thanks for providing the helpful data. It appears you have set 
>>>> your
>>>> nova.conf correctly (this is where your scheduler is running, 
>>>> yes?).
>>>> I
>>>> notice you have duplicated the ram_weight_multiplier setting but 
>>>> that
>>>> shouldn't hurt anything.
>>>>
>>>> The relevant scheduler log is this one:
>>>>
>>>> 2019-04-17 19:53:07.303 98874 DEBUG 
>>>> nova.scheduler.filter_scheduler
>>>> [req-02fb5504-cbdb-4219-9509-d2be9da7bb0e
>>>> 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 
>>>> -
>>>> default default] Weighed [(cpu1, cpu1) ram: 32153MB disk: 
>>>> 1906688MB
>>>> io_ops: 0 instances: 0, (cpu2, cpu2) ram: 30105MB disk: 1886208MB
>>>> io_ops: 0 instances: 1] _get_sorted_hosts
>>>>
>>>> 
>>>> /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:455
>>>>
>>>> and here we see that host 'cpu1' is being weighed ahead of host
>>>> 'cpu2', which is the problem. I don't understand this considering 
>>>> the
>>>> docs say that setting the ram_weight_multiplier to a negative 
>>>> value
>>>> should result in the host with the lesser RAM being weighed
>>>> higher/first. According to your log, the opposite is happening --
>>>> 'cpu1' with 32153MB RAM is being weighed higher than 'cpu2' with
>>>> 30105MB RAM.
>>>>
>>>> Either your ram_weight_multiplier setting is not being picked up 
>>>> or
>>>> there's a bug causing weight to be applied with reverse logic?
>>>>
>>>> Can you look at the scheduler debug log when the service first
>>>> started up and verify what value of ram_weight_multiplier the 
>>>> service
>>>> is using?
>>>>
>>>> -melanie
>>>>
>>>>>> On 4/16/2019 7:03 PM, melanie witt wrote:
>>>>>>> To debug further, you should set debug to True in the nova.conf 
>>>>>>> on
>>>>>>> your scheduler host and look for which filter is removing the
>>>>>>> desired
>>>>>>> host for the second VM. You can find where to start by looking 
>>>>>>> for
>>>>>>> a
>>>>>>> message like, "Starting with N host(s)". If you have two hosts
>>>>>>> with
>>>>>>> enough RAM, you should see "Starting with 2 host(s)" and then 
>>>>>>> look
>>>>>>> for
>>>>>>> the log message where it says "Filter returned 1 host(s)" and 
>>>>>>> that
>>>>>>> will be the filter that is removing the desired host. Once you
>>>>>>> know
>>>>>>> which filter is removing it, you can debug further.
>>>>>>
>>>>>> If the other host isn't getting filtered out, it could be the
>>>>>> weighers that aren't prioritizing the host you expect, but debug
>>>>>> logs
>>>>>> should dump the weighed hosts as well which might give a clue.
>>>>>
>>>>>
>>>
>>
>>
>>
>>