Open Stack

 Hello again and apologies for my absence!

 First of all let me express my gratitude for your valuable feedback.
 I understand what all of you are saying and I will try what Sean 
 suggested by setting the values for CPUs and Disk to something beyond 
 the default.

 Meanwhile what I 've though was to change the "weight_classes" 
 parameter to use only the "RAM" weight instead of all, since initially 
 this is what I would like to be based and then move on from there.
 Unfortunately I haven't found a way to properly set it and no matter 
 what I 've tried I always ended up with an error in nova-scheduler.log 
 saying "ERROR nova ValueError: Empty module name"

 Any ideas on how to set it based on the available weights? It seems 
 that it needs a list format but how?

 Best,

 G.

> On Wed, 2019-04-17 at 18:24 -0700, melanie witt wrote:
>> On Thu, 18 Apr 2019 01:38:17 +0100, Sean Mooney <smooney at redhat.com> 
>> wrote:
>> > On Wed, 2019-04-17 at 15:30 -0700, melanie witt wrote:
>> > > On Thu, 18 Apr 2019 01:13:42 +0300, Georgios Dimitrakakis
>> > > <giorgis at acmac.uoc.gr> wrote:
>> > > > Shouldn’t that be the correct behavior and place the new VM on 
>> the host with the smaller weight? Isn’t that what
>> > > > the
>> > > > negative value for “ram_weight_multiplier” does ?
>> > >
>> > > No, it's the opposite, higher weights win. That's why you have 
>> to use a
>> > > negative value for ram_weight_multiplier if you want hosts with 
>> _less_
>> > > RAM to win over hosts with more RAM (stacking).
>> > >
>> > > > Please let me know how I can provide to you more debug 
>> info....
>> > >
>> > > One thing I noticed from your log is on the second request, 
>> 'cpu1' has
>> > > io_ops: 0 whereas 'cpu2' has io_ops: 1 and the IoOpsWeigher [1] 
>> will
>> > > prefer hosts with fewer io_iops by default. Note that's only one 
>> piece
>> > > of the ending weight -- the weighing process will take a sum of 
>> all of
>> > > the weights each weigher returns. So the weight returned from 
>> RamWeigher
>> > > is added to the weight returned from IoOpsWeigher is added the 
>> weight
>> > > returned from CPUWeigher, and so on.
>> > >
>> > > So, as Matt said, we're a bit in the dark now as far as what 
>> each
>> > > weigher is returning and we don't currently have debug logging 
>> per
>> > > weigher the way we do for filters. That would be an enhancement 
>> we could
>> > > make to aid in debugging issues like this one. You could hack 
>> something
>> > > up locally to log/print the returned weight in each weight class 
>> under
>> > > the nova/scheduler/weights/ directory, if you want to dig into 
>> that.
>> > >
>> > > Another thing I noticed is that there are probably some new 
>> weighers
>> > > available by default that did not exist in the previous version 
>> of nova
>> > > that you were using in the past. By default, the config option 
>> for weighers:
>> > >
>> > > [filter_scheduler]weight_classes = 
>> ["nova.scheduler.weights.all_weighers"]
>> > >
>> > > will pick up all weigher classes in the nova/scheduler/weights/ 
>> code
>> > > directory. You might take a look at these and see if any are 
>> ones you
>> > > should exclude in your environment. For example, the CPUWeigher 
>> [1] (new
>> > > in Rocky) will spread VMs based on available CPU by default.
>> >
>> > most of the weighers spread by default so the cpu weigher may be a 
>> factor but
>> > the the disk weigher tends to all hevily impact the choice.
>> >
>> > we do not normalise any of the values retured by the different 
>> weighers
>> > the disk wighter is basically  host_state.free_disk_mb * 
>> disk_weight_multiplier
>>
>> Hm, I thought we do based on this code:
>>
>> 
>> https://github.com/openstack/nova/blob/stable/rocky/nova/weights.py#L135
>>
>> which normalizes the weight to a value between 0.0 and 1.0.
> yes we do but that is witnin the same resouce type.
>
> we do not normalise between resoucser types.
> on a typical host you will have between 2GB and 8GB of ram per cpu 
> core
> and you will have between 10G and 100G of local disk typically.
>
> so we dont look at the capasity of each resouce and renomalise 
> between them.
> if you want achive that you have to carefully tweek the weights to do
> that manually.
>
> that said i have not sat down and worked out the math to do that in
> about 3-4 years
> but i think i used to run my dev clouster with the
> disk_weight_multiplier around 0.04
> and i think i used to set the ram_weight_multiplier to 2.0
>
> anyway that is just my personal experience and i was trying to tweek
> the spreading behavior
> to spread based on ram then cpus then disk rather then pack based on
> ram but spread based on the rest.
> so i dont know if this is relvent or correct for this usecase.
>
>>
>> If the multiplier is large though, that could make the considered 
>> value
>>  > 1.0 (as is the case with the default 
>> build_failure_weight_multiplier:
>>
>> 
>> https://github.com/openstack/nova/blob/stable/rocky/nova/conf/scheduler.py#L501
>>
>> -melanie
>>
>> > althoer host_state.free_disk_mb is actully disk_available_least.
>> >
>> > as a result the disk filter will weigh cpu1 19456 height then cpu2
>> > the dela between the cpu1 and cpu2 based on the ram weigher is 
>> only 2048
>> >
>> > if you want the ram filter to take presedence over teh disk filter 
>> you will
>> > need to scale the disk filter down to be in a similar value range
>> >
>> > i woudl suggest setting disk_weight_multiplier=0.001
>> >
>> > > This
>> > > weigher might be contributing to the VM spreading you're seeing. 
>> You
>> > > might try playing with the '[filter_scheduler]weight_classes' 
>> config
>> > > option to select only the weighers you want or alternatively you 
>> could
>> > > set the weighers multipliers the way you prefer.
>> > >
>> > > -melanie
>> > >
>> > > [1] 
>> https://docs.openstack.org/nova/rocky/user/filter-scheduler.html#weights
>> > >
>> > > > > > On 4/17/2019 3:50 PM, Georgios Dimitrakakis wrote:
>> > > > > > And here is the new log where spawning of 2 VMs can be 
>> seen with a few seconds of difference:
>> > > > > > https://pastebin.com/Xy2FL2KL
>> > > > > > Initially both hosts are of weight 1.0 then the one with 
>> one VM already running has negative weight but the
>> > > > > > new
>> > > > > > VM is placed on the other host.
>> > > > > > Really-really strange why this is happening...
>> > > > >
>> > > > > < 2019-04-17 23:26:18.770 157355 DEBUG 
>> nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-
>> > > > > 377b3d37bff9
>> > > > > 6a4c2e32919e4a6fa5c5d956beb68eef 
>> 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filtered [(cpu2, 
>> cpu2)
>> > > > > ram:
>> > > > > 30105MB disk: 1887232MB io_ops: 1 instances: 1, (cpu1, cpu1) 
>> ram: 32153MB disk: 1906688MB io_ops: 0 instances:
>> > > > > 0]
>> > > > > _get_sorted_hosts 
>> /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:435
>> > > > >
>> > > > > < 2019-04-17 23:26:18.771 157355 DEBUG 
>> nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-
>> > > > > 377b3d37bff9
>> > > > > 6a4c2e32919e4a6fa5c5d956beb68eef 
>> 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed 
>> [WeighedHost
>> > > > > [host:
>> > > > > (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 
>> instances: 0, weight: 1.0], WeighedHost [host: (cpu2,
>> > > > > cpu2)
>> > > > > ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, weight: 
>> -0.00900862553213]] _get_sorted_hosts
>> > > > > 
>> /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:454
>> > > > >
>> > > > > cpu1 is definitely getting weighed higher but I'm not sure 
>> why. We likely need some debug logging on the
>> > > > > result of
>> > > > > each weigher like we have for each filter to figure out 
>> what's going on with the weighers.
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Matt
>> > >
>> > >
>> > >
>> > >
>> > >
>>
>>
>>