On Wed, 17 Apr 2019 01:15:49 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

Is there a way I can further debug this?

Setting debug to "true" didn't show a reason why VMs are spreading and which filter is used every time which will make VMs spread.

In addition I think that I 've read somewhere that placement does this (stack VMs) by default but have not seen that behavior at all...

Any ideas appreciated...

Regards,

G.

P.S: Added the "scheduler" tag to catch more attention from the correct people.

I had a look through the code and config options and have not yet found anything that could have caused the default stack behavior to have changed. The related config options are: * [placement]randomize_allocation_candidates which defaults to False * [filter_scheduler]host_subset_size which defaults to 1 * [filter_scheduler]shuffle_best_same_weighed_hosts which defaults to False If you have these set to the defaults, you should be getting stacking behavior. Can you double check whether those options are set (or left as default) properly in your scheduler configs? -melanie

...

All,

in the past I used to stack VMs based on RAM on hosts by putting the following settings in my nova.conf file:

->ram_weight_multiplier=-1.0 ->available_filters=nova.scheduler.filters.all_filters

->enabled_filters=RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter

I have now a Rocky installation and although I have set the above on my controller node and restarted corresponding services the result when spawning two VMs (one after another) is still to be distributed on two different hosts. Seems that the something is missing in order to override the default behavior.

Any suggestions please?

Best regards,

G.

On Wed, 17 Apr 2019 01:51:48 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

Hi Melanie,

thx for looking into this!

The values you are referring to have not been set in “nova.conf” neither at the controller nor at the compute hosts which as far as I understand means that they have their default values and the initial behavior was to spread VMs across hosts.

In order to change that behavior what I have changed are: 1)ram_weight_multiplier=-1.0 2)available_filters=nova.scheduler.filters.all_filters 3)enabled_filters=RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter

but the behavior remained the same without any change since when launching 2VMs the second one always go to the other host that the first is running. Launching more VMs has the same result.

OK, yeah if you have not set those config options in nova.conf then they should be defaulting to stacking behavior. I see and understand now that the default value 1.0 for [filter_scheduler]ram_weight_multiplier will spread VMs, but you have it correctly set to a negative value to instead stack VMs [1]. So, it seems by everything you have set up, you should be seeing stacking VM behavior. To debug further, you should set debug to True in the nova.conf on your scheduler host and look for which filter is removing the desired host for the second VM. You can find where to start by looking for a message like, "Starting with N host(s)". If you have two hosts with enough RAM, you should see "Starting with 2 host(s)" and then look for the log message where it says "Filter returned 1 host(s)" and that will be the filter that is removing the desired host. Once you know which filter is removing it, you can debug further. -melanie [1] https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights

...

...

On Wed, 17 Apr 2019 01:15:49 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote: Is there a way I can further debug this? Setting debug to "true" didn't show a reason why VMs are spreading and which filter is used every time which will make VMs spread. In addition I think that I 've read somewhere that placement does this (stack VMs) by default but have not seen that behavior at all... Any ideas appreciated... Regards, G. P.S: Added the "scheduler" tag to catch more attention from the correct people.

I had a look through the code and config options and have not yet found anything that could have caused the default stack behavior to have changed.

The related config options are:

* [placement]randomize_allocation_candidates which defaults to False * [filter_scheduler]host_subset_size which defaults to 1 * [filter_scheduler]shuffle_best_same_weighed_hosts which defaults to False

If you have these set to the defaults, you should be getting stacking behavior. Can you double check whether those options are set (or left as default) properly in your scheduler configs?

-melanie

...

...

All,

in the past I used to stack VMs based on RAM on hosts by putting the following settings in my nova.conf file:

->ram_weight_multiplier=-1.0 ->available_filters=nova.scheduler.filters.all_filters

->enabled_filters=RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter

I have now a Rocky installation and although I have set the above on my controller node and restarted corresponding services the result when spawning two VMs (one after another) is still to be distributed on two different hosts. Seems that the something is missing in order to override the default behavior.

Any suggestions please?

Best regards,

G.

Thank you both Melanie and Matt for trying to assist me. I have double checked the nova.conf at the controller and here is what I have (ignored hashed lines and obfuscating sensitive data): https://pastebin.com/hW1PE4U7 As you can see I have everything with default values as discussed before with Melanie except the filters and the weight that I have applied that should lead to VM stacking instead of spreading. My case scenario is with two compute hosts (let's call them "cpu1" and "cpu2") and when an instance is already placed on "cpu2" I expect the next instance to be placed also there. But instead is placed on "cpu1" as you can see from the scheduler log that can find here: https://pastebin.com/sCzB9L2e Do you see something strange that I fail to recognize? Best regards, G.

On 4/16/2019 7:03 PM, melanie witt wrote:

...

To debug further, you should set debug to True in the nova.conf on your scheduler host and look for which filter is removing the desired host for the second VM. You can find where to start by looking for a message like, "Starting with N host(s)". If you have two hosts with enough RAM, you should see "Starting with 2 host(s)" and then look for the log message where it says "Filter returned 1 host(s)" and that will be the filter that is removing the desired host. Once you know which filter is removing it, you can debug further.

If the other host isn't getting filtered out, it could be the weighers that aren't prioritizing the host you expect, but debug logs should dump the weighed hosts as well which might give a clue.

On 4/17/2019 2:19 PM, melanie witt wrote:

The relevant scheduler log is this one:

2019-04-17 19:53:07.303 98874 DEBUG nova.scheduler.filter_scheduler [req-02fb5504-cbdb-4219-9509-d2be9da7bb0e 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [(cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, (cpu2, cpu2) ram: 30105MB disk: 1886208MB io_ops: 0 instances: 1] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:455

and here we see that host 'cpu1' is being weighed ahead of host 'cpu2', which is the problem. I don't understand this considering the docs say that setting the ram_weight_multiplier to a negative value should result in the host with the lesser RAM being weighed higher/first. According to your log, the opposite is happening -- 'cpu1' with 32153MB RAM is being weighed higher than 'cpu2' with 30105MB RAM.

Either your ram_weight_multiplier setting is not being picked up or there's a bug causing weight to be applied with reverse logic?

Can you look at the scheduler debug log when the service first started up and verify what value of ram_weight_multiplier the service is using?

I agree with Melanie's assessment. Looking at the RAMWeigher code in Rocky we see it's weighing based on the free_ram_mb value in the HostState object: https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/weights/r... Looking at the filtered hosts that were logged: Filtered [(cpu2, cpu2) ram: 30105MB disk: 1886208MB io_ops: 0 instances: 1, (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0] The ram value that is logged is free_ram_mb: https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/host_mana... It looks like you also don't have this logging regression fix in your rocky scheduler code, so you might want to patch this in when getting new debug logs: https://review.openstack.org/#/c/641355/ That could tell us what the resulting weight is. -- Thanks, Matt

On Wed, 17 Apr 2019 22:45:45 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

Hello again Menalie!

Exactly this is what I am thinking...something is not working correctly!

To answer your questions there is one node acting as controller where the scheduler is running and I have pasted the nova.conf file from there.

I have also noticed that I have "ram_weight_multiplier" two times (one in [cells] and one in [filter_scheduler]) therefore I have removed the one in [cells] because I though it might give a problem but the results are still the same.

The log for the scheduler has this entry:

2019-04-17 22:04:50.045 131723 DEBUG oslo_service.service [req-7e548ecb-f3ed-4a4d-835f-b3a996e32534 - - - - -] filter_scheduler.ram_weight_multiplier = -1.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:3032

so it seems to be picked up correctly but without any influence.

Agreed, that log shows that the -1.0 value is being picked up properly by the scheduler service.

What also worries me from the scheduler log that I have send to you before is that in there I see an entry like this:

2019-04-17 19:53:07.298 98874 DEBUG nova.filters [req-02fb5504-cbdb-4219-9509-d2be9da7bb0e 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filter RamFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104

Shouldn't the RamFilter return 1host and the one with less RAM? Why does it return 2hosts??

If you have any other ideas or would like me to do some more checking I am all ears! At this point, you could take Matt's suggestion from his latest reply on

No -- the RamFilter will return any hosts that meet the RAM requirement. Filters do not weigh hosts. The RamFilter returns two hosts because both hosts have enough RAM to fulfill the request. FYI though, as of Pike [1], the (Core|Ram|Disk)Filter are redundant, as placement will do the filtering for those resources before the nova scheduler filters run. So you can safely remove (Core|Ram|Disk)Filter from your enabled_filters. [1] https://docs.openstack.org/releasenotes/nova/pike.html#relnotes-16-0-0-stabl... this thread and patch in the logging regression fix he linked. That would allow you to see in the debug log what weights nova is giving to the hosts. Aside from that, it's looking like we/I would need to reproduce this issue locally with a devstack and try to figure out what's causing this behavior. -melanie

...

...

Thank you both Melanie and Matt for trying to assist me. I have double checked the nova.conf at the controller and here is what I have (ignored hashed lines and obfuscating sensitive data): https://pastebin.com/hW1PE4U7 As you can see I have everything with default values as discussed before with Melanie except the filters and the weight that I have applied that should lead to VM stacking instead of spreading. My case scenario is with two compute hosts (let's call them "cpu1" and "cpu2") and when an instance is already placed on "cpu2" I expect the next instance to be placed also there. But instead is placed on "cpu1" as you can see from the scheduler log that can find here: https://pastebin.com/sCzB9L2e Do you see something strange that I fail to recognize?

Thanks for providing the helpful data. It appears you have set your nova.conf correctly (this is where your scheduler is running, yes?). I notice you have duplicated the ram_weight_multiplier setting but that shouldn't hurt anything.

The relevant scheduler log is this one:

2019-04-17 19:53:07.303 98874 DEBUG nova.scheduler.filter_scheduler [req-02fb5504-cbdb-4219-9509-d2be9da7bb0e 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [(cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, (cpu2, cpu2) ram: 30105MB disk: 1886208MB io_ops: 0 instances: 1] _get_sorted_hosts

/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:455

and here we see that host 'cpu1' is being weighed ahead of host 'cpu2', which is the problem. I don't understand this considering the docs say that setting the ram_weight_multiplier to a negative value should result in the host with the lesser RAM being weighed higher/first. According to your log, the opposite is happening -- 'cpu1' with 32153MB RAM is being weighed higher than 'cpu2' with 30105MB RAM.

Either your ram_weight_multiplier setting is not being picked up or there's a bug causing weight to be applied with reverse logic?

Can you look at the scheduler debug log when the service first started up and verify what value of ram_weight_multiplier the service is using?

-melanie

...

...

On 4/16/2019 7:03 PM, melanie witt wrote:

...

To debug further, you should set debug to True in the nova.conf on your scheduler host and look for which filter is removing the desired host for the second VM. You can find where to start by looking for a message like, "Starting with N host(s)". If you have two hosts with enough RAM, you should see "Starting with 2 host(s)" and then look for the log message where it says "Filter returned 1 host(s)" and that will be the filter that is removing the desired host. Once you know which filter is removing it, you can debug further.

If the other host isn't getting filtered out, it could be the weighers that aren't prioritizing the host you expect, but debug logs should dump the weighed hosts as well which might give a clue.

OK! I have applied the patch and now weights are shown! Furthermore as per your suggestion I have removed the "RamFilter" which was the only one present. And here is the new log where spawning of 2 VMs can be seen with a few seconds of difference: https://pastebin.com/Xy2FL2KL Initially both hosts are of weight 1.0 then the one with one VM already running has negative weight but the new VM is placed on the other host. Really-really strange why this is happening... G.

On Wed, 17 Apr 2019 12:55:45 -0700, Melanie Witt <melwittt@gmail.com> wrote:

...

...

Hello again Menalie!

Exactly this is what I am thinking...something is not working correctly!

To answer your questions there is one node acting as controller where the scheduler is running and I have pasted the nova.conf file from there.

I have also noticed that I have "ram_weight_multiplier" two times (one in [cells] and one in [filter_scheduler]) therefore I have removed the one in [cells] because I though it might give a problem but the results are still the same.

The log for the scheduler has this entry:

2019-04-17 22:04:50.045 131723 DEBUG oslo_service.service [req-7e548ecb-f3ed-4a4d-835f-b3a996e32534 - - - - -] filter_scheduler.ram_weight_multiplier = -1.0 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:3032

so it seems to be picked up correctly but without any influence. Agreed, that log shows that the -1.0 value is being picked up

On Wed, 17 Apr 2019 22:45:45 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote: properly by the scheduler service.

...

What also worries me from the scheduler log that I have send to you before is that in there I see an entry like this:

2019-04-17 19:53:07.298 98874 DEBUG nova.filters [req-02fb5504-cbdb-4219-9509-d2be9da7bb0e 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filter RamFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104

Shouldn't the RamFilter return 1host and the one with less RAM? Why does it return 2hosts?? No -- the RamFilter will return any hosts that meet the RAM requirement. Filters do not weigh hosts. The RamFilter returns two hosts because both hosts have enough RAM to fulfill the request. FYI though, as of Pike [1], the (Core|Ram|Disk)Filter are redundant, as placement will do the filtering for those resources before the nova scheduler filters run. So you can safely remove (Core|Ram|Disk)Filter from your enabled_filters. [1]

https://docs.openstack.org/releasenotes/nova/pike.html#relnotes-16-0-0-stabl...

...

If you have any other ideas or would like me to do some more checking I am all ears! At this point, you could take Matt's suggestion from his latest reply on this thread and patch in the logging regression fix he linked. That would allow you to see in the debug log what weights nova is giving to the hosts.

OK, so I just searched open nova bugs for "weigh" and found this issue, which isn't necessarily a defect:

https://bugs.launchpad.net/nova/+bug/1818239

but something that could be affecting the host weighing in your environment. There's something called the BuildFailureWeigher which will apply a low weight multiplier to hosts that have had VMs fail to build on them. And that weight resets when a host experiences a successful VM build.

If you apply the patch Matt suggested and take a look at the host weights, we should be able to see whether the BuildFailureWeigher is involved in the behavior you're seeing.

-melanie

...

Aside from that, it's looking like we/I would need to reproduce this issue locally with a devstack and try to figure out what's causing this behavior. -melanie

...

...

...

Thank you both Melanie and Matt for trying to assist me. I have double checked the nova.conf at the controller and here is what I have (ignored hashed lines and obfuscating sensitive data): https://pastebin.com/hW1PE4U7 As you can see I have everything with default values as discussed before with Melanie except the filters and the weight that I have applied that should lead to VM stacking instead of spreading. My case scenario is with two compute hosts (let's call them "cpu1" and "cpu2") and when an instance is already placed on "cpu2" I expect the next instance to be placed also there. But instead is placed on "cpu1" as you can see from the scheduler log that can find here: https://pastebin.com/sCzB9L2e Do you see something strange that I fail to recognize?

Thanks for providing the helpful data. It appears you have set your nova.conf correctly (this is where your scheduler is running, yes?). I notice you have duplicated the ram_weight_multiplier setting but that shouldn't hurt anything.

The relevant scheduler log is this one:

2019-04-17 19:53:07.303 98874 DEBUG nova.scheduler.filter_scheduler [req-02fb5504-cbdb-4219-9509-d2be9da7bb0e 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [(cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, (cpu2, cpu2) ram: 30105MB disk: 1886208MB io_ops: 0 instances: 1] _get_sorted_hosts

/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:455

and here we see that host 'cpu1' is being weighed ahead of host 'cpu2', which is the problem. I don't understand this considering the docs say that setting the ram_weight_multiplier to a negative value should result in the host with the lesser RAM being weighed higher/first. According to your log, the opposite is happening -- 'cpu1' with 32153MB RAM is being weighed higher than 'cpu2' with 30105MB RAM.

Either your ram_weight_multiplier setting is not being picked up or there's a bug causing weight to be applied with reverse logic?

Can you look at the scheduler debug log when the service first started up and verify what value of ram_weight_multiplier the service is using?

-melanie

...

...

On 4/16/2019 7:03 PM, melanie witt wrote: > To debug further, you should set debug to True in the nova.conf > on > your scheduler host and look for which filter is removing the > desired > host for the second VM. You can find where to start by looking > for > a > message like, "Starting with N host(s)". If you have two hosts > with > enough RAM, you should see "Starting with 2 host(s)" and then > look > for > the log message where it says "Filter returned 1 host(s)" and > that > will be the filter that is removing the desired host. Once you > know > which filter is removing it, you can debug further.

If the other host isn't getting filtered out, it could be the weighers that aren't prioritizing the host you expect, but debug logs should dump the weighed hosts as well which might give a clue.

On 4/17/2019 3:50 PM, Georgios Dimitrakakis wrote:

And here is the new log where spawning of 2 VMs can be seen with a few seconds of difference: https://pastebin.com/Xy2FL2KL

Initially both hosts are of weight 1.0 then the one with one VM already running has negative weight but the new VM is placed on the other host.

Really-really strange why this is happening...

< 2019-04-17 23:26:18.770 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filtered [(cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:435 < 2019-04-17 23:26:18.771 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [WeighedHost [host: (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, weight: 1.0], WeighedHost [host: (cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, weight: -0.00900862553213]] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:454 cpu1 is definitely getting weighed higher but I'm not sure why. We likely need some debug logging on the result of each weigher like we have for each filter to figure out what's going on with the weighers. -- Thanks, Matt

On Thu, 18 Apr 2019 01:13:42 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

Shouldn’t that be the correct behavior and place the new VM on the host with the smaller weight? Isn’t that what the negative value for “ram_weight_multiplier” does ?

No, it's the opposite, higher weights win. That's why you have to use a negative value for ram_weight_multiplier if you want hosts with _less_ RAM to win over hosts with more RAM (stacking).

Please let me know how I can provide to you more debug info....

One thing I noticed from your log is on the second request, 'cpu1' has io_ops: 0 whereas 'cpu2' has io_ops: 1 and the IoOpsWeigher [1] will prefer hosts with fewer io_iops by default. Note that's only one piece of the ending weight -- the weighing process will take a sum of all of the weights each weigher returns. So the weight returned from RamWeigher is added to the weight returned from IoOpsWeigher is added the weight returned from CPUWeigher, and so on. So, as Matt said, we're a bit in the dark now as far as what each weigher is returning and we don't currently have debug logging per weigher the way we do for filters. That would be an enhancement we could make to aid in debugging issues like this one. You could hack something up locally to log/print the returned weight in each weight class under the nova/scheduler/weights/ directory, if you want to dig into that. Another thing I noticed is that there are probably some new weighers available by default that did not exist in the previous version of nova that you were using in the past. By default, the config option for weighers: [filter_scheduler]weight_classes = ["nova.scheduler.weights.all_weighers"] will pick up all weigher classes in the nova/scheduler/weights/ code directory. You might take a look at these and see if any are ones you should exclude in your environment. For example, the CPUWeigher [1] (new in Rocky) will spread VMs based on available CPU by default. This weigher might be contributing to the VM spreading you're seeing. You might try playing with the '[filter_scheduler]weight_classes' config option to select only the weighers you want or alternatively you could set the weighers multipliers the way you prefer. -melanie [1] https://docs.openstack.org/nova/rocky/user/filter-scheduler.html#weights

...

...

On 4/17/2019 3:50 PM, Georgios Dimitrakakis wrote: And here is the new log where spawning of 2 VMs can be seen with a few seconds of difference: https://pastebin.com/Xy2FL2KL Initially both hosts are of weight 1.0 then the one with one VM already running has negative weight but the new VM is placed on the other host. Really-really strange why this is happening...

< 2019-04-17 23:26:18.770 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filtered [(cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:435

< 2019-04-17 23:26:18.771 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [WeighedHost [host: (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, weight: 1.0], WeighedHost [host: (cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, weight: -0.00900862553213]] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:454

cpu1 is definitely getting weighed higher but I'm not sure why. We likely need some debug logging on the result of each weigher like we have for each filter to figure out what's going on with the weighers.

--

Thanks,

Matt

On Thu, 18 Apr 2019 01:38:17 +0100, Sean Mooney <smooney@redhat.com> wrote:

...

On Thu, 18 Apr 2019 01:13:42 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

...

Shouldn’t that be the correct behavior and place the new VM on the host with the smaller weight? Isn’t that what the negative value for “ram_weight_multiplier” does ?

No, it's the opposite, higher weights win. That's why you have to use a negative value for ram_weight_multiplier if you want hosts with _less_ RAM to win over hosts with more RAM (stacking).

...

Please let me know how I can provide to you more debug info....

One thing I noticed from your log is on the second request, 'cpu1' has io_ops: 0 whereas 'cpu2' has io_ops: 1 and the IoOpsWeigher [1] will prefer hosts with fewer io_iops by default. Note that's only one piece of the ending weight -- the weighing process will take a sum of all of the weights each weigher returns. So the weight returned from RamWeigher is added to the weight returned from IoOpsWeigher is added the weight returned from CPUWeigher, and so on.

So, as Matt said, we're a bit in the dark now as far as what each weigher is returning and we don't currently have debug logging per weigher the way we do for filters. That would be an enhancement we could make to aid in debugging issues like this one. You could hack something up locally to log/print the returned weight in each weight class under the nova/scheduler/weights/ directory, if you want to dig into that.

Another thing I noticed is that there are probably some new weighers available by default that did not exist in the previous version of nova that you were using in the past. By default, the config option for weighers:

[filter_scheduler]weight_classes = ["nova.scheduler.weights.all_weighers"]

will pick up all weigher classes in the nova/scheduler/weights/ code directory. You might take a look at these and see if any are ones you should exclude in your environment. For example, the CPUWeigher [1] (new in Rocky) will spread VMs based on available CPU by default. most of the weighers spread by default so the cpu weigher may be a factor but

On Wed, 2019-04-17 at 15:30 -0700, melanie witt wrote: the the disk weigher tends to all hevily impact the choice.

we do not normalise any of the values retured by the different weighers the disk wighter is basically host_state.free_disk_mb * disk_weight_multiplier

Hm, I thought we do based on this code: https://github.com/openstack/nova/blob/stable/rocky/nova/weights.py#L135 which normalizes the weight to a value between 0.0 and 1.0. If the multiplier is large though, that could make the considered value

1.0 (as is the case with the default build_failure_weight_multiplier:

https://github.com/openstack/nova/blob/stable/rocky/nova/conf/scheduler.py#L... -melanie

althoer host_state.free_disk_mb is actully disk_available_least.

as a result the disk filter will weigh cpu1 19456 height then cpu2 the dela between the cpu1 and cpu2 based on the ram weigher is only 2048

if you want the ram filter to take presedence over teh disk filter you will need to scale the disk filter down to be in a similar value range

i woudl suggest setting disk_weight_multiplier=0.001

...

This weigher might be contributing to the VM spreading you're seeing. You might try playing with the '[filter_scheduler]weight_classes' config option to select only the weighers you want or alternatively you could set the weighers multipliers the way you prefer.

-melanie

[1] https://docs.openstack.org/nova/rocky/user/filter-scheduler.html#weights

...

...

...

On 4/17/2019 3:50 PM, Georgios Dimitrakakis wrote: And here is the new log where spawning of 2 VMs can be seen with a few seconds of difference: https://pastebin.com/Xy2FL2KL Initially both hosts are of weight 1.0 then the one with one VM already running has negative weight but the new VM is placed on the other host. Really-really strange why this is happening...

< 2019-04-17 23:26:18.770 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filtered [(cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:435

< 2019-04-17 23:26:18.771 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [WeighedHost [host: (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, weight: 1.0], WeighedHost [host: (cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, weight: -0.00900862553213]] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:454

cpu1 is definitely getting weighed higher but I'm not sure why. We likely need some debug logging on the result of each weigher like we have for each filter to figure out what's going on with the weighers.

--

Thanks,

Matt

Hello again and apologies for my absence! First of all let me express my gratitude for your valuable feedback. I understand what all of you are saying and I will try what Sean suggested by setting the values for CPUs and Disk to something beyond the default. Meanwhile what I 've though was to change the "weight_classes" parameter to use only the "RAM" weight instead of all, since initially this is what I would like to be based and then move on from there. Unfortunately I haven't found a way to properly set it and no matter what I 've tried I always ended up with an error in nova-scheduler.log saying "ERROR nova ValueError: Empty module name" Any ideas on how to set it based on the available weights? It seems that it needs a list format but how? Best, G.

On Wed, 2019-04-17 at 18:24 -0700, melanie witt wrote:

...

...

On Wed, 2019-04-17 at 15:30 -0700, melanie witt wrote:

...

On Thu, 18 Apr 2019 01:13:42 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

...

Shouldn’t that be the correct behavior and place the new VM on

...

...

...

the negative value for “ram_weight_multiplier” does ?

No, it's the opposite, higher weights win. That's why you have to use a negative value for ram_weight_multiplier if you want hosts with _less_ RAM to win over hosts with more RAM (stacking).

...

Please let me know how I can provide to you more debug info....

One thing I noticed from your log is on the second request, 'cpu1' has io_ops: 0 whereas 'cpu2' has io_ops: 1 and the IoOpsWeigher [1] will prefer hosts with fewer io_iops by default. Note that's only one

...

...

of the ending weight -- the weighing process will take a sum of all of the weights each weigher returns. So the weight returned from RamWeigher is added to the weight returned from IoOpsWeigher is added the weight returned from CPUWeigher, and so on.

So, as Matt said, we're a bit in the dark now as far as what each weigher is returning and we don't currently have debug logging

...

...

weigher the way we do for filters. That would be an enhancement we could make to aid in debugging issues like this one. You could hack something up locally to log/print the returned weight in each weight class under the nova/scheduler/weights/ directory, if you want to dig into

On Thu, 18 Apr 2019 01:38:17 +0100, Sean Mooney <smooney@redhat.com> wrote: the host with the smaller weight? Isn’t that what piece per that.

...

...

Another thing I noticed is that there are probably some new

weighers

...

available by default that did not exist in the previous version of nova that you were using in the past. By default, the config option for weighers:

[filter_scheduler]weight_classes = ["nova.scheduler.weights.all_weighers"]

will pick up all weigher classes in the nova/scheduler/weights/ code directory. You might take a look at these and see if any are ones you should exclude in your environment. For example, the CPUWeigher [1] (new in Rocky) will spread VMs based on available CPU by default.

most of the weighers spread by default so the cpu weigher may be a factor but the the disk weigher tends to all hevily impact the choice.

we do not normalise any of the values retured by the different weighers the disk wighter is basically host_state.free_disk_mb * disk_weight_multiplier

Hm, I thought we do based on this code:

https://github.com/openstack/nova/blob/stable/rocky/nova/weights.py#L135

which normalizes the weight to a value between 0.0 and 1.0. yes we do but that is witnin the same resouce type.

we do not normalise between resoucser types. on a typical host you will have between 2GB and 8GB of ram per cpu core and you will have between 10G and 100G of local disk typically.

so we dont look at the capasity of each resouce and renomalise between them. if you want achive that you have to carefully tweek the weights to do that manually.

that said i have not sat down and worked out the math to do that in about 3-4 years but i think i used to run my dev clouster with the disk_weight_multiplier around 0.04 and i think i used to set the ram_weight_multiplier to 2.0

anyway that is just my personal experience and i was trying to tweek the spreading behavior to spread based on ram then cpus then disk rather then pack based on ram but spread based on the rest. so i dont know if this is relvent or correct for this usecase.

...

If the multiplier is large though, that could make the considered value

...

1.0 (as is the case with the default build_failure_weight_multiplier:

https://github.com/openstack/nova/blob/stable/rocky/nova/conf/scheduler.py#L...

-melanie

...

althoer host_state.free_disk_mb is actully disk_available_least.

as a result the disk filter will weigh cpu1 19456 height then cpu2 the dela between the cpu1 and cpu2 based on the ram weigher is only 2048

if you want the ram filter to take presedence over teh disk filter you will need to scale the disk filter down to be in a similar value range

i woudl suggest setting disk_weight_multiplier=0.001

...

This weigher might be contributing to the VM spreading you're seeing. You might try playing with the '[filter_scheduler]weight_classes' config option to select only the weighers you want or alternatively you could set the weighers multipliers the way you prefer.

-melanie

[1] https://docs.openstack.org/nova/rocky/user/filter-scheduler.html#weights

...

...

> On 4/17/2019 3:50 PM, Georgios Dimitrakakis wrote: > And here is the new log where spawning of 2 VMs can be seen with a few seconds of difference: > https://pastebin.com/Xy2FL2KL > Initially both hosts are of weight 1.0 then the one with one VM already running has negative weight but the > new > VM is placed on the other host. > Really-really strange why this is happening...

< 2019-04-17 23:26:18.770 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e- 377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filtered [(cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:435

< 2019-04-17 23:26:18.771 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e- 377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [WeighedHost [host: (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, weight: 1.0], WeighedHost [host: (cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, weight: -0.00900862553213]] _get_sorted_hosts

/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:454

...

cpu1 is definitely getting weighed higher but I'm not sure

why. We likely need some debug logging on the

...

result of each weigher like we have for each filter to figure out what's going on with the weighers.

--

Thanks,

Matt

On Fri, 19 Apr 2019 22:47:23 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

Hello again and apologies for my absence!

First of all let me express my gratitude for your valuable feedback. I understand what all of you are saying and I will try what Sean suggested by setting the values for CPUs and Disk to something beyond the default.

Meanwhile what I 've though was to change the "weight_classes" parameter to use only the "RAM" weight instead of all, since initially this is what I would like to be based and then move on from there. Unfortunately I haven't found a way to properly set it and no matter what I 've tried I always ended up with an error in nova-scheduler.log saying "ERROR nova ValueError: Empty module name"

Any ideas on how to set it based on the available weights? It seems that it needs a list format but how?

Did you try: weight_classes = ['nova.scheduler.weights.ram.RAMWeigher'] That's what should work, based on the code I see here: https://github.com/openstack/nova/blob/stable/rocky/nova/loadables.py#L98 -melanie

...

On Wed, 2019-04-17 at 18:24 -0700, melanie witt wrote:

...

...

On Wed, 2019-04-17 at 15:30 -0700, melanie witt wrote:

...

On Thu, 18 Apr 2019 01:13:42 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

...

Shouldn’t that be the correct behavior and place the new VM on

...

...

...

the negative value for “ram_weight_multiplier” does ?

No, it's the opposite, higher weights win. That's why you have to use a negative value for ram_weight_multiplier if you want hosts with _less_ RAM to win over hosts with more RAM (stacking).

...

Please let me know how I can provide to you more debug info....

One thing I noticed from your log is on the second request, 'cpu1' has io_ops: 0 whereas 'cpu2' has io_ops: 1 and the IoOpsWeigher [1] will prefer hosts with fewer io_iops by default. Note that's only one

...

...

of the ending weight -- the weighing process will take a sum of all of the weights each weigher returns. So the weight returned from RamWeigher is added to the weight returned from IoOpsWeigher is added the weight returned from CPUWeigher, and so on.

So, as Matt said, we're a bit in the dark now as far as what each weigher is returning and we don't currently have debug logging

...

...

weigher the way we do for filters. That would be an enhancement we could make to aid in debugging issues like this one. You could hack something up locally to log/print the returned weight in each weight class under the nova/scheduler/weights/ directory, if you want to dig into

On Thu, 18 Apr 2019 01:38:17 +0100, Sean Mooney <smooney@redhat.com> wrote: the host with the smaller weight? Isn’t that what piece per that.

...

...

Another thing I noticed is that there are probably some new

weighers

...

available by default that did not exist in the previous version of nova that you were using in the past. By default, the config option for weighers:

[filter_scheduler]weight_classes = ["nova.scheduler.weights.all_weighers"]

will pick up all weigher classes in the nova/scheduler/weights/ code directory. You might take a look at these and see if any are ones you should exclude in your environment. For example, the CPUWeigher [1] (new in Rocky) will spread VMs based on available CPU by default.

most of the weighers spread by default so the cpu weigher may be a factor but the the disk weigher tends to all hevily impact the choice.

we do not normalise any of the values retured by the different weighers the disk wighter is basically host_state.free_disk_mb * disk_weight_multiplier

Hm, I thought we do based on this code:

https://github.com/openstack/nova/blob/stable/rocky/nova/weights.py#L135

which normalizes the weight to a value between 0.0 and 1.0. yes we do but that is witnin the same resouce type.

we do not normalise between resoucser types. on a typical host you will have between 2GB and 8GB of ram per cpu core and you will have between 10G and 100G of local disk typically.

so we dont look at the capasity of each resouce and renomalise between them. if you want achive that you have to carefully tweek the weights to do that manually.

that said i have not sat down and worked out the math to do that in about 3-4 years but i think i used to run my dev clouster with the disk_weight_multiplier around 0.04 and i think i used to set the ram_weight_multiplier to 2.0

anyway that is just my personal experience and i was trying to tweek the spreading behavior to spread based on ram then cpus then disk rather then pack based on ram but spread based on the rest. so i dont know if this is relvent or correct for this usecase.

...

If the multiplier is large though, that could make the considered value

...

1.0 (as is the case with the default build_failure_weight_multiplier:

https://github.com/openstack/nova/blob/stable/rocky/nova/conf/scheduler.py#L...

-melanie

...

althoer host_state.free_disk_mb is actully disk_available_least.

as a result the disk filter will weigh cpu1 19456 height then cpu2 the dela between the cpu1 and cpu2 based on the ram weigher is only 2048

if you want the ram filter to take presedence over teh disk filter you will need to scale the disk filter down to be in a similar value range

i woudl suggest setting disk_weight_multiplier=0.001

...

This weigher might be contributing to the VM spreading you're seeing. You might try playing with the '[filter_scheduler]weight_classes' config option to select only the weighers you want or alternatively you could set the weighers multipliers the way you prefer.

-melanie

[1] https://docs.openstack.org/nova/rocky/user/filter-scheduler.html#weights

...

>> On 4/17/2019 3:50 PM, Georgios Dimitrakakis wrote: >> And here is the new log where spawning of 2 VMs can be seen with a few seconds of difference: >> https://pastebin.com/Xy2FL2KL >> Initially both hosts are of weight 1.0 then the one with one VM already running has negative weight but the >> new >> VM is placed on the other host. >> Really-really strange why this is happening... > > < 2019-04-17 23:26:18.770 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e- > 377b3d37bff9 > 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filtered [(cpu2, cpu2) > ram: > 30105MB disk: 1887232MB io_ops: 1 instances: 1, (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: > 0] > _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:435 > > < 2019-04-17 23:26:18.771 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e- > 377b3d37bff9 > 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [WeighedHost > [host: > (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, weight: 1.0], WeighedHost [host: (cpu2, > cpu2) > ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, weight: -0.00900862553213]] _get_sorted_hosts > /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:454 > > cpu1 is definitely getting weighed higher but I'm not sure why. We likely need some debug logging on the > result of > each weigher like we have for each filter to figure out what's going on with the weighers. > > -- > > Thanks, > > Matt

On Sat, 20 Apr 2019 00:17:38 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

...

On Fri, 19 Apr 2019 22:47:23 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

...

Hello again and apologies for my absence! First of all let me express my gratitude for your valuable feedback. I understand what all of you are saying and I will try what Sean suggested by setting the values for CPUs and Disk to something beyond the default. Meanwhile what I 've though was to change the "weight_classes" parameter to use only the "RAM" weight instead of all, since initially this is what I would like to be based and then move on from there. Unfortunately I haven't found a way to properly set it and no matter what I 've tried I always ended up with an error in nova-scheduler.log saying "ERROR nova ValueError: Empty module name" Any ideas on how to set it based on the available weights? It seems that it needs a list format but how?

Did you try:

weight_classes = ['nova.scheduler.weights.ram.RAMWeigher']

That's what should work, based on the code I see here:

https://github.com/openstack/nova/blob/stable/rocky/nova/loadables.py#L98

-melanie

I 'll try that ASAP and get back to you. I believe I have failed to properly use CAPS letters.

Meanwhile, what I 've done is that by using Sean's suggestions about RAM, CPU and Disk like below:

ram_weight_multiplier=-1.0 cpu_weight_multiplier=-1.0 disk_weight_multiplier=0.001

I have managed to stack VMs on one node. VM stacking with the above is working correctly as long as no IOPS are involved because when they do weights are messed and VMs spread. But I assume that this can be dealt by playing with "io_ops_weight_multiplier" value.

So far so good if there wasn't for the following huge problem which is that stacking never stops even when resources are exceeded.

To explain what I mean I will give you an introduction of my hardware. My Compute Node has 1x Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8 core or 16 threads and 32GB of RAM.

Furthermore on nova.conf I have set:

cpu_allocation_ratio=1.0 ram_allocation_ratio=1.0

With the above settings I have been able to stack on this node 20 VMs with 2vCPUs each and 2GB of RAM. See the following excerpt (when spawning the 20th VM) from the scheduler log:

2019-04-19 23:26:42.322 379112 DEBUG nova.scheduler.filter_scheduler [req-73ddffc1-8cd2-4060-bd4c-786ba00793d1 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [WeighedHost [host: (cpu2, cpu2) ram: -6759MB disk: 1536000MB io_ops: 0 instances: 19, weight: 0.000805585392052], WeighedHost [host: (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, weight: -1.999]] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:454

To me is very clear that the aforementioned ratio settings are ignored and haven't tried what the limit would be (the final number of VMs that can be spawned).

Additionally I did try to perform this again but this time I set the following:

enabled_filters=RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter

This time only 15 VMs went to one compute node because on the 16th attempt the RAMFilter returned just one node (see log below):

2019-04-19 23:59:34.170 383851 DEBUG nova.filters [req-9cdd3c15-5bd8-4e06-bf0b-eb2d4d26d66d 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filter RamFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104

Melanie told me earlier on this thread that setting "enabled_filters" shouldn't make any difference but it is obvious from the above behavior that this is not the case.

In any case this is not a correct behavior since 15VMs x 2vCPUs/VM = 30vCPUs which is almost double the available 16cores, don't forget that I have ratio 1:1 for both CPU and RAM.

Does it seem logical to you?

I agree it sounds like your overcommit settings are not being honored. Placement will filter hosts taking overcommit into account, and if it doesn't have the values you set, then it will return more hosts than the RamFilter would filter, causing the difference in behavior. To debug this further, you need to start looking at what values placement has for the allocation ratios. You can do this by installing the osc-placement plugin [1] for the openstackclient. Once installed, you can do 'openstack resource provider inventory list <compute host UUID>' to see what allocation ratios are being used on a given compute host. If they're not 1.0, that would explain why it's returning too many hosts. If that's the case, next step would be trying to figure out why the values you're setting in nova.conf on the compute host (I assume you made sure to restart nova-compute after setting) are not being transferred to placement like they should be. -melanie [1] https://docs.openstack.org/osc-placement/rocky/cli

...

...

...

On Wed, 2019-04-17 at 18:24 -0700, melanie witt wrote:

...

...

On Wed, 2019-04-17 at 15:30 -0700, melanie witt wrote: > On Thu, 18 Apr 2019 01:13:42 +0300, Georgios Dimitrakakis > <giorgis@acmac.uoc.gr> wrote: >> Shouldn’t that be the correct behavior and place the new VM on

...

>> the >> negative value for “ram_weight_multiplier” does ? > > No, it's the opposite, higher weights win. That's why you have to use a > negative value for ram_weight_multiplier if you want hosts with _less_ > RAM to win over hosts with more RAM (stacking). > >> Please let me know how I can provide to you more debug info.... > > One thing I noticed from your log is on the second request, 'cpu1' has > io_ops: 0 whereas 'cpu2' has io_ops: 1 and the IoOpsWeigher [1] will > prefer hosts with fewer io_iops by default. Note that's only one

...

> of the ending weight -- the weighing process will take a sum of all of > the weights each weigher returns. So the weight returned from RamWeigher > is added to the weight returned from IoOpsWeigher is added the weight > returned from CPUWeigher, and so on. > > So, as Matt said, we're a bit in the dark now as far as what each > weigher is returning and we don't currently have debug logging

...

> weigher the way we do for filters. That would be an enhancement we could > make to aid in debugging issues like this one. You could hack something > up locally to log/print the returned weight in each weight class under > the nova/scheduler/weights/ directory, if you want to dig into

On Thu, 18 Apr 2019 01:38:17 +0100, Sean Mooney <smooney@redhat.com> wrote: the host with the smaller weight? Isn’t that what piece per that.

...

> > Another thing I noticed is that there are probably some new weighers > available by default that did not exist in the previous version of nova > that you were using in the past. By default, the config option for weighers: > > [filter_scheduler]weight_classes = ["nova.scheduler.weights.all_weighers"] > > will pick up all weigher classes in the nova/scheduler/weights/ code > directory. You might take a look at these and see if any are ones you > should exclude in your environment. For example, the CPUWeigher [1] (new > in Rocky) will spread VMs based on available CPU by default.

most of the weighers spread by default so the cpu weigher may be a factor but the the disk weigher tends to all hevily impact the choice.

we do not normalise any of the values retured by the different weighers the disk wighter is basically host_state.free_disk_mb * disk_weight_multiplier

Hm, I thought we do based on this code:

https://github.com/openstack/nova/blob/stable/rocky/nova/weights.py#L135

which normalizes the weight to a value between 0.0 and 1.0. yes we do but that is witnin the same resouce type.

we do not normalise between resoucser types. on a typical host you will have between 2GB and 8GB of ram per cpu core and you will have between 10G and 100G of local disk typically.

so we dont look at the capasity of each resouce and renomalise between them. if you want achive that you have to carefully tweek the weights to do that manually.

that said i have not sat down and worked out the math to do that in about 3-4 years but i think i used to run my dev clouster with the disk_weight_multiplier around 0.04 and i think i used to set the ram_weight_multiplier to 2.0

anyway that is just my personal experience and i was trying to tweek the spreading behavior to spread based on ram then cpus then disk rather then pack based on ram but spread based on the rest. so i dont know if this is relvent or correct for this usecase.

...

If the multiplier is large though, that could make the considered value

...

1.0 (as is the case with the default build_failure_weight_multiplier:

https://github.com/openstack/nova/blob/stable/rocky/nova/conf/scheduler.py#L...

-melanie

...

althoer host_state.free_disk_mb is actully disk_available_least.

as a result the disk filter will weigh cpu1 19456 height then cpu2 the dela between the cpu1 and cpu2 based on the ram weigher is only 2048

if you want the ram filter to take presedence over teh disk filter you will need to scale the disk filter down to be in a similar value range

i woudl suggest setting disk_weight_multiplier=0.001

> This > weigher might be contributing to the VM spreading you're seeing. You > might try playing with the '[filter_scheduler]weight_classes' config > option to select only the weighers you want or alternatively you could > set the weighers multipliers the way you prefer. > > -melanie > > [1]

https://docs.openstack.org/nova/rocky/user/filter-scheduler.html#weights

...

> >>>> On 4/17/2019 3:50 PM, Georgios Dimitrakakis wrote: >>>> And here is the new log where spawning of 2 VMs can be seen with a few seconds of difference: >>>> https://pastebin.com/Xy2FL2KL >>>> Initially both hosts are of weight 1.0 then the one with one VM already running has negative weight but the >>>> new >>>> VM is placed on the other host. >>>> Really-really strange why this is happening... >>> >>> < 2019-04-17 23:26:18.770 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e- >>> 377b3d37bff9 >>> 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filtered [(cpu2, cpu2) >>> ram: >>> 30105MB disk: 1887232MB io_ops: 1 instances: 1, (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: >>> 0] >>> _get_sorted_hosts

/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:435

...

>>> >>> < 2019-04-17 23:26:18.771 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e- >>> 377b3d37bff9 >>> 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [WeighedHost >>> [host: >>> (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, weight: 1.0], WeighedHost [host: (cpu2, >>> cpu2) >>> ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, weight: -0.00900862553213]] _get_sorted_hosts >>>

/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:454

...

>>> >>> cpu1 is definitely getting weighed higher but I'm not sure why. We likely need some debug logging on the >>> result of >>> each weigher like we have for each filter to figure out what's going on with the weighers. >>> >>> -- >>> >>> Thanks, >>> >>> Matt > > > > >

On Thu, 2019-04-18 at 01:38 +0100, Sean Mooney wrote:

On Wed, 2019-04-17 at 15:30 -0700, melanie witt wrote:

...

On Thu, 18 Apr 2019 01:13:42 +0300, Georgios Dimitrakakis <giorgis@acmac.uoc.gr> wrote:

...

Shouldn’t that be the correct behavior and place the new VM on the host with the smaller weight? Isn’t that what the negative value for “ram_weight_multiplier” does ?

No, it's the opposite, higher weights win. That's why you have to use a negative value for ram_weight_multiplier if you want hosts with _less_ RAM to win over hosts with more RAM (stacking).

...

Please let me know how I can provide to you more debug info....

One thing I noticed from your log is on the second request, 'cpu1' has io_ops: 0 whereas 'cpu2' has io_ops: 1 and the IoOpsWeigher [1] will prefer hosts with fewer io_iops by default. Note that's only one piece of the ending weight -- the weighing process will take a sum of all of the weights each weigher returns. So the weight returned from RamWeigher is added to the weight returned from IoOpsWeigher is added the weight returned from CPUWeigher, and so on.

So, as Matt said, we're a bit in the dark now as far as what each weigher is returning and we don't currently have debug logging per weigher the way we do for filters. That would be an enhancement we could make to aid in debugging issues like this one. You could hack something up locally to log/print the returned weight in each weight class under the nova/scheduler/weights/ directory, if you want to dig into that.

Another thing I noticed is that there are probably some new weighers available by default that did not exist in the previous version of nova that you were using in the past. By default, the config option for weighers:

[filter_scheduler]weight_classes = ["nova.scheduler.weights.all_weighers"]

will pick up all weigher classes in the nova/scheduler/weights/ code directory. You might take a look at these and see if any are ones you should exclude in your environment. For example, the CPUWeigher [1] (new in Rocky) will spread VMs based on available CPU by default.

most of the weighers spread by default so the cpu weigher may be a factor but the the disk weigher tends to all hevily impact the choice.

we do not normalise any of the values retured by the different weighers the disk wighter is basically host_state.free_disk_mb * disk_weight_multiplier

althoer host_state.free_disk_mb is actully disk_available_least.

as a result the disk filter will weigh cpu1 19456 height then cpu2 the dela between the cpu1 and cpu2 based on the ram weigher is only 2048

if you want the ram filter to take presedence over teh disk filter you will need to scale the disk filter down to be in a similar value range

i woudl suggest setting disk_weight_multiplier=0.001 actully based on https://github.com/openstack/nova/commit/e5ba8494374a1b049eae257fe05b10c5804... we do nomalise however i have had issues with this in the past where i have had to re addjsut the weights to effectivly renomalise the different weighers.

disk_weight_multiplier=0.001 is proably a little extream

...

This weigher might be contributing to the VM spreading you're seeing. You might try playing with the '[filter_scheduler]weight_classes' config option to select only the weighers you want or alternatively you could set the weighers multipliers the way you prefer.

-melanie

[1] https://docs.openstack.org/nova/rocky/user/filter-scheduler.html#weights

...

...

...

On 4/17/2019 3:50 PM, Georgios Dimitrakakis wrote: And here is the new log where spawning of 2 VMs can be seen with a few seconds of difference: https://pastebin.com/Xy2FL2KL Initially both hosts are of weight 1.0 then the one with one VM already running has negative weight but the new VM is placed on the other host. Really-really strange why this is happening...

< 2019-04-17 23:26:18.770 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Filtered [(cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:435

< 2019-04-17 23:26:18.771 157355 DEBUG nova.scheduler.filter_scheduler [req-14c666e4-3ff4-4d88-947e-377b3d37bff9 6a4c2e32919e4a6fa5c5d956beb68eef 9f22e9bfa7974e14871d58bbb62242b2 - default default] Weighed [WeighedHost [host: (cpu1, cpu1) ram: 32153MB disk: 1906688MB io_ops: 0 instances: 0, weight: 1.0], WeighedHost [host: (cpu2, cpu2) ram: 30105MB disk: 1887232MB io_ops: 1 instances: 1, weight: -0.00900862553213]] _get_sorted_hosts /usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py:454

cpu1 is definitely getting weighed higher but I'm not sure why. We likely need some debug logging on the result of each weigher like we have for each filter to figure out what's going on with the weighers.

--

Thanks,

Matt