<div dir="auto">Many thanks.<div dir="auto">I will check it</div><div dir="auto">Ignazio</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il Gio 13 Gen 2022, 19:13 Gorka Eguileor <<a href="mailto:geguileo@redhat.com">geguileo@redhat.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 13/01, Ignazio Cassano wrote:<br>
> Hellp Gorka,<br>
> here you can find more details:<br>
> <a href="https://paste.openstack.org/show/812091/" rel="noreferrer noreferrer" target="_blank">https://paste.openstack.org/show/812091/</a><br>
> Many thanks<br>
> Ignazio<br>
><br>
<br>
Hi,<br>
<br>
Given the reported data from the backends, which is:<br>
<br>
nfsgold1:<br>
max_over_subscription_ratio = 20.0<br>
total_capacity_gb = 1945.6<br>
free_capacity_gb = 609.68<br>
reserved_percentage = 0<br>
allocated_capacity_gb = 0<br>
<br>
nfsgold2:<br>
max_over_subscription_ratio = 20.0<br>
total_capacity_gb = 972.8<br>
free_capacity_gb = 970.36<br>
reserved_percentage = 0<br>
allocated_capacity_gb = 0<br>
<br>
Since those backends are not reporting the provisioned_capacity_gb, then<br>
it is assigned the same value as the allocated_capacity_gb, which is the<br>
sum of existing volumes in Cinder for that backend. I see it is<br>
reporting 0 here, so I assume you are using the same storage pool for<br>
multiple things and you still don't have volumes in Cinder.<br>
<br>
The calculation the scheduler does for the weighting is as follows:<br>
<br>
virtual-free-capacity = total_capacity_gb * max_over_subscription_ratio<br>
- provisioned_capacity<br>
- math.floor(total_capacity_gb * reserved_percentage)<br>
<br>
Which results in:<br>
<br>
nfsgold1 = 1945.6 * 20.0 - 0 - math.floor(1945.6 * 0) = 38,913.8<br>
<br>
nfsgold2 = 972.8 * 20.0 - 0 - math.floor(1945.6 * 0) = 19,456<br>
<br>
So nfsgold1 is returning a greater value, and therefore is winning the<br>
weighing, so only when there is no longer space in nfsgold1 and the<br>
filtering fails will nfsgold2 be used.<br>
<br>
If you look at the debug logs you should see that it describes which<br>
backends start the filtering, which ones pass each filter, and then<br>
which ones are weighed and which one wins.<br>
<br>
I see that the NetApp driver has a way to report the provisioned<br>
capacity (netapp_driver_reports_provisiones_capacity) that may be able<br>
to help you.<br>
<br>
Another way to resolve the issue may be to use an exclusive pool in the<br>
backend.<br>
<br>
Cheers,<br>
Gorka.<br>
<br>
> Il giorno gio 13 gen 2022 alle ore 13:08 Gorka Eguileor <<a href="mailto:geguileo@redhat.com" target="_blank" rel="noreferrer">geguileo@redhat.com</a>><br>
> ha scritto:<br>
><br>
> > On 13/01, Ignazio Cassano wrote:<br>
> > > Hello, I am using nfsgold volume type.<br>
> > > [root@tst-controller-01 ansible]# cinder type-show nfsgold<br>
> > ><br>
> > +---------------------------------+--------------------------------------+<br>
> > > | Property | Value<br>
> > |<br>
> > ><br>
> > +---------------------------------+--------------------------------------+<br>
> > > | description | None<br>
> > |<br>
> > > | extra_specs | volume_backend_name : nfsgold<br>
> > |<br>
> > > | id | fd8b1cc8-4c3a-490d-bc95-29e491f850cc<br>
> > |<br>
> > > | is_public | True<br>
> > |<br>
> > > | name | nfsgold<br>
> > |<br>
> > > | os-volume-type-access:is_public | True<br>
> > |<br>
> > > | qos_specs_id | None<br>
> > |<br>
> > ><br>
> > +---------------------------------+--------------------------------------+<br>
> > ><br>
> > > cinder get-pools<br>
> > ><br>
> > +----------+--------------------------------------------------------------------+<br>
> > > | Property | Value<br>
> > > |<br>
> > ><br>
> > +----------+--------------------------------------------------------------------+<br>
> > > | name | cinder-cluster-1@nfsgold2#<a href="http://10.102.189.156" rel="noreferrer noreferrer" target="_blank">10.102.189.156</a>:<br>
> > /svm_tstcinder_cl2_volssd<br>
> > > |<br>
> > ><br>
> > +----------+--------------------------------------------------------------------+<br>
> > ><br>
> > +----------+--------------------------------------------------------------------+<br>
> > > | Property | Value<br>
> > > |<br>
> > ><br>
> > +----------+--------------------------------------------------------------------+<br>
> > > | name | cinder-cluster-1@nfsgold1#<a href="http://10.102.189.155" rel="noreferrer noreferrer" target="_blank">10.102.189.155</a>:<br>
> > /svm_tstcinder_cl1_volssd<br>
> > > |<br>
> > ><br>
> > +----------+--------------------------------------------------------------------+<br>
> > ><br>
> ><br>
> > Hi,<br>
> ><br>
> > We would need to see the details of the pools to see additional<br>
> > information:<br>
> ><br>
> > $ cinder get-pools --detail<br>
> ><br>
> > > I noted that nfsgold2 is used also when nfsgold1 is almost full.<br>
> > > I expected the volume was created on share with more space availability.<br>
> > > Ignazio<br>
> > ><br>
> ><br>
> > Then the capacity filtering seems to be working as expected (we can<br>
> > confirm looking at the debug logs and seeing if both backends pass the<br>
> > filtering). You could see in the logs that both of them are passing the<br>
> > filtering and are valid to create volumes.<br>
> ><br>
> > The thing we'd have to look into is the weighing phase, where the<br>
> > scheduler is selecting nfsgold1 as the best option.<br>
> ><br>
> > I assume you haven't changed the defaults in the configuration options<br>
> > "scheduler_default_weighers" or in "scheduler_weight_handler".<br>
> ><br>
> > So it must be using the "CapacityWeigher". Are you using default values<br>
> > for "capacity_weigher_multiplier" and<br>
> > "allocated_capacity_weight_multiplier" config options?<br>
> ><br>
> > When using defaults the capacity weigher should be spread volumes<br>
> > instead of stacking them.<br>
> ><br>
> > I still think that the best way to debug this is to view the debug logs.<br>
> > In Stein you should be able to dynamically change the logging level of<br>
> > the scheduler services to debug without restarting the services, and<br>
> > then changing it back to info.<br>
> ><br>
> > Cheers,<br>
> > Gorka.<br>
> ><br>
> ><br>
> > ><br>
> > > Il giorno gio 13 gen 2022 alle ore 12:03 Gorka Eguileor <<br>
> > <a href="mailto:geguileo@redhat.com" target="_blank" rel="noreferrer">geguileo@redhat.com</a>><br>
> > > ha scritto:<br>
> > ><br>
> > > > On 13/01, Ignazio Cassano wrote:<br>
> > > > > Hello,<br>
> > > > > I am using openstack stein on centos 7 with netapp ontap driver.<br>
> > > > > Seems capacity filter is not working and volumes are always creed on<br>
> > the<br>
> > > > > first share where less space is available.<br>
> > > > > My configuration is posted here:<br>
> > > > > enabled_backends = nfsgold1, nfsgold2<br>
> > > > ><br>
> > > > > [nfsgold1]<br>
> > > > > nas_secure_file_operations = false<br>
> > > > > nas_secure_file_permissions = false<br>
> > > > > volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver<br>
> > > > > netapp_storage_family = ontap_cluster<br>
> > > > > netapp_storage_protocol = nfs<br>
> > > > > netapp_vserver = svm-tstcinder2-cl1<br>
> > > > > netapp_server_hostname = <a href="http://faspod2.csi.it" rel="noreferrer noreferrer" target="_blank">faspod2.csi.it</a><br>
> > > > > netapp_server_port = 80<br>
> > > > > netapp_login = apimanager<br>
> > > > > netapp_password = password<br>
> > > > > nfs_shares_config = /etc/cinder/nfsgold1_shares<br>
> > > > > volume_backend_name = nfsgold<br>
> > > > > #nfs_mount_options = lookupcache=pos<br>
> > > > > nfs_mount_options = lookupcache=pos<br>
> > > > ><br>
> > > > ><br>
> > > > > [nfsgold2]<br>
> > > > > nas_secure_file_operations = false<br>
> > > > > nas_secure_file_permissions = false<br>
> > > > > volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver<br>
> > > > > netapp_storage_family = ontap_cluster<br>
> > > > > netapp_storage_protocol = nfs<br>
> > > > > netapp_vserver = svm-tstcinder2-cl2<br>
> > > > > netapp_server_hostname = <a href="http://faspod2.csi.it" rel="noreferrer noreferrer" target="_blank">faspod2.csi.it</a><br>
> > > > > netapp_server_port = 80<br>
> > > > > netapp_login = apimanager<br>
> > > > > netapp_password = password<br>
> > > > > nfs_shares_config = /etc/cinder/nfsgold2_shares<br>
> > > > > volume_backend_name = nfsgold<br>
> > > > > #nfs_mount_options = lookupcache=pos<br>
> > > > > nfs_mount_options = lookupcache=pos<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > Volumes are created always on nfsgold1 also if has less space<br>
> > available<br>
> > > > of<br>
> > > > > nfsgold2 share<br>
> > > > > Thanks<br>
> > > > > Ignazio<br>
> > > ><br>
> > > > Hi,<br>
> > > ><br>
> > > > What volume type are you using to create the volumes? If you don't<br>
> > > > define it it would use the default from the cinder.conf file.<br>
> > > ><br>
> > > > What are the extra specs of the volume type?<br>
> > > ><br>
> > > > What pool info are the NetApp backends reporting?<br>
> > > ><br>
> > > > It's usually a good idea to enabled debugging on the schedulers and<br>
> > look<br>
> > > > at the details of how they are making the filtering and weighting<br>
> > > > decisions.<br>
> > > ><br>
> > > > Cheers,<br>
> > > > Gorka.<br>
> > > ><br>
> > > ><br>
> ><br>
> ><br>
<br>
</blockquote></div>