<div dir="ltr"><div>Thanks Mohammed for your feedback</div><div><br></div><div>This [*] is what I see in the log file. The 3 relevant nodes are filtered out by the AggregateMultiTenancyIsolation (i.e. they are not among the 10 returned hosts)</div><div><br></div><div>In the Train --> Yoga updated we also removed the AvailabilityZoneFilter (which has been deprecated): I tried to re-add it but this didn't help</div><div>I can't remember other changes that could be relevant with this issue</div><div><br></div><div>Cheers, Massimo</div><div><br></div><div><br></div><div>[*]</div><div dir="ltr"><br></div><div dir="ltr">2022-05-30 10:08:32.620 3168092 DEBUG nova.filters [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669 32b5d42c02b0411b8ebf2c3\<br>3079eeecf - default default] Filtering removed all hosts for the request with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results: [('\<br>AggregateMultiTenancyIsolation', [('<a href="http://cld-blu-12.cloud.pd.infn.it" target="_blank">cld-blu-12.cloud.pd.infn.it</a>', '<a href="http://cld-blu-12.cloud.pd.infn.it" target="_blank">cld-blu-12.cloud.pd.infn.it</a>'), ('<a href="http://cld-blu-02.cloud.pd.infn.it" target="_blank">cld-blu-02.cloud.pd.infn.it</a>', 'cld-blu-02.cloud.p\<br><a href="http://d.infn.it" target="_blank">d.infn.it</a>'), ('<a href="http://cld-blu-15.cloud.pd.infn.it" target="_blank">cld-blu-15.cloud.pd.infn.it</a>', '<a href="http://cld-blu-15.cloud.pd.infn.it" target="_blank">cld-blu-15.cloud.pd.infn.it</a>'), ('<a href="http://cld-blu-11.cloud.pd.infn.it" target="_blank">cld-blu-11.cloud.pd.infn.it</a>', '<a href="http://cld-blu-11.cloud.pd.infn.it" target="_blank">cld-blu-11.cloud.pd.infn.it</a>'), ('cld-bl\<br><a href="http://u-14.cloud.pd.infn.it" target="_blank">u-14.cloud.pd.infn.it</a>', '<a href="http://cld-blu-14.cloud.pd.infn.it" target="_blank">cld-blu-14.cloud.pd.infn.it</a>'), ('<a href="http://cld-blu-13.cloud.pd.infn.it" target="_blank">cld-blu-13.cloud.pd.infn.it</a>', '<a href="http://cld-blu-13.cloud.pd.infn.it" target="_blank">cld-blu-13.cloud.pd.infn.it</a>'), ('<a href="http://cld-blu-07.cloud.pd.infn.it" target="_blank">cld-blu-07.cloud.pd.infn.it</a>\<br>', '<a href="http://cld-blu-07.cloud.pd.infn.it" target="_blank">cld-blu-07.cloud.pd.infn.it</a>'), ('<a href="http://cld-blu-06.cloud.pd.infn.it" target="_blank">cld-blu-06.cloud.pd.infn.it</a>', '<a href="http://cld-blu-06.cloud.pd.infn.it" target="_blank">cld-blu-06.cloud.pd.infn.it</a>'), ('<a href="http://cld-blu-01.cloud.pd.infn.it" target="_blank">cld-blu-01.cloud.pd.infn.it</a>', 'cld-blu-01.cloud.\<br><a href="http://pd.infn.it" target="_blank">pd.infn.it</a>'), ('<a href="http://cld-blu-16.cloud.pd.infn.it" target="_blank">cld-blu-16.cloud.pd.infn.it</a>', '<a href="http://cld-blu-16.cloud.pd.infn.it" target="_blank">cld-blu-16.cloud.pd.infn.it</a>')]), ('AggregateInstanceExtraSpecsFilter', None)] get_filtered_objects /us\<br>r/lib/python3.6/site-packages/nova/filters.py:114<br>2022-05-30 10:08:32.620 3168092 INFO nova.filters [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669 32b5d42c02b0411b8ebf2c33\<br>079eeecf - default default] Filtering removed all hosts for the request with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results: ['Ag\<br>gregateMultiTenancyIsolation: (start: 58, end: 10)', 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 30, 2022 at 9:51 AM Mohammed Naser <<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi there,<br>
<br>
Are you sure you haven't done any other changes to your environment?<br>
Both those filters haven't been changed for years (~2017):<br>
<br>
<a href="https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_multitenancy_isolation.py" rel="noreferrer" target="_blank">https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_multitenancy_isolation.py</a><br>
<a href="https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_instance_extra_specs.py" rel="noreferrer" target="_blank">https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_instance_extra_specs.py</a><br>
<br>
I think you've got something else here, I'd suggest enabling debug and<br>
checking why these nodes are being filtered out.<br>
<br>
Mohammed<br>
<br>
On Mon, May 30, 2022 at 9:39 AM Massimo Sgaravatto<br>
<<a href="mailto:massimo.sgaravatto@gmail.com" target="_blank">massimo.sgaravatto@gmail.com</a>> wrote:<br>
><br>
> It looks like I need now to create a HostAggregate for "size=big" for each project, specifying as properties:<br>
> filter_tenant_id=<projectid>, size='big'<br>
><br>
> Is this the expected behaviour ?<br>
><br>
> Till Train a single HostAggregate with the property: size='big' was enough<br>
><br>
> Thanks, Massimo<br>
><br>
><br>
><br>
> On Fri, May 27, 2022 at 10:08 AM Massimo Sgaravatto <<a href="mailto:massimo.sgaravatto@gmail.com" target="_blank">massimo.sgaravatto@gmail.com</a>> wrote:<br>
>><br>
>> Dear all<br>
>><br>
>> We have the following use case:<br>
>><br>
>> - reserve 3 hypervisors for VMs with "big" flavors (whatever the users of these instances are)<br>
>> - partition the rest of the hypervisors according to the project (so projects A1,A2..,An can use only subset S1 of hypervisors, project B1,B2,..,Bm can use only subset S2 of hypervisors)<br>
>><br>
>> We implemented this:<br>
>><br>
>> 1- by setting an aggregate_instance_extra_spec 'size' property (with value 'normal' or 'big') for each flavor [*]<br>
>> 2- by creating a BigVMs HostAggregate for size=big [**]<br>
>> 3- by creating an HostAggregate for size=normal for each project, such as this one [***]<br>
>><br>
>><br>
>> This used to work.<br>
>><br>
>> A few days ago we updated our infrastructure from Train to Yoga<br>
>> This was an offline Fast Forward Update: we went through the intermediate releases just to do the dbsyncs.<br>
>> Since this update the instantiation of VMs with flavors with the size=big property doesn't work anymore<br>
>><br>
>><br>
>> This is what I see in nova-scheduler log:<br>
>> 2022-05-27 08:38:02.058 5273 INFO nova.filters [req-f92c0e38-262a-4d22-a7fd-8874c4265401 e237e43716fb490db5bda4b777835669 32b5d42c02b0411b8ebf2c33079eeecf - default default] Filtering removed all hosts for the request with instance ID '2412e188-9d5f-4812-ad21-195769a3c220'. Filter results: ['AggregateMultiTenancyIsolation: (start: 59, end: 10)', 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']<br>
>><br>
>><br>
>> Only modifying the property of the BigVMs HA using the filter_tenant_id adding the relevant project:<br>
>><br>
>> [root@cld-ctrl-01 ~]# openstack aggregate show BigVMs | grep prop<br>
>> | properties | filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='big' |<br>
>><br>
>> the scheduling works<br>
>><br>
>> Specifying each project and keeping the list up-to-date would be a problem. Moreover if I am not wrong there is a maximum length for the property field.<br>
>><br>
>> Any hints ?<br>
>> I didn't find anything related to this issue in the nova release notes for Openstack releases > Train<br>
>><br>
>><br>
>> These are the filters that we enabled:<br>
>><br>
>> [filter_scheduler]<br>
>> enabled_filters = AggregateMultiTenancyIsolation,AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGro\<br>
>> upAffinityFilter,PciPassthroughFilter,NUMATopologyFilter<br>
>><br>
>><br>
>> Thanks a lot, Massimo<br>
>><br>
>><br>
>> [*]<br>
>> E.g.<br>
>> [root@cld-ctrl-01 ~]# openstack flavor show cldareapd.medium | grep prope<br>
>> | properties | aggregate_instance_extra_specs:size='normal' |[root@cld-ctrl-01 ~]# openstack flavor show cloudvenetocloudveneto.40cores128GB25-bigunipd | grep prop<br>
>> | properties | aggregate_instance_extra_specs:size='big' |<br>
>><br>
>><br>
>> [**]<br>
>><br>
>> [root@cld-ctrl-01 ~]# openstack aggregate show BigVMs<br>
>> +-------------------+---------------------------------------------------------------------------------------+<br>
>> | Field | Value |<br>
>> +-------------------+---------------------------------------------------------------------------------------+<br>
>> | availability_zone | nova |<br>
>> | created_at | 2018-06-20T06:54:51.000000 |<br>
>> | deleted_at | None |<br>
>> | hosts | <a href="http://cld-blu-08.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-08.cloud.pd.infn.it</a>, <a href="http://cld-blu-09.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-09.cloud.pd.infn.it</a>, <a href="http://cld-blu-10.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-10.cloud.pd.infn.it</a> |<br>
>> | id | 135 |<br>
>> | is_deleted | False |<br>
>> | name | BigVMs |<br>
>> | properties |size='big' |<br>
>> | updated_at | None |<br>
>> | uuid | 4b593395-1c76-441c-9022-d421f4ea2dfb |<br>
>> +-------------------+---------------------------------------------------------------------------------------+<br>
>><br>
>><br>
>> [***]<br>
>> [root@cld-ctrl-01 ~]# openstack aggregate show Unipd-AdminTesting-Unipd<br>
>> +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>
>> | Field | Value |<br>
>> +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>
>> | availability_zone | nova |<br>
>> | created_at | 2018-04-12T07:31:01.000000 |<br>
>> | deleted_at | None |<br>
>> | hosts | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a>, <a href="http://cld-blu-02.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-02.cloud.pd.infn.it</a>, <a href="http://cld-blu-05.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-05.cloud.pd.infn.it</a>, <a href="http://cld-blu-06.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-06.cloud.pd.infn.it</a>, <a href="http://cld-blu-07.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-07.cloud.pd.infn.it</a>, <a href="http://cld-blu-11.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-11.cloud.pd.infn.it</a>, <a href="http://cld-blu-12.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-12.cloud.pd.infn.it</a>, <a href="http://cld-blu-13.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-13.cloud.pd.infn.it</a>, <a href="http://cld-blu-14.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-14.cloud.pd.infn.it</a>, <a href="http://cld-blu-15.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-15.cloud.pd.infn.it</a>, <a href="http://cld-blu-16.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-16.cloud.pd.infn.it</a> |<br>
>> | id | 126 |<br>
>> | is_deleted | False |<br>
>> | name | Unipd-AdminTesting-Unipd |<br>
>> | properties | filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='normal' |<br>
>> | updated_at | 2018-06-08T09:06:20.000000 |<br>
>> | uuid | 38f6a0d4-77ab-42e0-abeb-57e06ba13cca |<br>
>> +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>
>> [root@cld-ctrl-01 ~]#<br>
>><br>
<br>
<br>
-- <br>
Mohammed Naser<br>
VEXXHOST, Inc.<br>
</blockquote></div></div>