<div dir="ltr">It was my fault: there was a new Host Aggregate using filter_tenant_id which includes the 3 hosts<div>Apologies for the noise</div><div><br></div><div>Now I am going to study what Sean wrote :-)</div><div>Thanks, Massimo</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 30, 2022 at 1:27 PM Sean Mooney <<a href="mailto:smooney@redhat.com">smooney@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, 2022-05-30 at 10:44 +0200, Massimo Sgaravatto wrote:<br>
> Thanks Mohammed for your feedback<br>
> <br>
> This [*] is what I see in the log file. The 3 relevant nodes are filtered<br>
> out by the AggregateMultiTenancyIsolation (i.e. they are not among the 10<br>
> returned hosts)<br>
<br>
we havent actully chage the code that im aware of but<br>
one thing to note.<br>
it is considered bad pratice to use un namespaced flaovr extra specs with the<br>
AggregateInstanceExtraSpecsFilter or ComputeCapabilitiesFilter<br>
<br>
both have legacy support for unnamespced extra specs but you cannot enable both filters if you use that<br>
<br>
if you have both the AggregateInstanceExtraSpecsFilter and ComputeCapabilitiesFilter enabled<br>
you must use namespace custome extra specs.<br>
<br>
so aggregate_instance_extra_specs: and capabilities:<br>
<br>
you have both enabled below so "size=big" is not vaild a a flaovr extra spec in this configuration.<br>
<br>
yoga by the way the AggregateMultiTenancyIsolation filter is not needed anymore that can be done with placemnet<br>
<br>
<a href="https://github.com/openstack/nova/blob/stable/yoga/nova/scheduler/request_filter.py#L94-L135=" rel="noreferrer" target="_blank">https://github.com/openstack/nova/blob/stable/yoga/nova/scheduler/request_filter.py#L94-L135=</a><br>
<br>
by defiening <br>
[scheduler]<br>
limit_tenants_to_placement_aggregate=True<br>
placement_aggregate_required_for_tenants=True<br>
<br>
that was also possible in train<br>
<br>
we litrally ment to deprecate and remove teh AggregateMultiTenancyIsolatio a few years ago and never got around to it.<br>
it was the first filter implemetn as a placement prefilter in rocky<br>
<a href="https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-req-filter.html" rel="noreferrer" target="_blank">https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-req-filter.html</a><br>
so we problay should formally deprecate it this cycle and drop it in AA<br>
<br>
<a href="https://docs.openstack.org/nova/latest/admin/aggregates.html#tenant-isolation-with-placement" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/admin/aggregates.html#tenant-isolation-with-placement</a> explains in more relevent detail<br>
how to use placment for this. its basically a drop in replacement. you dont neeed to update teh aggreate metadta but there a<br>
re some subtlies with regards too what happens to host that are not mapped to any tenant which is why we have 2 config<br>
options so you can decide what you want to happen. i dont think there is any reason for an operator to ever use the<br>
AggregateMultiTenancyIsolation after rocky as the placment verions is much more effienct.<br>
<br>
<br>
you should alos proably read <a href="https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html</a><br>
while that is mainly inteded to replace the AggregateImagePropertiesIsolation it can also replace<br>
some of the usecasue enabeld by AggregateInstanceExtraSpecsFilter. simiarly due to bug 1677217<br>
<a href="https://bugs.launchpad.net/nova/+bug/1677217" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nova/+bug/1677217</a> you really shoudl not use the AggregateImagePropertiesIsolation<br>
any more and any enve that can use isolated aggreates should this however is not a drop in repalcemtn as it requires<br>
changes to the images properties/flavor extra specs to request the triats. this was intoduced in Train.<br>
<br>
<br>
> <br>
> In the Train --> Yoga updated we also removed the AvailabilityZoneFilter<br>
> (which has been deprecated): I tried to re-add it but this didn't help<br>
> I can't remember other changes that could be relevant with this issue<br>
> <br>
> Cheers, Massimo<br>
> <br>
> <br>
> [*]<br>
> <br>
> 2022-05-30 10:08:32.620 3168092 DEBUG nova.filters<br>
> [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669<br>
> 32b5d42c02b0411b8ebf2c3\<br>
> 3079eeecf - default default] Filtering removed all hosts for the request<br>
> with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results:<br>
> [('\<br>
> AggregateMultiTenancyIsolation', [('<a href="http://cld-blu-12.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-12.cloud.pd.infn.it</a>', '<br>
> <a href="http://cld-blu-12.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-12.cloud.pd.infn.it</a>'), ('<a href="http://cld-blu-02.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-02.cloud.pd.infn.it</a>',<br>
> 'cld-blu-02.cloud.p\<br>
> <a href="http://d.infn.it" rel="noreferrer" target="_blank">d.infn.it</a>'), ('<a href="http://cld-blu-15.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-15.cloud.pd.infn.it</a>', '<a href="http://cld-blu-15.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-15.cloud.pd.infn.it</a>'),<br>
> ('<a href="http://cld-blu-11.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-11.cloud.pd.infn.it</a>', '<a href="http://cld-blu-11.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-11.cloud.pd.infn.it</a>'), ('cld-bl\<br>
> <a href="http://u-14.cloud.pd.infn.it" rel="noreferrer" target="_blank">u-14.cloud.pd.infn.it</a>', '<a href="http://cld-blu-14.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-14.cloud.pd.infn.it</a>'), ('<br>
> <a href="http://cld-blu-13.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-13.cloud.pd.infn.it</a>', '<a href="http://cld-blu-13.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-13.cloud.pd.infn.it</a>'), ('<br>
> <a href="http://cld-blu-07.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-07.cloud.pd.infn.it</a>\<br>
> ', '<a href="http://cld-blu-07.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-07.cloud.pd.infn.it</a>'), ('<a href="http://cld-blu-06.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-06.cloud.pd.infn.it</a>', '<br>
> <a href="http://cld-blu-06.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-06.cloud.pd.infn.it</a>'), ('<a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a>',<br>
> 'cld-blu-01.cloud.\<br>
> <a href="http://pd.infn.it" rel="noreferrer" target="_blank">pd.infn.it</a>'), ('<a href="http://cld-blu-16.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-16.cloud.pd.infn.it</a>', '<a href="http://cld-blu-16.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-16.cloud.pd.infn.it</a>')]),<br>
> ('AggregateInstanceExtraSpecsFilter', None)] get_filtered_objects /us\<br>
> r/lib/python3.6/site-packages/nova/filters.py:114<br>
> 2022-05-30 10:08:32.620 3168092 INFO nova.filters<br>
> [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669<br>
> 32b5d42c02b0411b8ebf2c33\<br>
> 079eeecf - default default] Filtering removed all hosts for the request<br>
> with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results:<br>
> ['Ag\<br>
> gregateMultiTenancyIsolation: (start: 58, end: 10)',<br>
> 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']<br>
> <br>
> On Mon, May 30, 2022 at 9:51 AM Mohammed Naser <<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>> wrote:<br>
> <br>
> > Hi there,<br>
> > <br>
> > Are you sure you haven't done any other changes to your environment?<br>
> > Both those filters haven't been changed for years (~2017):<br>
> > <br>
> > <br>
> > <a href="https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_multitenancy_isolation.py" rel="noreferrer" target="_blank">https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_multitenancy_isolation.py</a><br>
> > <br>
> > <a href="https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_instance_extra_specs.py" rel="noreferrer" target="_blank">https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_instance_extra_specs.py</a><br>
> > <br>
> > I think you've got something else here, I'd suggest enabling debug and<br>
> > checking why these nodes are being filtered out.<br>
> > <br>
> > Mohammed<br>
> > <br>
> > On Mon, May 30, 2022 at 9:39 AM Massimo Sgaravatto<br>
> > <<a href="mailto:massimo.sgaravatto@gmail.com" target="_blank">massimo.sgaravatto@gmail.com</a>> wrote:<br>
> > > <br>
> > > It looks like I need now to create a HostAggregate for "size=big" for<br>
> > each project, specifying as properties:<br>
> > > filter_tenant_id=<projectid>, size='big'<br>
> > > <br>
> > > Is this the expected behaviour ?<br>
> > > <br>
> > > Till Train a single HostAggregate with the property: size='big' was<br>
> > enough<br>
> > > <br>
> > > Thanks, Massimo<br>
> > > <br>
> > > <br>
> > > <br>
> > > On Fri, May 27, 2022 at 10:08 AM Massimo Sgaravatto <<br>
> > <a href="mailto:massimo.sgaravatto@gmail.com" target="_blank">massimo.sgaravatto@gmail.com</a>> wrote:<br>
> > > > <br>
> > > > Dear all<br>
> > > > <br>
> > > > We have the following use case:<br>
> > > > <br>
> > > > - reserve 3 hypervisors for VMs with "big" flavors (whatever the users<br>
> > of these instances are)<br>
> > > > - partition the rest of the hypervisors according to the project (so<br>
> > projects A1,A2..,An can use only subset S1 of hypervisors, project<br>
> > B1,B2,..,Bm can use only subset S2 of hypervisors)<br>
> > > > <br>
> > > > We implemented this:<br>
> > > > <br>
> > > > 1- by setting an aggregate_instance_extra_spec 'size' property (with<br>
> > value 'normal' or 'big') for each flavor [*]<br>
> > > > 2- by creating a BigVMs HostAggregate for size=big [**]<br>
> > > > 3- by creating an HostAggregate for size=normal for each project, such<br>
> > as this one [***]<br>
> > > > <br>
> > > > <br>
> > > > This used to work.<br>
> > > > <br>
> > > > A few days ago we updated our infrastructure from Train to Yoga<br>
> > > > This was an offline Fast Forward Update: we went through the<br>
> > intermediate releases just to do the dbsyncs.<br>
> > > > Since this update the instantiation of VMs with flavors with the<br>
> > size=big property doesn't work anymore<br>
> > > > <br>
> > > > <br>
> > > > This is what I see in nova-scheduler log:<br>
> > > > 2022-05-27 08:38:02.058 5273 INFO nova.filters<br>
> > [req-f92c0e38-262a-4d22-a7fd-8874c4265401 e237e43716fb490db5bda4b777835669<br>
> > 32b5d42c02b0411b8ebf2c33079eeecf - default default] Filtering removed all<br>
> > hosts for the request with instance ID<br>
> > '2412e188-9d5f-4812-ad21-195769a3c220'. Filter results:<br>
> > ['AggregateMultiTenancyIsolation: (start: 59, end: 10)',<br>
> > 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']<br>
> > > > <br>
> > > > <br>
> > > > Only modifying the property of the BigVMs HA using the<br>
> > filter_tenant_id adding the relevant project:<br>
> > > > <br>
> > > > [root@cld-ctrl-01 ~]# openstack aggregate show BigVMs | grep prop<br>
> > > > > properties |<br>
> > filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='big'<br>
> > |<br>
> > > > <br>
> > > > the scheduling works<br>
> > > > <br>
> > > > Specifying each project and keeping the list up-to-date would be a<br>
> > problem. Moreover if I am not wrong there is a maximum length for the<br>
> > property field.<br>
> > > > <br>
> > > > Any hints ?<br>
> > > > I didn't find anything related to this issue in the nova release notes<br>
> > for Openstack releases > Train<br>
> > > > <br>
> > > > <br>
> > > > These are the filters that we enabled:<br>
> > > > <br>
> > > > [filter_scheduler]<br>
> > > > enabled_filters =<br>
> > AggregateMultiTenancyIsolation,AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGro\<br>
> > > > upAffinityFilter,PciPassthroughFilter,NUMATopologyFilter<br>
> > > > <br>
> > > > <br>
> > > > Thanks a lot, Massimo<br>
> > > > <br>
> > > > <br>
> > > > [*]<br>
> > > > E.g.<br>
> > > > [root@cld-ctrl-01 ~]# openstack flavor show cldareapd.medium | grep<br>
> > prope<br>
> > > > > properties |<br>
> > aggregate_instance_extra_specs:size='normal' |[root@cld-ctrl-01 ~]#<br>
> > openstack flavor show cloudvenetocloudveneto.40cores128GB25-bigunipd | grep<br>
> > prop<br>
> > > > > properties |<br>
> > aggregate_instance_extra_specs:size='big' |<br>
> > > > <br>
> > > > <br>
> > > > [**]<br>
> > > > <br>
> > > > [root@cld-ctrl-01 ~]# openstack aggregate show BigVMs<br>
> > > > <br>
> > +-------------------+---------------------------------------------------------------------------------------+<br>
> > > > > Field | Value<br>
> > |<br>
> > > > <br>
> > +-------------------+---------------------------------------------------------------------------------------+<br>
> > > > > availability_zone | nova<br>
> > |<br>
> > > > > created_at | 2018-06-20T06:54:51.000000<br>
> > |<br>
> > > > > deleted_at | None<br>
> > |<br>
> > > > > hosts | <a href="http://cld-blu-08.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-08.cloud.pd.infn.it</a>,<br>
> > <a href="http://cld-blu-09.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-09.cloud.pd.infn.it</a>, <a href="http://cld-blu-10.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-10.cloud.pd.infn.it</a> |<br>
> > > > > id | 135<br>
> > |<br>
> > > > > is_deleted | False<br>
> > |<br>
> > > > > name | BigVMs<br>
> > |<br>
> > > > > properties |size='big' |<br>
> > > > > updated_at | None<br>
> > |<br>
> > > > > uuid | 4b593395-1c76-441c-9022-d421f4ea2dfb<br>
> > |<br>
> > > > <br>
> > +-------------------+---------------------------------------------------------------------------------------+<br>
> > > > <br>
> > > > <br>
> > > > [***]<br>
> > > > [root@cld-ctrl-01 ~]# openstack aggregate show Unipd-AdminTesting-Unipd<br>
> > > > <br>
> > +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>
> > > > > Field | Value<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > <br>
> > +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>
> > > > > availability_zone | nova<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > > created_at | 2018-04-12T07:31:01.000000<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > > deleted_at | None<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > > hosts | <a href="http://cld-blu-01.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-01.cloud.pd.infn.it</a>,<br>
> > <a href="http://cld-blu-02.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-02.cloud.pd.infn.it</a>, <a href="http://cld-blu-05.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-05.cloud.pd.infn.it</a>,<br>
> > <a href="http://cld-blu-06.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-06.cloud.pd.infn.it</a>, <a href="http://cld-blu-07.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-07.cloud.pd.infn.it</a>,<br>
> > <a href="http://cld-blu-11.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-11.cloud.pd.infn.it</a>, <a href="http://cld-blu-12.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-12.cloud.pd.infn.it</a>,<br>
> > <a href="http://cld-blu-13.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-13.cloud.pd.infn.it</a>, <a href="http://cld-blu-14.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-14.cloud.pd.infn.it</a>,<br>
> > <a href="http://cld-blu-15.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-15.cloud.pd.infn.it</a>, <a href="http://cld-blu-16.cloud.pd.infn.it" rel="noreferrer" target="_blank">cld-blu-16.cloud.pd.infn.it</a> |<br>
> > > > > id | 126<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > > is_deleted | False<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > > name | Unipd-AdminTesting-Unipd<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > > properties |<br>
> > filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='normal'<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > > updated_at | 2018-06-08T09:06:20.000000<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > > uuid | 38f6a0d4-77ab-42e0-abeb-57e06ba13cca<br>
> > <br>
> > <br>
> > <br>
> > |<br>
> > > > <br>
> > +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br>
> > > > [root@cld-ctrl-01 ~]#<br>
> > > > <br>
> > <br>
> > <br>
> > --<br>
> > Mohammed Naser<br>
> > VEXXHOST, Inc.<br>
> > <br>
<br>
</blockquote></div>