On Mon, 2022-05-30 at 10:44 +0200, Massimo Sgaravatto wrote:
Thanks Mohammed for your feedback
This [*] is what I see in the log file. The 3 relevant nodes are filtered out by the AggregateMultiTenancyIsolation (i.e. they are not among the 10 returned hosts)
we havent actully chage the code that im aware of but one thing to note. it is considered bad pratice to use un namespaced flaovr extra specs with the AggregateInstanceExtraSpecsFilter or ComputeCapabilitiesFilter both have legacy support for unnamespced extra specs but you cannot enable both filters if you use that if you have both the AggregateInstanceExtraSpecsFilter and ComputeCapabilitiesFilter enabled you must use namespace custome extra specs. so aggregate_instance_extra_specs: and capabilities: you have both enabled below so "size=big" is not vaild a a flaovr extra spec in this configuration. yoga by the way the AggregateMultiTenancyIsolation filter is not needed anymore that can be done with placemnet https://github.com/openstack/nova/blob/stable/yoga/nova/scheduler/request_fi... by defiening [scheduler] limit_tenants_to_placement_aggregate=True placement_aggregate_required_for_tenants=True that was also possible in train we litrally ment to deprecate and remove teh AggregateMultiTenancyIsolatio a few years ago and never got around to it. it was the first filter implemetn as a placement prefilter in rocky https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/pla... so we problay should formally deprecate it this cycle and drop it in AA https://docs.openstack.org/nova/latest/admin/aggregates.html#tenant-isolatio... explains in more relevent detail how to use placment for this. its basically a drop in replacement. you dont neeed to update teh aggreate metadta but there a re some subtlies with regards too what happens to host that are not mapped to any tenant which is why we have 2 config options so you can decide what you want to happen. i dont think there is any reason for an operator to ever use the AggregateMultiTenancyIsolation after rocky as the placment verions is much more effienct. you should alos proably read https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html while that is mainly inteded to replace the AggregateImagePropertiesIsolation it can also replace some of the usecasue enabeld by AggregateInstanceExtraSpecsFilter. simiarly due to bug 1677217 https://bugs.launchpad.net/nova/+bug/1677217 you really shoudl not use the AggregateImagePropertiesIsolation any more and any enve that can use isolated aggreates should this however is not a drop in repalcemtn as it requires changes to the images properties/flavor extra specs to request the triats. this was intoduced in Train.
In the Train --> Yoga updated we also removed the AvailabilityZoneFilter (which has been deprecated): I tried to re-add it but this didn't help I can't remember other changes that could be relevant with this issue
Cheers, Massimo
[*]
2022-05-30 10:08:32.620 3168092 DEBUG nova.filters [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669 32b5d42c02b0411b8ebf2c3\ 3079eeecf - default default] Filtering removed all hosts for the request with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results: [('\ AggregateMultiTenancyIsolation', [('cld-blu-12.cloud.pd.infn.it', ' cld-blu-12.cloud.pd.infn.it'), ('cld-blu-02.cloud.pd.infn.it', 'cld-blu-02.cloud.p\ d.infn.it'), ('cld-blu-15.cloud.pd.infn.it', 'cld-blu-15.cloud.pd.infn.it'), ('cld-blu-11.cloud.pd.infn.it', 'cld-blu-11.cloud.pd.infn.it'), ('cld-bl\ u-14.cloud.pd.infn.it', 'cld-blu-14.cloud.pd.infn.it'), (' cld-blu-13.cloud.pd.infn.it', 'cld-blu-13.cloud.pd.infn.it'), (' cld-blu-07.cloud.pd.infn.it\ ', 'cld-blu-07.cloud.pd.infn.it'), ('cld-blu-06.cloud.pd.infn.it', ' cld-blu-06.cloud.pd.infn.it'), ('cld-blu-01.cloud.pd.infn.it', 'cld-blu-01.cloud.\ pd.infn.it'), ('cld-blu-16.cloud.pd.infn.it', 'cld-blu-16.cloud.pd.infn.it')]), ('AggregateInstanceExtraSpecsFilter', None)] get_filtered_objects /us\ r/lib/python3.6/site-packages/nova/filters.py:114 2022-05-30 10:08:32.620 3168092 INFO nova.filters [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669 32b5d42c02b0411b8ebf2c33\ 079eeecf - default default] Filtering removed all hosts for the request with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results: ['Ag\ gregateMultiTenancyIsolation: (start: 58, end: 10)', 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']
On Mon, May 30, 2022 at 9:51 AM Mohammed Naser <mnaser@vexxhost.com> wrote:
Hi there,
Are you sure you haven't done any other changes to your environment? Both those filters haven't been changed for years (~2017):
https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggr...
https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggr...
I think you've got something else here, I'd suggest enabling debug and checking why these nodes are being filtered out.
Mohammed
On Mon, May 30, 2022 at 9:39 AM Massimo Sgaravatto <massimo.sgaravatto@gmail.com> wrote:
It looks like I need now to create a HostAggregate for "size=big" for
filter_tenant_id=<projectid>, size='big'
Is this the expected behaviour ?
Till Train a single HostAggregate with the property: size='big' was enough
Thanks, Massimo
On Fri, May 27, 2022 at 10:08 AM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
Dear all
We have the following use case:
- reserve 3 hypervisors for VMs with "big" flavors (whatever the users
of these instances are)
- partition the rest of the hypervisors according to the project (so
We implemented this:
1- by setting an aggregate_instance_extra_spec 'size' property (with
value 'normal' or 'big') for each flavor [*]
2- by creating a BigVMs HostAggregate for size=big [**] 3- by creating an HostAggregate for size=normal for each project, such as this one [***]
This used to work.
A few days ago we updated our infrastructure from Train to Yoga This was an offline Fast Forward Update: we went through the intermediate releases just to do the dbsyncs. Since this update the instantiation of VMs with flavors with the size=big property doesn't work anymore
This is what I see in nova-scheduler log: 2022-05-27 08:38:02.058 5273 INFO nova.filters [req-f92c0e38-262a-4d22-a7fd-8874c4265401 e237e43716fb490db5bda4b777835669 32b5d42c02b0411b8ebf2c33079eeecf - default default] Filtering removed all hosts for the request with instance ID '2412e188-9d5f-4812-ad21-195769a3c220'. Filter results: ['AggregateMultiTenancyIsolation: (start: 59, end: 10)', 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']
Only modifying the property of the BigVMs HA using the filter_tenant_id adding the relevant project:
[root@cld-ctrl-01 ~]# openstack aggregate show BigVMs | grep prop
properties | filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='big' |
the scheduling works
Specifying each project and keeping the list up-to-date would be a
Any hints ? I didn't find anything related to this issue in the nova release notes
for Openstack releases > Train
These are the filters that we enabled:
[filter_scheduler] enabled_filters =
AggregateMultiTenancyIsolation,AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGro\
upAffinityFilter,PciPassthroughFilter,NUMATopologyFilter
Thanks a lot, Massimo
[*] E.g. [root@cld-ctrl-01 ~]# openstack flavor show cldareapd.medium | grep
properties | aggregate_instance_extra_specs:size='normal' |[root@cld-ctrl-01 ~]# openstack flavor show cloudvenetocloudveneto.40cores128GB25-bigunipd | grep
each project, specifying as properties: projects A1,A2..,An can use only subset S1 of hypervisors, project B1,B2,..,Bm can use only subset S2 of hypervisors) problem. Moreover if I am not wrong there is a maximum length for the property field. prope prop
properties | aggregate_instance_extra_specs:size='big' |
[**]
[root@cld-ctrl-01 ~]# openstack aggregate show BigVMs
+-------------------+---------------------------------------------------------------------------------------+
Field | Value |
+-------------------+---------------------------------------------------------------------------------------+
availability_zone | nova | created_at | 2018-06-20T06:54:51.000000 | deleted_at | None | hosts | cld-blu-08.cloud.pd.infn.it, cld-blu-09.cloud.pd.infn.it, cld-blu-10.cloud.pd.infn.it | id | 135 | is_deleted | False | name | BigVMs | properties |size='big' | updated_at | None | uuid | 4b593395-1c76-441c-9022-d421f4ea2dfb |
+-------------------+---------------------------------------------------------------------------------------+
[***] [root@cld-ctrl-01 ~]# openstack aggregate show Unipd-AdminTesting-Unipd
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Field | Value
|
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
availability_zone | nova
|
created_at | 2018-04-12T07:31:01.000000
|
deleted_at | None
|
hosts | cld-blu-01.cloud.pd.infn.it, cld-blu-02.cloud.pd.infn.it, cld-blu-05.cloud.pd.infn.it, cld-blu-06.cloud.pd.infn.it, cld-blu-07.cloud.pd.infn.it, cld-blu-11.cloud.pd.infn.it, cld-blu-12.cloud.pd.infn.it, cld-blu-13.cloud.pd.infn.it, cld-blu-14.cloud.pd.infn.it, cld-blu-15.cloud.pd.infn.it, cld-blu-16.cloud.pd.infn.it | id | 126
|
is_deleted | False
|
name | Unipd-AdminTesting-Unipd
|
properties | filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='normal'
|
updated_at | 2018-06-08T09:06:20.000000
|
uuid | 38f6a0d4-77ab-42e0-abeb-57e06ba13cca
|
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@cld-ctrl-01 ~]#
-- Mohammed Naser VEXXHOST, Inc.