Thanks Mohammed for your feedback This [*] is what I see in the log file. The 3 relevant nodes are filtered out by the AggregateMultiTenancyIsolation (i.e. they are not among the 10 returned hosts) In the Train --> Yoga updated we also removed the AvailabilityZoneFilter (which has been deprecated): I tried to re-add it but this didn't help I can't remember other changes that could be relevant with this issue Cheers, Massimo [*] 2022-05-30 10:08:32.620 3168092 DEBUG nova.filters [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669 32b5d42c02b0411b8ebf2c3\ 3079eeecf - default default] Filtering removed all hosts for the request with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results: [('\ AggregateMultiTenancyIsolation', [('cld-blu-12.cloud.pd.infn.it', ' cld-blu-12.cloud.pd.infn.it'), ('cld-blu-02.cloud.pd.infn.it', 'cld-blu-02.cloud.p\ d.infn.it'), ('cld-blu-15.cloud.pd.infn.it', 'cld-blu-15.cloud.pd.infn.it'), ('cld-blu-11.cloud.pd.infn.it', 'cld-blu-11.cloud.pd.infn.it'), ('cld-bl\ u-14.cloud.pd.infn.it', 'cld-blu-14.cloud.pd.infn.it'), (' cld-blu-13.cloud.pd.infn.it', 'cld-blu-13.cloud.pd.infn.it'), (' cld-blu-07.cloud.pd.infn.it\ ', 'cld-blu-07.cloud.pd.infn.it'), ('cld-blu-06.cloud.pd.infn.it', ' cld-blu-06.cloud.pd.infn.it'), ('cld-blu-01.cloud.pd.infn.it', 'cld-blu-01.cloud.\ pd.infn.it'), ('cld-blu-16.cloud.pd.infn.it', 'cld-blu-16.cloud.pd.infn.it')]), ('AggregateInstanceExtraSpecsFilter', None)] get_filtered_objects /us\ r/lib/python3.6/site-packages/nova/filters.py:114 2022-05-30 10:08:32.620 3168092 INFO nova.filters [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669 32b5d42c02b0411b8ebf2c33\ 079eeecf - default default] Filtering removed all hosts for the request with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results: ['Ag\ gregateMultiTenancyIsolation: (start: 58, end: 10)', 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)'] On Mon, May 30, 2022 at 9:51 AM Mohammed Naser <mnaser@vexxhost.com> wrote:
Hi there,
Are you sure you haven't done any other changes to your environment? Both those filters haven't been changed for years (~2017):
https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggr...
https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggr...
I think you've got something else here, I'd suggest enabling debug and checking why these nodes are being filtered out.
Mohammed
On Mon, May 30, 2022 at 9:39 AM Massimo Sgaravatto <massimo.sgaravatto@gmail.com> wrote:
It looks like I need now to create a HostAggregate for "size=big" for
filter_tenant_id=<projectid>, size='big'
Is this the expected behaviour ?
Till Train a single HostAggregate with the property: size='big' was enough
Thanks, Massimo
On Fri, May 27, 2022 at 10:08 AM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
Dear all
We have the following use case:
- reserve 3 hypervisors for VMs with "big" flavors (whatever the users
of these instances are)
- partition the rest of the hypervisors according to the project (so
We implemented this:
1- by setting an aggregate_instance_extra_spec 'size' property (with
value 'normal' or 'big') for each flavor [*]
2- by creating a BigVMs HostAggregate for size=big [**] 3- by creating an HostAggregate for size=normal for each project, such as this one [***]
This used to work.
A few days ago we updated our infrastructure from Train to Yoga This was an offline Fast Forward Update: we went through the intermediate releases just to do the dbsyncs. Since this update the instantiation of VMs with flavors with the size=big property doesn't work anymore
This is what I see in nova-scheduler log: 2022-05-27 08:38:02.058 5273 INFO nova.filters [req-f92c0e38-262a-4d22-a7fd-8874c4265401 e237e43716fb490db5bda4b777835669 32b5d42c02b0411b8ebf2c33079eeecf - default default] Filtering removed all hosts for the request with instance ID '2412e188-9d5f-4812-ad21-195769a3c220'. Filter results: ['AggregateMultiTenancyIsolation: (start: 59, end: 10)', 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']
Only modifying the property of the BigVMs HA using the filter_tenant_id adding the relevant project:
[root@cld-ctrl-01 ~]# openstack aggregate show BigVMs | grep prop | properties | filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='big' |
the scheduling works
Specifying each project and keeping the list up-to-date would be a
Any hints ? I didn't find anything related to this issue in the nova release notes
for Openstack releases > Train
These are the filters that we enabled:
[filter_scheduler] enabled_filters =
AggregateMultiTenancyIsolation,AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGro\
upAffinityFilter,PciPassthroughFilter,NUMATopologyFilter
Thanks a lot, Massimo
[*] E.g. [root@cld-ctrl-01 ~]# openstack flavor show cldareapd.medium | grep
| properties | aggregate_instance_extra_specs:size='normal' |[root@cld-ctrl-01 ~]# openstack flavor show cloudvenetocloudveneto.40cores128GB25-bigunipd | grep
each project, specifying as properties: projects A1,A2..,An can use only subset S1 of hypervisors, project B1,B2,..,Bm can use only subset S2 of hypervisors) problem. Moreover if I am not wrong there is a maximum length for the property field. prope prop
| properties | aggregate_instance_extra_specs:size='big' |
[**]
[root@cld-ctrl-01 ~]# openstack aggregate show BigVMs
+-------------------+---------------------------------------------------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------------------------------------------------+
| availability_zone | nova | | created_at | 2018-06-20T06:54:51.000000 | | deleted_at | None | | hosts | cld-blu-08.cloud.pd.infn.it, cld-blu-09.cloud.pd.infn.it, cld-blu-10.cloud.pd.infn.it | | id | 135 | | is_deleted | False | | name | BigVMs | | properties |size='big' | | updated_at | None | | uuid | 4b593395-1c76-441c-9022-d421f4ea2dfb |
+-------------------+---------------------------------------------------------------------------------------+
[***] [root@cld-ctrl-01 ~]# openstack aggregate show Unipd-AdminTesting-Unipd
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value
|
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| availability_zone | nova
|
| created_at | 2018-04-12T07:31:01.000000
|
| deleted_at | None
|
| hosts | cld-blu-01.cloud.pd.infn.it, cld-blu-02.cloud.pd.infn.it, cld-blu-05.cloud.pd.infn.it, cld-blu-06.cloud.pd.infn.it, cld-blu-07.cloud.pd.infn.it, cld-blu-11.cloud.pd.infn.it, cld-blu-12.cloud.pd.infn.it, cld-blu-13.cloud.pd.infn.it, cld-blu-14.cloud.pd.infn.it, cld-blu-15.cloud.pd.infn.it, cld-blu-16.cloud.pd.infn.it | | id | 126
|
| is_deleted | False
|
| name | Unipd-AdminTesting-Unipd
|
| properties | filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='normal'
|
| updated_at | 2018-06-08T09:06:20.000000
|
| uuid | 38f6a0d4-77ab-42e0-abeb-57e06ba13cca
|
+-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@cld-ctrl-01 ~]#
-- Mohammed Naser VEXXHOST, Inc.