[ops][nova] Problems with AggregateMultiTenancyIsolation/AggregateInstanceExtraSpecsFilter after Train --> Yoga update
Sean Mooney
smooney at redhat.com
Mon May 30 11:26:32 UTC 2022
On Mon, 2022-05-30 at 10:44 +0200, Massimo Sgaravatto wrote:
> Thanks Mohammed for your feedback
>
> This [*] is what I see in the log file. The 3 relevant nodes are filtered
> out by the AggregateMultiTenancyIsolation (i.e. they are not among the 10
> returned hosts)
we havent actully chage the code that im aware of but
one thing to note.
it is considered bad pratice to use un namespaced flaovr extra specs with the
AggregateInstanceExtraSpecsFilter or ComputeCapabilitiesFilter
both have legacy support for unnamespced extra specs but you cannot enable both filters if you use that
if you have both the AggregateInstanceExtraSpecsFilter and ComputeCapabilitiesFilter enabled
you must use namespace custome extra specs.
so aggregate_instance_extra_specs: and capabilities:
you have both enabled below so "size=big" is not vaild a a flaovr extra spec in this configuration.
yoga by the way the AggregateMultiTenancyIsolation filter is not needed anymore that can be done with placemnet
https://github.com/openstack/nova/blob/stable/yoga/nova/scheduler/request_filter.py#L94-L135=
by defiening
[scheduler]
limit_tenants_to_placement_aggregate=True
placement_aggregate_required_for_tenants=True
that was also possible in train
we litrally ment to deprecate and remove teh AggregateMultiTenancyIsolatio a few years ago and never got around to it.
it was the first filter implemetn as a placement prefilter in rocky
https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-req-filter.html
so we problay should formally deprecate it this cycle and drop it in AA
https://docs.openstack.org/nova/latest/admin/aggregates.html#tenant-isolation-with-placement explains in more relevent detail
how to use placment for this. its basically a drop in replacement. you dont neeed to update teh aggreate metadta but there a
re some subtlies with regards too what happens to host that are not mapped to any tenant which is why we have 2 config
options so you can decide what you want to happen. i dont think there is any reason for an operator to ever use the
AggregateMultiTenancyIsolation after rocky as the placment verions is much more effienct.
you should alos proably read https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html
while that is mainly inteded to replace the AggregateImagePropertiesIsolation it can also replace
some of the usecasue enabeld by AggregateInstanceExtraSpecsFilter. simiarly due to bug 1677217
https://bugs.launchpad.net/nova/+bug/1677217 you really shoudl not use the AggregateImagePropertiesIsolation
any more and any enve that can use isolated aggreates should this however is not a drop in repalcemtn as it requires
changes to the images properties/flavor extra specs to request the triats. this was intoduced in Train.
>
> In the Train --> Yoga updated we also removed the AvailabilityZoneFilter
> (which has been deprecated): I tried to re-add it but this didn't help
> I can't remember other changes that could be relevant with this issue
>
> Cheers, Massimo
>
>
> [*]
>
> 2022-05-30 10:08:32.620 3168092 DEBUG nova.filters
> [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669
> 32b5d42c02b0411b8ebf2c3\
> 3079eeecf - default default] Filtering removed all hosts for the request
> with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results:
> [('\
> AggregateMultiTenancyIsolation', [('cld-blu-12.cloud.pd.infn.it', '
> cld-blu-12.cloud.pd.infn.it'), ('cld-blu-02.cloud.pd.infn.it',
> 'cld-blu-02.cloud.p\
> d.infn.it'), ('cld-blu-15.cloud.pd.infn.it', 'cld-blu-15.cloud.pd.infn.it'),
> ('cld-blu-11.cloud.pd.infn.it', 'cld-blu-11.cloud.pd.infn.it'), ('cld-bl\
> u-14.cloud.pd.infn.it', 'cld-blu-14.cloud.pd.infn.it'), ('
> cld-blu-13.cloud.pd.infn.it', 'cld-blu-13.cloud.pd.infn.it'), ('
> cld-blu-07.cloud.pd.infn.it\
> ', 'cld-blu-07.cloud.pd.infn.it'), ('cld-blu-06.cloud.pd.infn.it', '
> cld-blu-06.cloud.pd.infn.it'), ('cld-blu-01.cloud.pd.infn.it',
> 'cld-blu-01.cloud.\
> pd.infn.it'), ('cld-blu-16.cloud.pd.infn.it', 'cld-blu-16.cloud.pd.infn.it')]),
> ('AggregateInstanceExtraSpecsFilter', None)] get_filtered_objects /us\
> r/lib/python3.6/site-packages/nova/filters.py:114
> 2022-05-30 10:08:32.620 3168092 INFO nova.filters
> [req-f698ce3b-18c9-4f3f-9b73-49496e19c237 e237e43716fb490db5bda4b777835669
> 32b5d42c02b0411b8ebf2c33\
> 079eeecf - default default] Filtering removed all hosts for the request
> with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results:
> ['Ag\
> gregateMultiTenancyIsolation: (start: 58, end: 10)',
> 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']
>
> On Mon, May 30, 2022 at 9:51 AM Mohammed Naser <mnaser at vexxhost.com> wrote:
>
> > Hi there,
> >
> > Are you sure you haven't done any other changes to your environment?
> > Both those filters haven't been changed for years (~2017):
> >
> >
> > https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_multitenancy_isolation.py
> >
> > https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_instance_extra_specs.py
> >
> > I think you've got something else here, I'd suggest enabling debug and
> > checking why these nodes are being filtered out.
> >
> > Mohammed
> >
> > On Mon, May 30, 2022 at 9:39 AM Massimo Sgaravatto
> > <massimo.sgaravatto at gmail.com> wrote:
> > >
> > > It looks like I need now to create a HostAggregate for "size=big" for
> > each project, specifying as properties:
> > > filter_tenant_id=<projectid>, size='big'
> > >
> > > Is this the expected behaviour ?
> > >
> > > Till Train a single HostAggregate with the property: size='big' was
> > enough
> > >
> > > Thanks, Massimo
> > >
> > >
> > >
> > > On Fri, May 27, 2022 at 10:08 AM Massimo Sgaravatto <
> > massimo.sgaravatto at gmail.com> wrote:
> > > >
> > > > Dear all
> > > >
> > > > We have the following use case:
> > > >
> > > > - reserve 3 hypervisors for VMs with "big" flavors (whatever the users
> > of these instances are)
> > > > - partition the rest of the hypervisors according to the project (so
> > projects A1,A2..,An can use only subset S1 of hypervisors, project
> > B1,B2,..,Bm can use only subset S2 of hypervisors)
> > > >
> > > > We implemented this:
> > > >
> > > > 1- by setting an aggregate_instance_extra_spec 'size' property (with
> > value 'normal' or 'big') for each flavor [*]
> > > > 2- by creating a BigVMs HostAggregate for size=big [**]
> > > > 3- by creating an HostAggregate for size=normal for each project, such
> > as this one [***]
> > > >
> > > >
> > > > This used to work.
> > > >
> > > > A few days ago we updated our infrastructure from Train to Yoga
> > > > This was an offline Fast Forward Update: we went through the
> > intermediate releases just to do the dbsyncs.
> > > > Since this update the instantiation of VMs with flavors with the
> > size=big property doesn't work anymore
> > > >
> > > >
> > > > This is what I see in nova-scheduler log:
> > > > 2022-05-27 08:38:02.058 5273 INFO nova.filters
> > [req-f92c0e38-262a-4d22-a7fd-8874c4265401 e237e43716fb490db5bda4b777835669
> > 32b5d42c02b0411b8ebf2c33079eeecf - default default] Filtering removed all
> > hosts for the request with instance ID
> > '2412e188-9d5f-4812-ad21-195769a3c220'. Filter results:
> > ['AggregateMultiTenancyIsolation: (start: 59, end: 10)',
> > 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']
> > > >
> > > >
> > > > Only modifying the property of the BigVMs HA using the
> > filter_tenant_id adding the relevant project:
> > > >
> > > > [root at cld-ctrl-01 ~]# openstack aggregate show BigVMs | grep prop
> > > > > properties |
> > filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='big'
> > |
> > > >
> > > > the scheduling works
> > > >
> > > > Specifying each project and keeping the list up-to-date would be a
> > problem. Moreover if I am not wrong there is a maximum length for the
> > property field.
> > > >
> > > > Any hints ?
> > > > I didn't find anything related to this issue in the nova release notes
> > for Openstack releases > Train
> > > >
> > > >
> > > > These are the filters that we enabled:
> > > >
> > > > [filter_scheduler]
> > > > enabled_filters =
> > AggregateMultiTenancyIsolation,AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGro\
> > > > upAffinityFilter,PciPassthroughFilter,NUMATopologyFilter
> > > >
> > > >
> > > > Thanks a lot, Massimo
> > > >
> > > >
> > > > [*]
> > > > E.g.
> > > > [root at cld-ctrl-01 ~]# openstack flavor show cldareapd.medium | grep
> > prope
> > > > > properties |
> > aggregate_instance_extra_specs:size='normal' |[root at cld-ctrl-01 ~]#
> > openstack flavor show cloudvenetocloudveneto.40cores128GB25-bigunipd | grep
> > prop
> > > > > properties |
> > aggregate_instance_extra_specs:size='big' |
> > > >
> > > >
> > > > [**]
> > > >
> > > > [root at cld-ctrl-01 ~]# openstack aggregate show BigVMs
> > > >
> > +-------------------+---------------------------------------------------------------------------------------+
> > > > > Field | Value
> > |
> > > >
> > +-------------------+---------------------------------------------------------------------------------------+
> > > > > availability_zone | nova
> > |
> > > > > created_at | 2018-06-20T06:54:51.000000
> > |
> > > > > deleted_at | None
> > |
> > > > > hosts | cld-blu-08.cloud.pd.infn.it,
> > cld-blu-09.cloud.pd.infn.it, cld-blu-10.cloud.pd.infn.it |
> > > > > id | 135
> > |
> > > > > is_deleted | False
> > |
> > > > > name | BigVMs
> > |
> > > > > properties |size='big' |
> > > > > updated_at | None
> > |
> > > > > uuid | 4b593395-1c76-441c-9022-d421f4ea2dfb
> > |
> > > >
> > +-------------------+---------------------------------------------------------------------------------------+
> > > >
> > > >
> > > > [***]
> > > > [root at cld-ctrl-01 ~]# openstack aggregate show Unipd-AdminTesting-Unipd
> > > >
> > +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > > > > Field | Value
> >
> >
> >
> > |
> > > >
> > +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > > > > availability_zone | nova
> >
> >
> >
> > |
> > > > > created_at | 2018-04-12T07:31:01.000000
> >
> >
> >
> > |
> > > > > deleted_at | None
> >
> >
> >
> > |
> > > > > hosts | cld-blu-01.cloud.pd.infn.it,
> > cld-blu-02.cloud.pd.infn.it, cld-blu-05.cloud.pd.infn.it,
> > cld-blu-06.cloud.pd.infn.it, cld-blu-07.cloud.pd.infn.it,
> > cld-blu-11.cloud.pd.infn.it, cld-blu-12.cloud.pd.infn.it,
> > cld-blu-13.cloud.pd.infn.it, cld-blu-14.cloud.pd.infn.it,
> > cld-blu-15.cloud.pd.infn.it, cld-blu-16.cloud.pd.infn.it |
> > > > > id | 126
> >
> >
> >
> > |
> > > > > is_deleted | False
> >
> >
> >
> > |
> > > > > name | Unipd-AdminTesting-Unipd
> >
> >
> >
> > |
> > > > > properties |
> > filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='normal'
> >
> >
> >
> > |
> > > > > updated_at | 2018-06-08T09:06:20.000000
> >
> >
> >
> > |
> > > > > uuid | 38f6a0d4-77ab-42e0-abeb-57e06ba13cca
> >
> >
> >
> > |
> > > >
> > +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > > > [root at cld-ctrl-01 ~]#
> > > >
> >
> >
> > --
> > Mohammed Naser
> > VEXXHOST, Inc.
> >
More information about the openstack-discuss
mailing list