[ops][nova] Problems with AggregateMultiTenancyIsolation/AggregateInstanceExtraSpecsFilter after Train --> Yoga update
Massimo Sgaravatto
massimo.sgaravatto at gmail.com
Mon May 30 13:35:04 UTC 2022
It was my fault: there was a new Host Aggregate using filter_tenant_id
which includes the 3 hosts
Apologies for the noise
Now I am going to study what Sean wrote :-)
Thanks, Massimo
On Mon, May 30, 2022 at 1:27 PM Sean Mooney <smooney at redhat.com> wrote:
> On Mon, 2022-05-30 at 10:44 +0200, Massimo Sgaravatto wrote:
> > Thanks Mohammed for your feedback
> >
> > This [*] is what I see in the log file. The 3 relevant nodes are filtered
> > out by the AggregateMultiTenancyIsolation (i.e. they are not among the 10
> > returned hosts)
>
> we havent actully chage the code that im aware of but
> one thing to note.
> it is considered bad pratice to use un namespaced flaovr extra specs with
> the
> AggregateInstanceExtraSpecsFilter or ComputeCapabilitiesFilter
>
> both have legacy support for unnamespced extra specs but you cannot enable
> both filters if you use that
>
> if you have both the AggregateInstanceExtraSpecsFilter and
> ComputeCapabilitiesFilter enabled
> you must use namespace custome extra specs.
>
> so aggregate_instance_extra_specs: and capabilities:
>
> you have both enabled below so "size=big" is not vaild a a flaovr extra
> spec in this configuration.
>
> yoga by the way the AggregateMultiTenancyIsolation filter is not needed
> anymore that can be done with placemnet
>
>
> https://github.com/openstack/nova/blob/stable/yoga/nova/scheduler/request_filter.py#L94-L135=
>
> by defiening
> [scheduler]
> limit_tenants_to_placement_aggregate=True
> placement_aggregate_required_for_tenants=True
>
> that was also possible in train
>
> we litrally ment to deprecate and remove teh AggregateMultiTenancyIsolatio
> a few years ago and never got around to it.
> it was the first filter implemetn as a placement prefilter in rocky
>
> https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-req-filter.html
> so we problay should formally deprecate it this cycle and drop it in AA
>
>
> https://docs.openstack.org/nova/latest/admin/aggregates.html#tenant-isolation-with-placement
> explains in more relevent detail
> how to use placment for this. its basically a drop in replacement. you
> dont neeed to update teh aggreate metadta but there a
> re some subtlies with regards too what happens to host that are not mapped
> to any tenant which is why we have 2 config
> options so you can decide what you want to happen. i dont think there is
> any reason for an operator to ever use the
> AggregateMultiTenancyIsolation after rocky as the placment verions is much
> more effienct.
>
>
> you should alos proably read
> https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html
> while that is mainly inteded to replace the
> AggregateImagePropertiesIsolation it can also replace
> some of the usecasue enabeld by AggregateInstanceExtraSpecsFilter.
> simiarly due to bug 1677217
> https://bugs.launchpad.net/nova/+bug/1677217 you really shoudl not use
> the AggregateImagePropertiesIsolation
> any more and any enve that can use isolated aggreates should this however
> is not a drop in repalcemtn as it requires
> changes to the images properties/flavor extra specs to request the triats.
> this was intoduced in Train.
>
>
> >
> > In the Train --> Yoga updated we also removed the AvailabilityZoneFilter
> > (which has been deprecated): I tried to re-add it but this didn't help
> > I can't remember other changes that could be relevant with this issue
> >
> > Cheers, Massimo
> >
> >
> > [*]
> >
> > 2022-05-30 10:08:32.620 3168092 DEBUG nova.filters
> > [req-f698ce3b-18c9-4f3f-9b73-49496e19c237
> e237e43716fb490db5bda4b777835669
> > 32b5d42c02b0411b8ebf2c3\
> > 3079eeecf - default default] Filtering removed all hosts for the request
> > with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results:
> > [('\
> > AggregateMultiTenancyIsolation', [('cld-blu-12.cloud.pd.infn.it', '
> > cld-blu-12.cloud.pd.infn.it'), ('cld-blu-02.cloud.pd.infn.it',
> > 'cld-blu-02.cloud.p\
> > d.infn.it'), ('cld-blu-15.cloud.pd.infn.it', '
> cld-blu-15.cloud.pd.infn.it'),
> > ('cld-blu-11.cloud.pd.infn.it', 'cld-blu-11.cloud.pd.infn.it'),
> ('cld-bl\
> > u-14.cloud.pd.infn.it', 'cld-blu-14.cloud.pd.infn.it'), ('
> > cld-blu-13.cloud.pd.infn.it', 'cld-blu-13.cloud.pd.infn.it'), ('
> > cld-blu-07.cloud.pd.infn.it\
> > ', 'cld-blu-07.cloud.pd.infn.it'), ('cld-blu-06.cloud.pd.infn.it', '
> > cld-blu-06.cloud.pd.infn.it'), ('cld-blu-01.cloud.pd.infn.it',
> > 'cld-blu-01.cloud.\
> > pd.infn.it'), ('cld-blu-16.cloud.pd.infn.it', '
> cld-blu-16.cloud.pd.infn.it')]),
> > ('AggregateInstanceExtraSpecsFilter', None)] get_filtered_objects /us\
> > r/lib/python3.6/site-packages/nova/filters.py:114
> > 2022-05-30 10:08:32.620 3168092 INFO nova.filters
> > [req-f698ce3b-18c9-4f3f-9b73-49496e19c237
> e237e43716fb490db5bda4b777835669
> > 32b5d42c02b0411b8ebf2c33\
> > 079eeecf - default default] Filtering removed all hosts for the request
> > with instance ID 'fd6d978a-3739-45c6-b1cc-28fc9e57d381'. Filter results:
> > ['Ag\
> > gregateMultiTenancyIsolation: (start: 58, end: 10)',
> > 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']
> >
> > On Mon, May 30, 2022 at 9:51 AM Mohammed Naser <mnaser at vexxhost.com>
> wrote:
> >
> > > Hi there,
> > >
> > > Are you sure you haven't done any other changes to your environment?
> > > Both those filters haven't been changed for years (~2017):
> > >
> > >
> > >
> https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_multitenancy_isolation.py
> > >
> > >
> https://github.com/openstack/nova/commits/master/nova/scheduler/filters/aggregate_instance_extra_specs.py
> > >
> > > I think you've got something else here, I'd suggest enabling debug and
> > > checking why these nodes are being filtered out.
> > >
> > > Mohammed
> > >
> > > On Mon, May 30, 2022 at 9:39 AM Massimo Sgaravatto
> > > <massimo.sgaravatto at gmail.com> wrote:
> > > >
> > > > It looks like I need now to create a HostAggregate for "size=big" for
> > > each project, specifying as properties:
> > > > filter_tenant_id=<projectid>, size='big'
> > > >
> > > > Is this the expected behaviour ?
> > > >
> > > > Till Train a single HostAggregate with the property: size='big' was
> > > enough
> > > >
> > > > Thanks, Massimo
> > > >
> > > >
> > > >
> > > > On Fri, May 27, 2022 at 10:08 AM Massimo Sgaravatto <
> > > massimo.sgaravatto at gmail.com> wrote:
> > > > >
> > > > > Dear all
> > > > >
> > > > > We have the following use case:
> > > > >
> > > > > - reserve 3 hypervisors for VMs with "big" flavors (whatever the
> users
> > > of these instances are)
> > > > > - partition the rest of the hypervisors according to the project
> (so
> > > projects A1,A2..,An can use only subset S1 of hypervisors, project
> > > B1,B2,..,Bm can use only subset S2 of hypervisors)
> > > > >
> > > > > We implemented this:
> > > > >
> > > > > 1- by setting an aggregate_instance_extra_spec 'size' property
> (with
> > > value 'normal' or 'big') for each flavor [*]
> > > > > 2- by creating a BigVMs HostAggregate for size=big [**]
> > > > > 3- by creating an HostAggregate for size=normal for each project,
> such
> > > as this one [***]
> > > > >
> > > > >
> > > > > This used to work.
> > > > >
> > > > > A few days ago we updated our infrastructure from Train to Yoga
> > > > > This was an offline Fast Forward Update: we went through the
> > > intermediate releases just to do the dbsyncs.
> > > > > Since this update the instantiation of VMs with flavors with the
> > > size=big property doesn't work anymore
> > > > >
> > > > >
> > > > > This is what I see in nova-scheduler log:
> > > > > 2022-05-27 08:38:02.058 5273 INFO nova.filters
> > > [req-f92c0e38-262a-4d22-a7fd-8874c4265401
> e237e43716fb490db5bda4b777835669
> > > 32b5d42c02b0411b8ebf2c33079eeecf - default default] Filtering removed
> all
> > > hosts for the request with instance ID
> > > '2412e188-9d5f-4812-ad21-195769a3c220'. Filter results:
> > > ['AggregateMultiTenancyIsolation: (start: 59, end: 10)',
> > > 'AggregateInstanceExtraSpecsFilter: (start: 10, end: 0)']
> > > > >
> > > > >
> > > > > Only modifying the property of the BigVMs HA using the
> > > filter_tenant_id adding the relevant project:
> > > > >
> > > > > [root at cld-ctrl-01 ~]# openstack aggregate show BigVMs | grep prop
> > > > > > properties |
> > > filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='big'
> > > |
> > > > >
> > > > > the scheduling works
> > > > >
> > > > > Specifying each project and keeping the list up-to-date would be a
> > > problem. Moreover if I am not wrong there is a maximum length for the
> > > property field.
> > > > >
> > > > > Any hints ?
> > > > > I didn't find anything related to this issue in the nova release
> notes
> > > for Openstack releases > Train
> > > > >
> > > > >
> > > > > These are the filters that we enabled:
> > > > >
> > > > > [filter_scheduler]
> > > > > enabled_filters =
> > >
> AggregateMultiTenancyIsolation,AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGro\
> > > > > upAffinityFilter,PciPassthroughFilter,NUMATopologyFilter
> > > > >
> > > > >
> > > > > Thanks a lot, Massimo
> > > > >
> > > > >
> > > > > [*]
> > > > > E.g.
> > > > > [root at cld-ctrl-01 ~]# openstack flavor show cldareapd.medium |
> grep
> > > prope
> > > > > > properties |
> > > aggregate_instance_extra_specs:size='normal' |[root at cld-ctrl-01 ~]#
> > > openstack flavor show cloudvenetocloudveneto.40cores128GB25-bigunipd |
> grep
> > > prop
> > > > > > properties |
> > > aggregate_instance_extra_specs:size='big' |
> > > > >
> > > > >
> > > > > [**]
> > > > >
> > > > > [root at cld-ctrl-01 ~]# openstack aggregate show BigVMs
> > > > >
> > >
> +-------------------+---------------------------------------------------------------------------------------+
> > > > > > Field | Value
> > > |
> > > > >
> > >
> +-------------------+---------------------------------------------------------------------------------------+
> > > > > > availability_zone | nova
> > > |
> > > > > > created_at | 2018-06-20T06:54:51.000000
> > > |
> > > > > > deleted_at | None
> > > |
> > > > > > hosts | cld-blu-08.cloud.pd.infn.it,
> > > cld-blu-09.cloud.pd.infn.it, cld-blu-10.cloud.pd.infn.it |
> > > > > > id | 135
> > > |
> > > > > > is_deleted | False
> > > |
> > > > > > name | BigVMs
> > > |
> > > > > > properties |size='big' |
> > > > > > updated_at | None
> > > |
> > > > > > uuid | 4b593395-1c76-441c-9022-d421f4ea2dfb
> > > |
> > > > >
> > >
> +-------------------+---------------------------------------------------------------------------------------+
> > > > >
> > > > >
> > > > > [***]
> > > > > [root at cld-ctrl-01 ~]# openstack aggregate show
> Unipd-AdminTesting-Unipd
> > > > >
> > >
> +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > > > > > Field | Value
> > >
> > >
> > >
> > > |
> > > > >
> > >
> +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > > > > > availability_zone | nova
> > >
> > >
> > >
> > > |
> > > > > > created_at | 2018-04-12T07:31:01.000000
> > >
> > >
> > >
> > > |
> > > > > > deleted_at | None
> > >
> > >
> > >
> > > |
> > > > > > hosts | cld-blu-01.cloud.pd.infn.it,
> > > cld-blu-02.cloud.pd.infn.it, cld-blu-05.cloud.pd.infn.it,
> > > cld-blu-06.cloud.pd.infn.it, cld-blu-07.cloud.pd.infn.it,
> > > cld-blu-11.cloud.pd.infn.it, cld-blu-12.cloud.pd.infn.it,
> > > cld-blu-13.cloud.pd.infn.it, cld-blu-14.cloud.pd.infn.it,
> > > cld-blu-15.cloud.pd.infn.it, cld-blu-16.cloud.pd.infn.it |
> > > > > > id | 126
> > >
> > >
> > >
> > > |
> > > > > > is_deleted | False
> > >
> > >
> > >
> > > |
> > > > > > name | Unipd-AdminTesting-Unipd
> > >
> > >
> > >
> > > |
> > > > > > properties |
> > > filter_tenant_id='32b5d42c02b0411b8ebf2c33079eeecf', size='normal'
> > >
> > >
> > >
> > > |
> > > > > > updated_at | 2018-06-08T09:06:20.000000
> > >
> > >
> > >
> > > |
> > > > > > uuid | 38f6a0d4-77ab-42e0-abeb-57e06ba13cca
> > >
> > >
> > >
> > > |
> > > > >
> > >
> +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> > > > > [root at cld-ctrl-01 ~]#
> > > > >
> > >
> > >
> > > --
> > > Mohammed Naser
> > > VEXXHOST, Inc.
> > >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20220530/bc246023/attachment-0001.htm>
More information about the openstack-discuss
mailing list