[nova] How to use $aggregates with JsonFilter
Hey Team, I'm trying to use scheduler hints using the JsonFilter that involve host aggregates. Based on the documentation and my quick perusal of the scheduler's HostState class... it seems like $aggregates are populated there for usage, but I can't seem to find the right magick to make a json query work to schedule an instance on a host that belongs to a specific aggregate based on simply the name of the aggregate... or perhaps metadata contained within. I think my lack of understanding of the structure of the $aggregates list in the scheduler's HostState class is tripping me up. Does anyone have any insight into this?... Thanks. -- Andy Speagle
the aggreates are populated in the hoststate object for usage by other filters and weighers the json filter was not designed to be used with them in general it was intend for use with the other filed that don't have dynamic subscriptable content. i.e. where the key si not know ahead of time the operations supprote dby it are basic comparsion that work on singal values with the excption fo in https://github.com/openstack/nova/blob/master/nova/scheduler/filters/json_fi... so you can check fi a instnce uuid is in the set of instnaces on a host that work becasue its a dict keyed by the instance uuid so in or container works since the aggreats are a list you cant check if a host aggregate is in the host aggregates list because you woudl have to construct a host aggret object for in to work so as i said above aggregates are not supported by this filter as it was not part of its original design. it was orginally inteded form matching onf files like hypervior_type or version https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py... On 10/12/2025 01:43, Andy Speagle wrote:
Hey Team,
I'm trying to use scheduler hints using the JsonFilter that involve host aggregates. Based on the documentation and my quick perusal of the scheduler's HostState class... it seems like $aggregates are populated there for usage, but I can't seem to find the right magick to make a json query work to schedule an instance on a host that belongs to a specific aggregate based on simply the name of the aggregate... or perhaps metadata contained within.
I think my lack of understanding of the structure of the $aggregates list in the scheduler's HostState class is tripping me up.
Does anyone have any insight into this?...
Thanks.
Hmm... that seems like a pretty big miss for JsonFilter. I see in the code that there's logging built into the update() function for the HostState class... how would one trigger and then see that? I'm just morbidly curious what actually gets set in the $aggregates list. Thanks. On Wed, 2025-12-10 at 12:46 +0000, Sean Mooney wrote:
⚠️CAUTION⚠️ This email originated outside of Toyon's network. Do not download attachments or click links unless you know the content is safe.
the aggreates are populated in the hoststate object for usage by other filters and weighers the json filter was not designed to be used with them in general
it was intend for use with the other filed that don't have dynamic subscriptable content. i.e. where the key si not know ahead of time
the operations supprote dby it are basic comparsion that work on singal values with the excption fo in
https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
so you can check fi a instnce uuid is in the set of instnaces on a host that work becasue its a dict keyed by the instance uuid so in or container works since the aggreats are a list you cant check if a host aggregate is in the host aggregates list because you woudl have to construct a host aggret object for in to work
so as i said above aggregates are not supported by this filter as it was not part of its original design.
it was orginally inteded form matching onf files like hypervior_type or version https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
On 10/12/2025 01:43, Andy Speagle wrote:
Hey Team,
I'm trying to use scheduler hints using the JsonFilter that involve host aggregates. Based on the documentation and my quick perusal of the scheduler's HostState class... it seems like $aggregates are populated there for usage, but I can't seem to find the right magick to make a json query work to schedule an instance on a host that belongs to a specific aggregate based on simply the name of the aggregate... or perhaps metadata contained within.
I think my lack of understanding of the structure of the $aggregates list in the scheduler's HostState class is tripping me up.
Does anyone have any insight into this?...
Thanks.
-- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431
Also... if this isn't "the way"... then I'm curious what is the correct path forward. I'm using terraform to build up my infrastructure on openstack and I'm looking for a way to ensure instances are scheduled on specific hosts. Aggregates seemed the right way. I'd rather not have dozens of different flavors doing this work. Scheduler hints seemed to be the simplest way. Thanks if advance for any advice. On Wed, 2025-12-10 at 12:46 +0000, Sean Mooney wrote:
⚠️CAUTION⚠️ This email originated outside of Toyon's network. Do not download attachments or click links unless you know the content is safe.
the aggreates are populated in the hoststate object for usage by other filters and weighers the json filter was not designed to be used with them in general
it was intend for use with the other filed that don't have dynamic subscriptable content. i.e. where the key si not know ahead of time
the operations supprote dby it are basic comparsion that work on singal values with the excption fo in
https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
so you can check fi a instnce uuid is in the set of instnaces on a host that work becasue its a dict keyed by the instance uuid so in or container works since the aggreats are a list you cant check if a host aggregate is in the host aggregates list because you woudl have to construct a host aggret object for in to work
so as i said above aggregates are not supported by this filter as it was not part of its original design.
it was orginally inteded form matching onf files like hypervior_type or version https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
On 10/12/2025 01:43, Andy Speagle wrote:
Hey Team,
I'm trying to use scheduler hints using the JsonFilter that involve host aggregates. Based on the documentation and my quick perusal of the scheduler's HostState class... it seems like $aggregates are populated there for usage, but I can't seem to find the right magick to make a json query work to schedule an instance on a host that belongs to a specific aggregate based on simply the name of the aggregate... or perhaps metadata contained within.
I think my lack of understanding of the structure of the $aggregates list in the scheduler's HostState class is tripping me up.
Does anyone have any insight into this?...
Thanks.
-- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431
if you have admin ascess you can do that vai passing the host on server create or on move operations. https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... https://docs.openstack.org/api-ref/compute/#id11 https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht... https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/add... with the json filter you could also jsut match on the hypervior hostname or host filed in the hoststate object but its simpler to do it with the native supprot in stead of buildign it your self again in the jsonfilter. there is also an older way to do this by abusing the AZ the aviablity zone filed supprot <AZ>:<host>:<node> syntax so you can just do `:<hostname>` for the az or "my_az:compute-42" if you can create flavor that implies your an admin so you shoudl proably just use the host or hypervior_hostname fiels in the server create rerequest. On 10/12/2025 15:23, Andy Speagle wrote:
Also... if this isn't "the way"... then I'm curious what is the correct path forward.
I'm using terraform to build up my infrastructure on openstack and I'm looking for a way to ensure instances are scheduled on specific hosts. Aggregates seemed the right way. I'd rather not have dozens of different flavors doing this work. Scheduler hints seemed to be the simplest way.
Thanks if advance for any advice.
On Wed, 2025-12-10 at 12:46 +0000, Sean Mooney wrote:
⚠️CAUTION⚠️ This email originated outside of Toyon's network. Do not download attachments or click links unless you know the content is safe.
the aggreates are populated in the hoststate object for usage by other filters and weighers the json filter was not designed to be used with them in general
it was intend for use with the other filed that don't have dynamic subscriptable content. i.e. where the key si not know ahead of time
the operations supprote dby it are basic comparsion that work on singal values with the excption fo in
https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
so you can check fi a instnce uuid is in the set of instnaces on a host that work becasue its a dict keyed by the instance uuid so in or container works since the aggreats are a list you cant check if a host aggregate is in the host aggregates list because you woudl have to construct a host aggret object for in to work
so as i said above aggregates are not supported by this filter as it was not part of its original design.
it was orginally inteded form matching onf files like hypervior_type or version https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
On 10/12/2025 01:43, Andy Speagle wrote:
Hey Team,
I'm trying to use scheduler hints using the JsonFilter that involve host aggregates. Based on the documentation and my quick perusal of the scheduler's HostState class... it seems like $aggregates are populated there for usage, but I can't seem to find the right magick to make a json query work to schedule an instance on a host that belongs to a specific aggregate based on simply the name of the aggregate... or perhaps metadata contained within.
I think my lack of understanding of the structure of the $aggregates list in the scheduler's HostState class is tripping me up.
Does anyone have any insight into this?...
Thanks.
Yeah... I've done this before with flavors and was looking for a way to schedule instances using aggregates without the need to create more flavors that would pin instances to specific hosts. The main use case here is that we have a number of windows server based instances that due to licensing need to normally be colocated on a few hypervisors... it would be nice to have a feature that would allow me to do this simply... this is all rather klunky. On Wed, 2025-12-10 at 16:04 +0000, Sean Mooney wrote:
if you have admin ascess you can do that vai passing the host on server create or on move operations.
https://urldefense.us/v3/__https://docs.openstack.org/python-openstackclient... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*id11_... https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/... https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
with the json filter you could also jsut match on the hypervior hostname or host filed in the hoststate object but its simpler to do it with the native supprot in stead of buildign it your self again in the jsonfilter.
there is also an older way to do this by abusing the AZ
the aviablity zone filed supprot <AZ>:<host>:<node> syntax
so you can just do `:<hostname>` for the az or "my_az:compute-42"
if you can create flavor that implies your an admin so you shoudl proably just use the host or hypervior_hostname fiels in the server create rerequest.
On 10/12/2025 15:23, Andy Speagle wrote:
Also... if this isn't "the way"... then I'm curious what is the correct path forward.
I'm using terraform to build up my infrastructure on openstack and I'm looking for a way to ensure instances are scheduled on specific hosts. Aggregates seemed the right way. I'd rather not have dozens of different flavors doing this work. Scheduler hints seemed to be the simplest way.
Thanks if advance for any advice.
On Wed, 2025-12-10 at 12:46 +0000, Sean Mooney wrote:
⚠️CAUTION⚠️ This email originated outside of Toyon's network. Do not download attachments or click links unless you know the content is safe.
the aggreates are populated in the hoststate object for usage by other filters and weighers the json filter was not designed to be used with them in general
it was intend for use with the other filed that don't have dynamic subscriptable content. i.e. where the key si not know ahead of time
the operations supprote dby it are basic comparsion that work on singal values with the excption fo in
https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
so you can check fi a instnce uuid is in the set of instnaces on a host that work becasue its a dict keyed by the instance uuid so in or container works since the aggreats are a list you cant check if a host aggregate is in the host aggregates list because you woudl have to construct a host aggret object for in to work
so as i said above aggregates are not supported by this filter as it was not part of its original design.
it was orginally inteded form matching onf files like hypervior_type or version https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
On 10/12/2025 01:43, Andy Speagle wrote:
Hey Team,
I'm trying to use scheduler hints using the JsonFilter that involve host aggregates. Based on the documentation and my quick perusal of the scheduler's HostState class... it seems like $aggregates are populated there for usage, but I can't seem to find the right magick to make a json query work to schedule an instance on a host that belongs to a specific aggregate based on simply the name of the aggregate... or perhaps metadata contained within.
I think my lack of understanding of the structure of the $aggregates list in the scheduler's HostState class is tripping me up.
Does anyone have any insight into this?...
Thanks.
-- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431
so orignally you ask for a way to not do this via flavor so i provied 2 ways to do it via teh server creeate command using --host or via the az on a per instance baseis what your acully asking for is effectivly to do it vai the image we develope a feature precisly for the windows usecase https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html you can do it via filter as well if you want too https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregateimagep... but the recommend way to do this is to add a requried trait to the winwos images and use the placemet functionality to enforce it for you. https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/pla... you can declaratively add the CUSTOM_LICENSED_WINDOWS trait to your compute nodes using the provider.yaml feature https://specs.openstack.org/openstack/nova-specs/specs/train/approved/provid... https://docs.openstack.org/nova/latest/admin/managing-resource-providers.htm... if you do not want to do that via the placement apias documetned in https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html if nethigh of those options appeal you can use https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregateimagep... the eexample is actully lightly wrong because it suggest using os_distro='windows' you shoudl be useing 'os_type=windows' since that is the image property that enable the hyperv enlightment to improve the performance of windows on qemu/kvm. https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt... os_distor is ment to be "|win2k25||"|i belvie for windows server 2025 with os_version holdign something more human readble if needed. although folk often just use human readbale names instead of the osinfo ones. https://gitlab.com/libosinfo/osinfo-db/-/tree/main/data/os/microsoft.com?ref... we are incorectly using os_disto to mathc on the hosrt id i belive. there are a couple of other ways to do this as well but isolate aggregates in placment and the aggregateimagepropertiesisolation filter are the two solutions that were built specific to adress the windows server licening usecase. On 11/12/2025 05:52, Andy Speagle wrote:
Yeah... I've done this before with flavors and was looking for a way to schedule instances using aggregates without the need to create more flavors that would pin instances to specific hosts.
The main use case here is that we have a number of windows server based instances that due to licensing need to normally be colocated on a few hypervisors... it would be nice to have a feature that would allow me to do this simply... this is all rather klunky.
On Wed, 2025-12-10 at 16:04 +0000, Sean Mooney wrote:
if you have admin ascess you can do that vai passing the host on server create or on move operations.
https://urldefense.us/v3/__https://docs.openstack.org/python-openstackclient...
https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*id11_...
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/...
https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
with the json filter you could also jsut match on the hypervior hostname or host filed in the hoststate object but its simpler to do it with the native supprot in stead of buildign it your self again in the jsonfilter.
there is also an older way to do this by abusing the AZ
the aviablity zone filed supprot <AZ>:<host>:<node> syntax
so you can just do `:<hostname>` for the az or "my_az:compute-42"
if you can create flavor that implies your an admin so you shoudl proably just use the host or hypervior_hostname fiels in the server create rerequest.
On 10/12/2025 15:23, Andy Speagle wrote:
Also... if this isn't "the way"... then I'm curious what is the correct path forward.
I'm using terraform to build up my infrastructure on openstack and I'm looking for a way to ensure instances are scheduled on specific hosts. Aggregates seemed the right way. I'd rather not have dozens of different flavors doing this work. Scheduler hints seemed to be the simplest way.
Thanks if advance for any advice.
On Wed, 2025-12-10 at 12:46 +0000, Sean Mooney wrote:
⚠️CAUTION⚠️ This email originated outside of Toyon's network. Do not download attachments or click links unless you know the content is safe.
the aggreates are populated in the hoststate object for usage by other filters and weighers the json filter was not designed to be used with them in general
it was intend for use with the other filed that don't have dynamic subscriptable content. i.e. where the key si not know ahead of time
the operations supprote dby it are basic comparsion that work on singal values with the excption fo in
https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
so you can check fi a instnce uuid is in the set of instnaces on a host that work becasue its a dict keyed by the instance uuid so in or container works since the aggreats are a list you cant check if a host aggregate is in the host aggregates list because you woudl have to construct a host aggret object for in to work
so as i said above aggregates are not supported by this filter as it was not part of its original design.
it was orginally inteded form matching onf files like hypervior_type or version https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
On 10/12/2025 01:43, Andy Speagle wrote:
Hey Team,
I'm trying to use scheduler hints using the JsonFilter that involve host aggregates. Based on the documentation and my quick perusal of the scheduler's HostState class... it seems like $aggregates are populated there for usage, but I can't seem to find the right magick to make a json query work to schedule an instance on a host that belongs to a specific aggregate based on simply the name of the aggregate... or perhaps metadata contained within.
I think my lack of understanding of the structure of the $aggregates list in the scheduler's HostState class is tripping me up.
Does anyone have any insight into this?...
Thanks.
Okay... thanks for that. I can go the custom trait route. It feels a like a lot of work for something that should be easy with scheduler hints... and it also makes things difficult in a maintenance scenario when those windoze instances need to be scheduled elsewhere so work can be done on their "licensed" hypervisors. On Thu, 2025-12-11 at 13:56 +0000, Sean Mooney wrote:
so orignally you ask for a way to not do this via flavor so i provied 2 ways to do it via teh server creeate command using --host or via the az on a per instance baseis
what your acully asking for is effectivly to do it vai the image
we develope a feature precisly for the windows usecase
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/... you can do it via filter as well if you want too https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/sche... but the recommend way to do this is to add a requried trait to the winwos images and use the placemet functionality to enforce it for you.
https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
you can declaratively add the CUSTOM_LICENSED_WINDOWS trait to your compute nodes using the provider.yaml feature https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/... https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/mana...
if you do not want to do that via the placement apias documetned in https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/...
if nethigh of those options appeal you can use
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/sche...
the eexample is actully lightly wrong because it suggest using
os_distro='windows' you shoudl be useing 'os_type=windows' since that is the image property that enable the hyperv enlightment to improve the performance of windows on qemu/kvm. https://urldefense.us/v3/__https://github.com/openstack/glance/blob/master/e...
os_distor is ment to be "|win2k25||"|i belvie for windows server 2025 with os_version holdign something more human readble if needed. although folk often just use human readbale names instead of the osinfo ones. https://urldefense.us/v3/__https://gitlab.com/libosinfo/osinfo-db/-/tree/mai... we are incorectly using os_disto to mathc on the hosrt id i belive.
there are a couple of other ways to do this as well but isolate aggregates in placment and the aggregateimagepropertiesisolation filter are the two solutions that were built specific to adress the windows server licening usecase.
On 11/12/2025 05:52, Andy Speagle wrote:
Yeah... I've done this before with flavors and was looking for a way to schedule instances using aggregates without the need to create more flavors that would pin instances to specific hosts.
The main use case here is that we have a number of windows server based instances that due to licensing need to normally be colocated on a few hypervisors... it would be nice to have a feature that would allow me to do this simply... this is all rather klunky.
On Wed, 2025-12-10 at 16:04 +0000, Sean Mooney wrote:
if you have admin ascess you can do that vai passing the host on server create or on move operations.
https://urldefense.us/v3/__https://docs.openstack.org/python-openstackclient... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*id11_... https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/... https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
with the json filter you could also jsut match on the hypervior hostname or host filed in the hoststate object but its simpler to do it with the native supprot in stead of buildign it your self again in the jsonfilter.
there is also an older way to do this by abusing the AZ
the aviablity zone filed supprot <AZ>:<host>:<node> syntax
so you can just do `:<hostname>` for the az or "my_az:compute- 42"
if you can create flavor that implies your an admin so you shoudl proably just use the host or hypervior_hostname fiels in the server create rerequest.
On 10/12/2025 15:23, Andy Speagle wrote:
Also... if this isn't "the way"... then I'm curious what is the correct path forward.
I'm using terraform to build up my infrastructure on openstack and I'm looking for a way to ensure instances are scheduled on specific hosts. Aggregates seemed the right way. I'd rather not have dozens of different flavors doing this work. Scheduler hints seemed to be the simplest way.
Thanks if advance for any advice.
On Wed, 2025-12-10 at 12:46 +0000, Sean Mooney wrote:
⚠️CAUTION⚠️ This email originated outside of Toyon's network. Do not download attachments or click links unless you know the content is safe.
the aggreates are populated in the hoststate object for usage by other filters and weighers the json filter was not designed to be used with them in general
it was intend for use with the other filed that don't have dynamic subscriptable content. i.e. where the key si not know ahead of time
the operations supprote dby it are basic comparsion that work on singal values with the excption fo in
https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
so you can check fi a instnce uuid is in the set of instnaces on a host that work becasue its a dict keyed by the instance uuid so in or container works since the aggreats are a list you cant check if a host aggregate is in the host aggregates list because you woudl have to construct a host aggret object for in to work
so as i said above aggregates are not supported by this filter as it was not part of its original design.
it was orginally inteded form matching onf files like hypervior_type or version https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
On 10/12/2025 01:43, Andy Speagle wrote:
Hey Team,
I'm trying to use scheduler hints using the JsonFilter that involve host aggregates. Based on the documentation and my quick perusal of the scheduler's HostState class... it seems like $aggregates are populated there for usage, but I can't seem to find the right magick to make a json query work to schedule an instance on a host that belongs to a specific aggregate based on simply the name of the aggregate... or perhaps metadata contained within.
I think my lack of understanding of the structure of the $aggregates list in the scheduler's HostState class is tripping me up.
Does anyone have any insight into this?...
Thanks.
-- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431
scheduler hints are backed into the instance request at server create and cant be changed after that point. they will be applied to all move operations as well. there is no way to pass scheduler hint on live migrate evacuate or cold migrate/resize https://docs.openstack.org/api-ref/compute/#id129 https://docs.openstack.org/api-ref/compute/#migrate-server-migrate-action https://docs.openstack.org/api-ref/compute/#resize-server-resize-action https://docs.openstack.org/api-ref/compute/#evacuate-server-evacuate-action we call out the fact that """Hints are not more “dynamic” than flavor extra specs. The end user specifies a flavor and optionally a hint when creating a server, but ultimately what they can specify is static and defined by the deployment.""" in the docs but hints are the least flexibale aprpoch we have. https://docs.openstack.org/nova/latest/reference/scheduler-hints-vs-flavor-e... so if you force a host with scheduler hints i.e. be specifying a spricfc host and using the json filter it woudl prevent you moving it to a diffent host for mainatnce. if you do it the trait/aggrate way you can also add the trat to a new host or add a new host to the aggregate to enable maintenance if needed but you cant do that with scheduler hints. On 11/12/2025 16:24, Andy Speagle wrote:
Okay... thanks for that. I can go the custom trait route. It feels a like a lot of work for something that should be easy with scheduler hints... and it also makes things difficult in a maintenance scenario when those windoze instances need to be scheduled elsewhere so work can be done on their "licensed" hypervisors.
On Thu, 2025-12-11 at 13:56 +0000, Sean Mooney wrote:
so orignally you ask for a way to not do this via flavor so i provied 2 ways to do it via teh server creeate command using --host or via the az on a per instance baseis
what your acully asking for is effectivly to do it vai the image
we develope a feature precisly for the windows usecase
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/...
you can do it via filter as well if you want too https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/sche...
but the recommend way to do this is to add a requried trait to the winwos images and use the placemet functionality to enforce it for you.
https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
you can declaratively add the CUSTOM_LICENSED_WINDOWS trait to your compute nodes using the provider.yaml feature https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/mana...
if you do not want to do that via the placement apias documetned in https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/...
if nethigh of those options appeal you can use
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/sche...
the eexample is actully lightly wrong because it suggest using
os_distro='windows' you shoudl be useing 'os_type=windows' since that is the image property that enable the hyperv enlightment to improve the performance of windows on qemu/kvm. https://urldefense.us/v3/__https://github.com/openstack/glance/blob/master/e...
os_distor is ment to be "|win2k25||"|i belvie for windows server 2025 with os_version holdign something more human readble if needed. although folk often just use human readbale names instead of the osinfo ones. https://urldefense.us/v3/__https://gitlab.com/libosinfo/osinfo-db/-/tree/mai...
we are incorectly using os_disto to mathc on the hosrt id i belive.
there are a couple of other ways to do this as well but isolate aggregates in placment and the aggregateimagepropertiesisolation filter are the two solutions that were built specific to adress the windows server licening usecase.
On 11/12/2025 05:52, Andy Speagle wrote:
Yeah... I've done this before with flavors and was looking for a way to schedule instances using aggregates without the need to create more flavors that would pin instances to specific hosts.
The main use case here is that we have a number of windows server based instances that due to licensing need to normally be colocated on a few hypervisors... it would be nice to have a feature that would allow me to do this simply... this is all rather klunky.
On Wed, 2025-12-10 at 16:04 +0000, Sean Mooney wrote:
if you have admin ascess you can do that vai passing the host on server create or on move operations.
https://urldefense.us/v3/__https://docs.openstack.org/python-openstackclient...
https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*id11_...
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/...
https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
with the json filter you could also jsut match on the hypervior hostname or host filed in the hoststate object but its simpler to do it with the native supprot in stead of buildign it your self again in the jsonfilter.
there is also an older way to do this by abusing the AZ
the aviablity zone filed supprot <AZ>:<host>:<node> syntax
so you can just do `:<hostname>` for the az or "my_az:compute- 42"
if you can create flavor that implies your an admin so you shoudl proably just use the host or hypervior_hostname fiels in the server create rerequest.
On 10/12/2025 15:23, Andy Speagle wrote:
Also... if this isn't "the way"... then I'm curious what is the correct path forward.
I'm using terraform to build up my infrastructure on openstack and I'm looking for a way to ensure instances are scheduled on specific hosts. Aggregates seemed the right way. I'd rather not have dozens of different flavors doing this work. Scheduler hints seemed to be the simplest way.
Thanks if advance for any advice.
On Wed, 2025-12-10 at 12:46 +0000, Sean Mooney wrote:
⚠️CAUTION⚠️ This email originated outside of Toyon's network. Do not download attachments or click links unless you know the content is safe.
the aggreates are populated in the hoststate object for usage by other filters and weighers the json filter was not designed to be used with them in general
it was intend for use with the other filed that don't have dynamic subscriptable content. i.e. where the key si not know ahead of time
the operations supprote dby it are basic comparsion that work on singal values with the excption fo in
https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
so you can check fi a instnce uuid is in the set of instnaces on a host that work becasue its a dict keyed by the instance uuid so in or container works since the aggreats are a list you cant check if a host aggregate is in the host aggregates list because you woudl have to construct a host aggret object for in to work
so as i said above aggregates are not supported by this filter as it was not part of its original design.
it was orginally inteded form matching onf files like hypervior_type or version https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov...
On 10/12/2025 01:43, Andy Speagle wrote: > Hey Team, > > I'm trying to use scheduler hints using the JsonFilter that > involve > host aggregates. Based on the documentation and my quick > perusal of > the > scheduler's HostState class... it seems like $aggregates > are > populated > there for usage, but I can't seem to find the right magick > to > make > a > json query work to schedule an instance on a host that > belongs > to a > specific aggregate based on simply the name of the > aggregate... > or > perhaps metadata contained within. > > I think my lack of understanding of the structure of the > $aggregates > list in the scheduler's HostState class is tripping me up. > > Does anyone have any insight into this?... > > Thanks.
That's a perfectly fair point. Thanks for that. I've implemented this and it seems to be working. Cheers. On Thu, 2025-12-11 at 17:11 +0000, Sean Mooney wrote:
scheduler hints are backed into the instance request at server create and cant be changed after that point. they will be applied to all move operations as well. there is no way to pass scheduler hint on live migrate evacuate or cold migrate/resize
https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*id129... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*migra... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*resiz... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*evacu...
we call out the fact that """Hints are not more “dynamic” than flavor extra specs. The end user specifies a flavor and optionally a hint when creating a server, but ultimately what they can specify is static and defined by the deployment.""" in the docs but hints are the least flexibale aprpoch we have.
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/...
so if you force a host with scheduler hints i.e. be specifying a spricfc host and using the json filter it woudl prevent you moving it to a diffent host for mainatnce.
if you do it the trait/aggrate way you can also add the trat to a new host or add a new host to the aggregate to enable maintenance if needed but you cant do that with scheduler hints.
On 11/12/2025 16:24, Andy Speagle wrote:
Okay... thanks for that. I can go the custom trait route. It feels a like a lot of work for something that should be easy with scheduler hints... and it also makes things difficult in a maintenance scenario when those windoze instances need to be scheduled elsewhere so work can be done on their "licensed" hypervisors.
On Thu, 2025-12-11 at 13:56 +0000, Sean Mooney wrote:
so orignally you ask for a way to not do this via flavor so i provied 2 ways to do it via teh server creeate command using --host or via the az on a per instance baseis
what your acully asking for is effectivly to do it vai the image
we develope a feature precisly for the windows usecase
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/... you can do it via filter as well if you want too https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/sche... but the recommend way to do this is to add a requried trait to the winwos images and use the placemet functionality to enforce it for you.
https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
you can declaratively add the CUSTOM_LICENSED_WINDOWS trait to your compute nodes using the provider.yaml feature https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/... https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/mana...
if you do not want to do that via the placement apias documetned in https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/...
if nethigh of those options appeal you can use
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/sche...
the eexample is actully lightly wrong because it suggest using
os_distro='windows' you shoudl be useing 'os_type=windows' since that is the image property that enable the hyperv enlightment to improve the performance of windows on qemu/kvm. https://urldefense.us/v3/__https://github.com/openstack/glance/blob/master/e...
os_distor is ment to be "|win2k25||"|i belvie for windows server 2025 with os_version holdign something more human readble if needed. although folk often just use human readbale names instead of the osinfo ones. https://urldefense.us/v3/__https://gitlab.com/libosinfo/osinfo-db/-/tree/mai... we are incorectly using os_disto to mathc on the hosrt id i belive.
there are a couple of other ways to do this as well but isolate aggregates in placment and the aggregateimagepropertiesisolation filter are the two solutions that were built specific to adress the windows server licening usecase.
On 11/12/2025 05:52, Andy Speagle wrote:
Yeah... I've done this before with flavors and was looking for a way to schedule instances using aggregates without the need to create more flavors that would pin instances to specific hosts.
The main use case here is that we have a number of windows server based instances that due to licensing need to normally be colocated on a few hypervisors... it would be nice to have a feature that would allow me to do this simply... this is all rather klunky.
On Wed, 2025-12-10 at 16:04 +0000, Sean Mooney wrote:
if you have admin ascess you can do that vai passing the host on server create or on move operations.
https://urldefense.us/v3/__https://docs.openstack.org/python-openstackclient... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*id11_... https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/... https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
with the json filter you could also jsut match on the hypervior hostname or host filed in the hoststate object but its simpler to do it with the native supprot in stead of buildign it your self again in the jsonfilter.
there is also an older way to do this by abusing the AZ
the aviablity zone filed supprot <AZ>:<host>:<node> syntax
so you can just do `:<hostname>` for the az or "my_az:compute- 42"
if you can create flavor that implies your an admin so you shoudl proably just use the host or hypervior_hostname fiels in the server create rerequest.
On 10/12/2025 15:23, Andy Speagle wrote:
Also... if this isn't "the way"... then I'm curious what is the correct path forward.
I'm using terraform to build up my infrastructure on openstack and I'm looking for a way to ensure instances are scheduled on specific hosts. Aggregates seemed the right way. I'd rather not have dozens of different flavors doing this work. Scheduler hints seemed to be the simplest way.
Thanks if advance for any advice.
On Wed, 2025-12-10 at 12:46 +0000, Sean Mooney wrote: > ⚠️CAUTION⚠️ > This email originated outside of Toyon's network. Do not > download > attachments or click links unless you know the content is > safe. > > the aggreates are populated in the hoststate object for > usage > by > other > filters and weighers > the json filter was not designed to be used with them in > general > > it was intend for use with the other filed that don't > have > dynamic > subscriptable content. > i.e. where the key si not know ahead of time > > the operations supprote dby it are basic comparsion that > work > on > singal > values with the excption fo in > > https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov... > > so you can check fi a instnce uuid is in the set of > instnaces > on > a > host > that work becasue its a dict keyed by the instance uuid > so in > or > container works > since the aggreats are a list you cant check if a host > aggregate > is > in > the host aggregates list > because you woudl have to construct a host aggret object > for > in > to > work > > so as i said above aggregates are not supported by this > filter as > it > was > not part of its original design. > > it was orginally inteded form matching onf files like > hypervior_type > or > version > https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov... > > On 10/12/2025 01:43, Andy Speagle wrote: > > Hey Team, > > > > I'm trying to use scheduler hints using the JsonFilter > > that > > involve > > host aggregates. Based on the documentation and my > > quick > > perusal of > > the > > scheduler's HostState class... it seems like > > $aggregates > > are > > populated > > there for usage, but I can't seem to find the right > > magick > > to > > make > > a > > json query work to schedule an instance on a host that > > belongs > > to a > > specific aggregate based on simply the name of the > > aggregate... > > or > > perhaps metadata contained within. > > > > I think my lack of understanding of the structure of > > the > > $aggregates > > list in the scheduler's HostState class is tripping me > > up. > > > > Does anyone have any insight into this?... > > > > Thanks.
-- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431
Hmm... okay, maybe this isn't working quite like I'd hoped. We use terraform to deploy our instances... and our we build them by creating a volume from our image and then booting the instance from that volume. So... the image info for the instance says this: N/A (booted from volume) I rather expected that this would read the image uuid from the boot volume... but that doesn't appear to be the case... Is there any way forward in this use case? On Thu, 2025-12-11 at 17:11 +0000, Sean Mooney wrote:
scheduler hints are backed into the instance request at server create and cant be changed after that point. they will be applied to all move operations as well. there is no way to pass scheduler hint on live migrate evacuate or cold migrate/resize
https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*id129... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*migra... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*resiz... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*evacu...
we call out the fact that """Hints are not more “dynamic” than flavor extra specs. The end user specifies a flavor and optionally a hint when creating a server, but ultimately what they can specify is static and defined by the deployment.""" in the docs but hints are the least flexibale aprpoch we have.
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/...
so if you force a host with scheduler hints i.e. be specifying a spricfc host and using the json filter it woudl prevent you moving it to a diffent host for mainatnce.
if you do it the trait/aggrate way you can also add the trat to a new host or add a new host to the aggregate to enable maintenance if needed but you cant do that with scheduler hints.
On 11/12/2025 16:24, Andy Speagle wrote:
Okay... thanks for that. I can go the custom trait route. It feels a like a lot of work for something that should be easy with scheduler hints... and it also makes things difficult in a maintenance scenario when those windoze instances need to be scheduled elsewhere so work can be done on their "licensed" hypervisors.
On Thu, 2025-12-11 at 13:56 +0000, Sean Mooney wrote:
so orignally you ask for a way to not do this via flavor so i provied 2 ways to do it via teh server creeate command using --host or via the az on a per instance baseis
what your acully asking for is effectivly to do it vai the image
we develope a feature precisly for the windows usecase
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/... you can do it via filter as well if you want too https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/sche... but the recommend way to do this is to add a requried trait to the winwos images and use the placemet functionality to enforce it for you.
https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
you can declaratively add the CUSTOM_LICENSED_WINDOWS trait to your compute nodes using the provider.yaml feature https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/... https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/mana...
if you do not want to do that via the placement apias documetned in https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/...
if nethigh of those options appeal you can use
https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/admin/sche...
the eexample is actully lightly wrong because it suggest using
os_distro='windows' you shoudl be useing 'os_type=windows' since that is the image property that enable the hyperv enlightment to improve the performance of windows on qemu/kvm. https://urldefense.us/v3/__https://github.com/openstack/glance/blob/master/e...
os_distor is ment to be "|win2k25||"|i belvie for windows server 2025 with os_version holdign something more human readble if needed. although folk often just use human readbale names instead of the osinfo ones. https://urldefense.us/v3/__https://gitlab.com/libosinfo/osinfo-db/-/tree/mai... we are incorectly using os_disto to mathc on the hosrt id i belive.
there are a couple of other ways to do this as well but isolate aggregates in placment and the aggregateimagepropertiesisolation filter are the two solutions that were built specific to adress the windows server licening usecase.
On 11/12/2025 05:52, Andy Speagle wrote:
Yeah... I've done this before with flavors and was looking for a way to schedule instances using aggregates without the need to create more flavors that would pin instances to specific hosts.
The main use case here is that we have a number of windows server based instances that due to licensing need to normally be colocated on a few hypervisors... it would be nice to have a feature that would allow me to do this simply... this is all rather klunky.
On Wed, 2025-12-10 at 16:04 +0000, Sean Mooney wrote:
if you have admin ascess you can do that vai passing the host on server create or on move operations.
https://urldefense.us/v3/__https://docs.openstack.org/python-openstackclient... https://urldefense.us/v3/__https://docs.openstack.org/api-ref/compute/*id11_... https://urldefense.us/v3/__https://docs.openstack.org/nova/latest/reference/... https://urldefense.us/v3/__https://specs.openstack.org/openstack/nova-specs/...
with the json filter you could also jsut match on the hypervior hostname or host filed in the hoststate object but its simpler to do it with the native supprot in stead of buildign it your self again in the jsonfilter.
there is also an older way to do this by abusing the AZ
the aviablity zone filed supprot <AZ>:<host>:<node> syntax
so you can just do `:<hostname>` for the az or "my_az:compute- 42"
if you can create flavor that implies your an admin so you shoudl proably just use the host or hypervior_hostname fiels in the server create rerequest.
On 10/12/2025 15:23, Andy Speagle wrote:
Also... if this isn't "the way"... then I'm curious what is the correct path forward.
I'm using terraform to build up my infrastructure on openstack and I'm looking for a way to ensure instances are scheduled on specific hosts. Aggregates seemed the right way. I'd rather not have dozens of different flavors doing this work. Scheduler hints seemed to be the simplest way.
Thanks if advance for any advice.
On Wed, 2025-12-10 at 12:46 +0000, Sean Mooney wrote: > ⚠️CAUTION⚠️ > This email originated outside of Toyon's network. Do not > download > attachments or click links unless you know the content is > safe. > > the aggreates are populated in the hoststate object for > usage > by > other > filters and weighers > the json filter was not designed to be used with them in > general > > it was intend for use with the other filed that don't > have > dynamic > subscriptable content. > i.e. where the key si not know ahead of time > > the operations supprote dby it are basic comparsion that > work > on > singal > values with the excption fo in > > https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov... > > so you can check fi a instnce uuid is in the set of > instnaces > on > a > host > that work becasue its a dict keyed by the instance uuid > so in > or > container works > since the aggreats are a list you cant check if a host > aggregate > is > in > the host aggregates list > because you woudl have to construct a host aggret object > for > in > to > work > > so as i said above aggregates are not supported by this > filter as > it > was > not part of its original design. > > it was orginally inteded form matching onf files like > hypervior_type > or > version > https://urldefense.us/v3/__https://github.com/openstack/nova/blob/master/nov... > > On 10/12/2025 01:43, Andy Speagle wrote: > > Hey Team, > > > > I'm trying to use scheduler hints using the JsonFilter > > that > > involve > > host aggregates. Based on the documentation and my > > quick > > perusal of > > the > > scheduler's HostState class... it seems like > > $aggregates > > are > > populated > > there for usage, but I can't seem to find the right > > magick > > to > > make > > a > > json query work to schedule an instance on a host that > > belongs > > to a > > specific aggregate based on simply the name of the > > aggregate... > > or > > perhaps metadata contained within. > > > > I think my lack of understanding of the structure of > > the > > $aggregates > > list in the scheduler's HostState class is tripping me > > up. > > > > Does anyone have any insight into this?... > > > > Thanks.
-- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431
participants (2)
-
Andy Speagle
-
Sean Mooney