[nova] allow only specific instances on host aggregate
Hi, I'm playing around with host aggregates and the aggregate_instance_extra_specs for flavors. I successfully created a host aggregate, it allows me to schedule instances with a specific flavor on that aggregate only. But is there a way to prevent other VMs from starting on that aggregate (which were launched with different flavors)? The only thing coming to mind is to put that compute node(s) into a different availbility zone. Since the default is "nova", this would prevent anything else to start on that hypervisor. Is there anything I missed with aggregates? Any pointers are appreciated! Thanks! Eugen
Hello. I think we can meta data to control instanes spw or not spw on host aggregate. On Thu, Apr 25, 2024, 9:39 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I'm playing around with host aggregates and the aggregate_instance_extra_specs for flavors. I successfully created a host aggregate, it allows me to schedule instances with a specific flavor on that aggregate only. But is there a way to prevent other VMs from starting on that aggregate (which were launched with different flavors)? The only thing coming to mind is to put that compute node(s) into a different availbility zone. Since the default is "nova", this would prevent anything else to start on that hypervisor. Is there anything I missed with aggregates? Any pointers are appreciated!
Thanks! Eugen
On Thu, 2024-04-25 at 16:32 +0200, Eugen Block wrote:
Hi,
I'm playing around with host aggregates and the aggregate_instance_extra_specs for flavors. I successfully created a host aggregate, it allows me to schedule instances with a specific flavor on that aggregate only. But is there a way to prevent other VMs from starting on that aggregate (which were launched with different flavors)?
The only thing coming to mind is to put that compute node(s) into a different availbility zone. Since the default is "nova", this would prevent anything else to start on that hypervisor. Is there anything I missed with aggregates? Any pointers are appreciated! you could use AZ but that si not what its for. even with an az you would still need to use one of the filters above or write your own to do the enforcemnt
there are a few ways depending on exactly what you want to do and if you want to write your own filter or not the first way to do this is via https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html if you use isolating aggreates in placment you first have to tag each of the host in that aggreate with a trait then you configure that trait as a required trait in the host agreate metadata finally you create a flavor that request that required trait. the easies way to tag the compute nodes with that custom trait it to use provider.yaml https://docs.openstack.org/nova/latest/admin/managing-resource-providers.htm... but you can also do ti via the rest api this will enforce the isolation in placment instead of nova and is the most scalable solution you can use the https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregateinstan... filter to do this to do this but its much slower then using placemnt the next approch which is the one i would recomend is to use the AggregateTypeAffinityFilter https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregatetypeaf... this is the older way to do that and was designed for this exact usecase dont be fooled by the name instance_type is what flavors used to be called. the AggregateTypeAffinityFilter allwo you to create a host aggrate and then spcify a comma seperated list of flavors that are allowed the main downside is that thre is a change that if the placement result set is set too low (default is 1000 allcoation candiates) on a large cloud you might not get an allcoation candiate for one of the host in that aggreate. if you have less then 1000 compute nodes that not na isssue. that is what i ment by writing code. if the exsting filters do not work you can write one out of tree for your own deployment but i belive the AggregateTypeAffinityFilter is what you are loooking for.
Thanks! Eugen
Sean, as always you're incredibly helpful. I think the AggregateTypeAffinityFilter is the right choice here (I'm supporting a customer, so it's not always easy to be on the same page). I did some limited tests and it seems to do exactly what I hoped for. Now the customer will continue with their tests. Implementing a custom filter is definitely not necessary. I was not aware that the AZ part depends on other filters as well, thanks for the pointer. Thanks again, I really appreciate it! Eugen Zitat von smooney@redhat.com:
On Thu, 2024-04-25 at 16:32 +0200, Eugen Block wrote:
Hi,
I'm playing around with host aggregates and the aggregate_instance_extra_specs for flavors. I successfully created a host aggregate, it allows me to schedule instances with a specific flavor on that aggregate only. But is there a way to prevent other VMs from starting on that aggregate (which were launched with different flavors)?
there are a few ways depending on exactly what you want to do and if you want to write your own filter or not
the first way to do this is via https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html
if you use isolating aggreates in placment you first have to tag each of the host in that aggreate with a trait then you configure that trait as a required trait in the host agreate metadata finally you create a flavor that request that required trait.
the easies way to tag the compute nodes with that custom trait it to use provider.yaml https://docs.openstack.org/nova/latest/admin/managing-resource-providers.htm... but you can also do ti via the rest api
this will enforce the isolation in placment instead of nova and is the most scalable solution you can use the https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregateinstan... filter to do this to do this but its much slower then using placemnt
the next approch which is the one i would recomend is to use the AggregateTypeAffinityFilter https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregatetypeaf... this is the older way to do that and was designed for this exact usecase dont be fooled by the name instance_type is what flavors used to be called.
the AggregateTypeAffinityFilter allwo you to create a host aggrate and then spcify a comma seperated list of flavors that are allowed the main downside is that thre is a change that if the placement result set is set too low (default is 1000 allcoation candiates) on a large cloud you might not get an allcoation candiate for one of the host in that aggreate.
if you have less then 1000 compute nodes that not na isssue.
The only thing coming to mind is to put that compute node(s) into a different availbility zone. Since the default is "nova", this would prevent anything else to start on that hypervisor. Is there anything I missed with aggregates? Any pointers are appreciated! you could use AZ but that si not what its for. even with an az you would still need to use one of the filters above or write your own to do the enforcemnt that is what i ment by writing code. if the exsting filters do not work you can write one out of tree for your own deployment but i belive the AggregateTypeAffinityFilter is what you are loooking for.
Thanks! Eugen
Hi again, although it started promising, the tests with a bunch of VMs are not scheduled on the target compute node. This part I don't understand, I would have expected the scheduler to ignore other compute nodes alltogether. I added a sleep 30 command in the for loop when creating 5 tiny VMs (cirros image), and only 3 of them were scheduled on the target node, the other two were built somewhere else (why though?). Adding sleep 60 spawns all 5 VMs on the target host. So there seems to be some timing issue with the scheduler? There are 3 control nodes in this cloud, running on Victoria version. Is this affinity filter only some kind of "soft" filter? How could I enforce it? Thanks, Eugen Zitat von Eugen Block <eblock@nde.ag>:
Sean, as always you're incredibly helpful. I think the AggregateTypeAffinityFilter is the right choice here (I'm supporting a customer, so it's not always easy to be on the same page). I did some limited tests and it seems to do exactly what I hoped for. Now the customer will continue with their tests. Implementing a custom filter is definitely not necessary. I was not aware that the AZ part depends on other filters as well, thanks for the pointer. Thanks again, I really appreciate it! Eugen
Zitat von smooney@redhat.com:
On Thu, 2024-04-25 at 16:32 +0200, Eugen Block wrote:
Hi,
I'm playing around with host aggregates and the aggregate_instance_extra_specs for flavors. I successfully created a host aggregate, it allows me to schedule instances with a specific flavor on that aggregate only. But is there a way to prevent other VMs from starting on that aggregate (which were launched with different flavors)?
there are a few ways depending on exactly what you want to do and if you want to write your own filter or not
the first way to do this is via https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html
if you use isolating aggreates in placment you first have to tag each of the host in that aggreate with a trait then you configure that trait as a required trait in the host agreate metadata finally you create a flavor that request that required trait.
the easies way to tag the compute nodes with that custom trait it to use provider.yaml https://docs.openstack.org/nova/latest/admin/managing-resource-providers.htm... but you can also do ti via the rest api
this will enforce the isolation in placment instead of nova and is the most scalable solution you can use the https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregateinstan... filter to do this to do this but its much slower then using placemnt
the next approch which is the one i would recomend is to use the AggregateTypeAffinityFilter https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregatetypeaf... this is the older way to do that and was designed for this exact usecase dont be fooled by the name instance_type is what flavors used to be called.
the AggregateTypeAffinityFilter allwo you to create a host aggrate and then spcify a comma seperated list of flavors that are allowed the main downside is that thre is a change that if the placement result set is set too low (default is 1000 allcoation candiates) on a large cloud you might not get an allcoation candiate for one of the host in that aggreate.
if you have less then 1000 compute nodes that not na isssue.
The only thing coming to mind is to put that compute node(s) into a different availbility zone. Since the default is "nova", this would prevent anything else to start on that hypervisor. Is there anything I missed with aggregates? Any pointers are appreciated! you could use AZ but that si not what its for. even with an az you would still need to use one of the filters above or write your own to do the enforcemnt that is what i ment by writing code. if the exsting filters do not work you can write one out of tree for your own deployment but i belive the AggregateTypeAffinityFilter is what you are loooking for.
Thanks! Eugen
On Fri, 2024-04-26 at 08:01 +0000, Eugen Block wrote:
Hi again,
although it started promising, the tests with a bunch of VMs are not scheduled on the target compute node. This part I don't understand, I would have expected the scheduler to ignore other compute nodes alltogether.
so the way the filters work in nova is the remove hosts if they dont meet the critia of the filter. if you want to use soly a filter the every host must be in an aggreate that is resticted to a specific falvor any host that dont have a falvor resriction will be valid for all vms the placemetn approch with the isolationg aggretes works teh other way you are making an explcit request for somethign that is only advertised on specific hosts and enforcing that if you dont ask for that you are not allowed to use other hosts so using placment traits you can get the behavior you were hoping for. that why i presented that option first even if it is more complicated. effectivly you would tag the host with CUSTOM_FLAVOR_<FLAVOR_NAME> add a requried trati to the flavor openstack flavor set --property traits:CUSTOM_FLAVOR_<FLAVOR_NAME>=required <FLAVOR_NAME> and configre the host aggreat to require that trait in the request
I added a sleep 30 command in the for loop when creating 5 tiny VMs (cirros image), and only 3 of them were scheduled on the target node, the other two were built somewhere else (why though?). Adding sleep 60 spawns all 5 VMs on the target host. So there seems to be some timing issue with the scheduler? There are 3 control nodes in this cloud, running on Victoria version. Is this affinity filter only some kind of "soft" filter? no its hard affinity but it only applies the the the host in the aggreate that you define the metadata on it does not apply to all the other hosts. unlike the tenant isolation filter we dont have a way to reject all hosts that done have a flavor mapped to them so if you only using the AggregateTypeAffinityFilter you will unfortunetly need to ensure every host is added to an aggreate that is configured for specific flavors. so that there are not host that are unresticted. How could I enforce it?
Thanks, Eugen
Zitat von Eugen Block <eblock@nde.ag>:
Sean, as always you're incredibly helpful. I think the AggregateTypeAffinityFilter is the right choice here (I'm supporting a customer, so it's not always easy to be on the same page). I did some limited tests and it seems to do exactly what I hoped for. Now the customer will continue with their tests. Implementing a custom filter is definitely not necessary. I was not aware that the AZ part depends on other filters as well, thanks for the pointer. Thanks again, I really appreciate it! Eugen
Zitat von smooney@redhat.com:
On Thu, 2024-04-25 at 16:32 +0200, Eugen Block wrote:
Hi,
I'm playing around with host aggregates and the aggregate_instance_extra_specs for flavors. I successfully created a host aggregate, it allows me to schedule instances with a specific flavor on that aggregate only. But is there a way to prevent other VMs from starting on that aggregate (which were launched with different flavors)?
there are a few ways depending on exactly what you want to do and if you want to write your own filter or not
the first way to do this is via https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html
if you use isolating aggreates in placment you first have to tag each of the host in that aggreate with a trait then you configure that trait as a required trait in the host agreate metadata finally you create a flavor that request that required trait.
the easies way to tag the compute nodes with that custom trait it to use provider.yaml https://docs.openstack.org/nova/latest/admin/managing-resource-providers.htm... but you can also do ti via the rest api
this will enforce the isolation in placment instead of nova and is the most scalable solution you can use the https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregateinstan... filter to do this to do this but its much slower then using placemnt
the next approch which is the one i would recomend is to use the AggregateTypeAffinityFilter https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregatetypeaf... this is the older way to do that and was designed for this exact usecase dont be fooled by the name instance_type is what flavors used to be called.
the AggregateTypeAffinityFilter allwo you to create a host aggrate and then spcify a comma seperated list of flavors that are allowed the main downside is that thre is a change that if the placement result set is set too low (default is 1000 allcoation candiates) on a large cloud you might not get an allcoation candiate for one of the host in that aggreate.
if you have less then 1000 compute nodes that not na isssue.
The only thing coming to mind is to put that compute node(s) into a different availbility zone. Since the default is "nova", this would prevent anything else to start on that hypervisor. Is there anything I missed with aggregates? Any pointers are appreciated! you could use AZ but that si not what its for. even with an az you would still need to use one of the filters above or write your own to do the enforcemnt that is what i ment by writing code. if the exsting filters do not work you can write one out of tree for your own deployment but i belive the AggregateTypeAffinityFilter is what you are loooking for.
Thanks! Eugen
Alright, thanks for the explanation. I will try the trait approach on Monday then. Have a nice weekend! Zitat von smooney@redhat.com:
On Fri, 2024-04-26 at 08:01 +0000, Eugen Block wrote:
Hi again,
although it started promising, the tests with a bunch of VMs are not scheduled on the target compute node. This part I don't understand, I would have expected the scheduler to ignore other compute nodes alltogether.
so the way the filters work in nova is the remove hosts if they dont meet the critia of the filter.
if you want to use soly a filter the every host must be in an aggreate that is resticted to a specific falvor
any host that dont have a falvor resriction will be valid for all vms
the placemetn approch with the isolationg aggretes works teh other way
you are making an explcit request for somethign that is only advertised on specific hosts and enforcing that if you dont ask for that you are not allowed to use other hosts
so using placment traits you can get the behavior you were hoping for.
that why i presented that option first even if it is more complicated.
effectivly you would tag the host with CUSTOM_FLAVOR_<FLAVOR_NAME>
add a requried trati to the flavor
openstack flavor set --property traits:CUSTOM_FLAVOR_<FLAVOR_NAME>=required <FLAVOR_NAME>
and configre the host aggreat to require that trait in the request
I added a sleep 30 command in the for loop when creating 5 tiny VMs (cirros image), and only 3 of them were scheduled on the target node, the other two were built somewhere else (why though?). Adding sleep 60 spawns all 5 VMs on the target host. So there seems to be some timing issue with the scheduler? There are 3 control nodes in this cloud, running on Victoria version. Is this affinity filter only some kind of "soft" filter? no its hard affinity but it only applies the the the host in the aggreate that you define the metadata on it does not apply to all the other hosts. unlike the tenant isolation filter we dont have a way to reject all hosts that done have a flavor mapped to them so if you only using the AggregateTypeAffinityFilter you will unfortunetly need to ensure every host is added to an aggreate that is configured for specific flavors. so that there are not host that are unresticted. How could I enforce it?
Thanks, Eugen
Zitat von Eugen Block <eblock@nde.ag>:
Sean, as always you're incredibly helpful. I think the AggregateTypeAffinityFilter is the right choice here (I'm supporting a customer, so it's not always easy to be on the same page). I did some limited tests and it seems to do exactly what I hoped for. Now the customer will continue with their tests. Implementing a custom filter is definitely not necessary. I was not aware that the AZ part depends on other filters as well, thanks for the pointer. Thanks again, I really appreciate it! Eugen
Zitat von smooney@redhat.com:
On Thu, 2024-04-25 at 16:32 +0200, Eugen Block wrote:
Hi,
I'm playing around with host aggregates and the aggregate_instance_extra_specs for flavors. I successfully created a host aggregate, it allows me to schedule instances with a specific flavor on that aggregate only. But is there a way to prevent other VMs from starting on that aggregate (which were launched with different flavors)?
there are a few ways depending on exactly what you want to do and if you want to write your own filter or not
the first way to do this is via https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html
if you use isolating aggreates in placment you first have to tag each of the host in that aggreate with a trait then you configure that trait as a required trait in the host agreate metadata finally you create a flavor that request that required trait.
the easies way to tag the compute nodes with that custom trait it to use provider.yaml
https://docs.openstack.org/nova/latest/admin/managing-resource-providers.htm...
but you can also do ti via the rest api
this will enforce the isolation in placment instead of nova and is the most scalable solution you can use the
https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregateinstan... filter
to do this to do this but its much slower then using placemnt
the next approch which is the one i would recomend is to use the AggregateTypeAffinityFilter
https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregatetypeaf...
this is the older way to do that and was designed for this exact usecase dont be fooled by the name instance_type is what flavors used to be called.
the AggregateTypeAffinityFilter allwo you to create a host aggrate and then spcify a comma seperated list of flavors that are allowed the main downside is that thre is a change that if the placement result set is set too low (default is 1000 allcoation candiates) on a large cloud you might not get an allcoation candiate for one of the host in that aggreate.
if you have less then 1000 compute nodes that not na isssue.
The only thing coming to mind is to put that compute node(s) into a different availbility zone. Since the default is "nova", this would prevent anything else to start on that hypervisor. Is there anything I missed with aggregates? Any pointers are appreciated! you could use AZ but that si not what its for. even with an az you would still need to use one of the filters above or write your own to do the enforcemnt that is what i ment by writing code. if the exsting filters do not work you can write one out of tree for your own deployment but i belive the AggregateTypeAffinityFilter is what you are loooking for.
Thanks! Eugen
Ah, I forwarded the traits approach to the customer and they already tested it successfully. It looks like that does what they expect, thanks again! Zitat von smooney@redhat.com:
On Fri, 2024-04-26 at 08:01 +0000, Eugen Block wrote:
Hi again,
although it started promising, the tests with a bunch of VMs are not scheduled on the target compute node. This part I don't understand, I would have expected the scheduler to ignore other compute nodes alltogether.
so the way the filters work in nova is the remove hosts if they dont meet the critia of the filter.
if you want to use soly a filter the every host must be in an aggreate that is resticted to a specific falvor
any host that dont have a falvor resriction will be valid for all vms
the placemetn approch with the isolationg aggretes works teh other way
you are making an explcit request for somethign that is only advertised on specific hosts and enforcing that if you dont ask for that you are not allowed to use other hosts
so using placment traits you can get the behavior you were hoping for.
that why i presented that option first even if it is more complicated.
effectivly you would tag the host with CUSTOM_FLAVOR_<FLAVOR_NAME>
add a requried trati to the flavor
openstack flavor set --property traits:CUSTOM_FLAVOR_<FLAVOR_NAME>=required <FLAVOR_NAME>
and configre the host aggreat to require that trait in the request
I added a sleep 30 command in the for loop when creating 5 tiny VMs (cirros image), and only 3 of them were scheduled on the target node, the other two were built somewhere else (why though?). Adding sleep 60 spawns all 5 VMs on the target host. So there seems to be some timing issue with the scheduler? There are 3 control nodes in this cloud, running on Victoria version. Is this affinity filter only some kind of "soft" filter? no its hard affinity but it only applies the the the host in the aggreate that you define the metadata on it does not apply to all the other hosts. unlike the tenant isolation filter we dont have a way to reject all hosts that done have a flavor mapped to them so if you only using the AggregateTypeAffinityFilter you will unfortunetly need to ensure every host is added to an aggreate that is configured for specific flavors. so that there are not host that are unresticted. How could I enforce it?
Thanks, Eugen
Zitat von Eugen Block <eblock@nde.ag>:
Sean, as always you're incredibly helpful. I think the AggregateTypeAffinityFilter is the right choice here (I'm supporting a customer, so it's not always easy to be on the same page). I did some limited tests and it seems to do exactly what I hoped for. Now the customer will continue with their tests. Implementing a custom filter is definitely not necessary. I was not aware that the AZ part depends on other filters as well, thanks for the pointer. Thanks again, I really appreciate it! Eugen
Zitat von smooney@redhat.com:
On Thu, 2024-04-25 at 16:32 +0200, Eugen Block wrote:
Hi,
I'm playing around with host aggregates and the aggregate_instance_extra_specs for flavors. I successfully created a host aggregate, it allows me to schedule instances with a specific flavor on that aggregate only. But is there a way to prevent other VMs from starting on that aggregate (which were launched with different flavors)?
there are a few ways depending on exactly what you want to do and if you want to write your own filter or not
the first way to do this is via https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html
if you use isolating aggreates in placment you first have to tag each of the host in that aggreate with a trait then you configure that trait as a required trait in the host agreate metadata finally you create a flavor that request that required trait.
the easies way to tag the compute nodes with that custom trait it to use provider.yaml
https://docs.openstack.org/nova/latest/admin/managing-resource-providers.htm...
but you can also do ti via the rest api
this will enforce the isolation in placment instead of nova and is the most scalable solution you can use the
https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregateinstan... filter
to do this to do this but its much slower then using placemnt
the next approch which is the one i would recomend is to use the AggregateTypeAffinityFilter
https://docs.openstack.org/nova/latest/admin/scheduling.html#aggregatetypeaf...
this is the older way to do that and was designed for this exact usecase dont be fooled by the name instance_type is what flavors used to be called.
the AggregateTypeAffinityFilter allwo you to create a host aggrate and then spcify a comma seperated list of flavors that are allowed the main downside is that thre is a change that if the placement result set is set too low (default is 1000 allcoation candiates) on a large cloud you might not get an allcoation candiate for one of the host in that aggreate.
if you have less then 1000 compute nodes that not na isssue.
The only thing coming to mind is to put that compute node(s) into a different availbility zone. Since the default is "nova", this would prevent anything else to start on that hypervisor. Is there anything I missed with aggregates? Any pointers are appreciated! you could use AZ but that si not what its for. even with an az you would still need to use one of the filters above or write your own to do the enforcemnt that is what i ment by writing code. if the exsting filters do not work you can write one out of tree for your own deployment but i belive the AggregateTypeAffinityFilter is what you are loooking for.
Thanks! Eugen
participants (3)
-
Eugen Block
-
Nguyễn Hữu Khôi
-
smooney＠redhat.com