[nova] force pinning instances to AZ
Hey all, I have a region with multiple AZ. If the server was not created in a specific zone then it is free to be moved to other zones. Is there any way in nova config to prevent this? Regards, Arnaud.
Hello, IIRC this is the reason we are setting `default_schedule_zone` in nova.conf so that the zone is populated explicitly in the request_spec in the database so that it can’t be moved to another zone later, but im not 100% sure. Obviously the drawback being that you populate one zone only if none is selected. I don’t think there is an option to “schedule to any zone and then hardcode it in request_spec”, we would be interested if that’s available today or something that could be added. /Tobias
On 19 Dec 2024, at 11:54, Arnaud Morin <arnaud.morin@gmail.com> wrote:
[You don't often get email from arnaud.morin@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Hey all,
I have a region with multiple AZ. If the server was not created in a specific zone then it is free to be moved to other zones.
Is there any way in nova config to prevent this?
Regards,
Arnaud.
Hi, (Very) interested in this as well. So: +1 Cheers, Kees On 19-12-2024 12:36, Tobias Urdin - Binero IT wrote:
IIRC this is the reason we are setting `default_schedule_zone` in nova.conf so that the zone is populated explicitly in the request_spec in the database so that it can’t be moved to another zone later, but im not 100% sure.
Obviously the drawback being that you populate one zone only if none is selected.
I don’t think there is an option to “schedule to any zone and then hardcode it in request_spec”, we would be interested if that’s available today or something that could be added.
Hi! yes, setting `default_schedule_zone` should help. But it doesn't really help when you don't want to have a default AZ and you just want to prevent a VM from being migrated between AZs. I've described the issue in this bug report some time ago: https://bugs.launchpad.net/nova/+bug/2047182 Unfortunately, it turned out that it's quite hard to fix this behavior :/ On Thu, Dec 19, 2024 at 12:56 PM Kees Meijs | Nefos <keesm@nefos.nl> wrote:
Hi,
(Very) interested in this as well.
So: +1
Cheers, Kees
On 19-12-2024 12:36, Tobias Urdin - Binero IT wrote:
IIRC this is the reason we are setting `default_schedule_zone` in nova.conf so that the zone is populated explicitly in the request_spec in the database so that it can’t be moved to another zone later, but im not 100% sure.
Obviously the drawback being that you populate one zone only if none is selected.
I don’t think there is an option to “schedule to any zone and then hardcode it in request_spec”, we would be interested if that’s available today or something that could be added.
Ok, thank you all for your answers. I see multiples options on the table here: - have my controllers equally split across AZ and set a different default_schedule_zone on each (sounds hacky to me) - rework or add a new parameter that would behave like default_schedule_zone but selecting randomly from a list instead of a string - add some sort of middleware so we can set the availability_zone param if it's absent Cheers, Arnaud On 19.12.24 - 13:03, Damian Dabrowski wrote:
Hi!
yes, setting `default_schedule_zone` should help. But it doesn't really help when you don't want to have a default AZ and you just want to prevent a VM from being migrated between AZs.
I've described the issue in this bug report some time ago: https://bugs.launchpad.net/nova/+bug/2047182
Unfortunately, it turned out that it's quite hard to fix this behavior :/
On Thu, Dec 19, 2024 at 12:56 PM Kees Meijs | Nefos <keesm@nefos.nl> wrote:
Hi,
(Very) interested in this as well.
So: +1
Cheers, Kees
On 19-12-2024 12:36, Tobias Urdin - Binero IT wrote:
IIRC this is the reason we are setting `default_schedule_zone` in nova.conf so that the zone is populated explicitly in the request_spec in the database so that it can’t be moved to another zone later, but im not 100% sure.
Obviously the drawback being that you populate one zone only if none is selected.
I don’t think there is an option to “schedule to any zone and then hardcode it in request_spec”, we would be interested if that’s available today or something that could be added.
Le jeu. 19 déc. 2024 à 14:20, Arnaud Morin <arnaud.morin@gmail.com> a écrit :
Ok, thank you all for your answers. I see multiples options on the table here: - have my controllers equally split across AZ and set a different default_schedule_zone on each (sounds hacky to me) - rework or add a new parameter that would behave like default_schedule_zone but selecting randomly from a list instead of a string - add some sort of middleware so we can set the availability_zone param if it's absent
There is another possibility to pin an instance : if the user attaches it to a volume and if cross_az_attach config option is set to False (default being True), then the instance is pinned to the volume AZ at the creation time. I hear your thoughts on having a UX being "default to a random zone of from a list", this could be a blueprint and IMHO we wouldn't need to add a microversion as this is not a new API behaviour. That said, let's discuss this if you want at some meeting. For your own cloud, well, indeed you only have to use different values for default_schedule_zone (that's a nova-api configuration option, so every API service could use a different value) for the moment. -Sylvain
Cheers, Arnaud
On 19.12.24 - 13:03, Damian Dabrowski wrote:
Hi!
yes, setting `default_schedule_zone` should help. But it doesn't really help when you don't want to have a default AZ and you just want to prevent a VM from being migrated between AZs.
I've described the issue in this bug report some time ago: https://bugs.launchpad.net/nova/+bug/2047182
Unfortunately, it turned out that it's quite hard to fix this behavior :/
On Thu, Dec 19, 2024 at 12:56 PM Kees Meijs | Nefos <keesm@nefos.nl> wrote:
Hi,
(Very) interested in this as well.
So: +1
Cheers, Kees
On 19-12-2024 12:36, Tobias Urdin - Binero IT wrote:
IIRC this is the reason we are setting `default_schedule_zone` in nova.conf so that the zone is populated explicitly in the request_spec in the database so that it can’t be moved to another zone later, but im not 100% sure.
Obviously the drawback being that you populate one zone only if none is selected.
I don’t think there is an option to “schedule to any zone and then hardcode it in request_spec”, we would be interested if that’s available today or something that could be added.
Hi, On 23-12-2024 10:55, Sylvain Bauza wrote:
There is another possibility to pin an instance : if the user attaches it to a volume and if cross_az_attach config option is set to False (default being True), then the instance is pinned to the volume AZ at the creation time.
I'm afraid this one is not true: for example on a resize or migration and accidentally ending up in another AZ results in the Instance having ERROR state. Please don't rely on this one, is all I want to point out. Cheers and enjoy the holidays, Kees
There is another possibility to pin an instance : if the user attaches it to a volume and if cross_az_attach config option is set to False (default being True), then the instance is pinned to the volume AZ at the creation time.
I _think_ this is only true, if Volume was created in advance, but not during the instance creation process. As then request_specs are processed before volume is present, meaning that instance is not pinned to AZ anyway. At least, what was written in the bug report, unless smth changed in Dalmatian.
I hear your thoughts on having a UX being "default to a random zone of from a list", this could be a blueprint and IMHO we wouldn't need to add a microversion as this is not a new API behaviour. That said, let's discuss this if you want at some meeting. For your own cloud, well, indeed you only have to use different values for default_schedule_zone (that's a nova-api configuration option, so every API service could use a different value) for the moment.
-Sylvain
On 12/19/24 14:19, Arnaud Morin wrote:
Ok, thank you all for your answers. I see multiples options on the table here: - have my controllers equally split across AZ and set a different default_schedule_zone on each (sounds hacky to me)
That's exactly what Sylvain B. (PTL of Nova) suggested to me. We implemented it, and it works very well. As our users query Nova to schedule new VMs, they end up in a random API server (one of the 3 backend nova-api behind our haproxy, in our case), and then the VM are scheduled randomly on one of the 3 AZ, as expected. Cheers, Thomas Goirand (zigo)
I would really argue how well it works, as it is a blind random method. There's no awareness about resource availability and it will work well only until you run out of resources in one of AZ, and then you need to track that externally and reconfigure the controller in advance to prevent random scheduling failures. While I agree this works, there's quite some room for improvement. Though, this improvement is gonna be not an easy one for sure. On Tue, 24 Dec 2024, 12:18 Thomas Goirand, <zigo@debian.org> wrote:
On 12/19/24 14:19, Arnaud Morin wrote:
Ok, thank you all for your answers. I see multiples options on the table here: - have my controllers equally split across AZ and set a different default_schedule_zone on each (sounds hacky to me)
That's exactly what Sylvain B. (PTL of Nova) suggested to me. We implemented it, and it works very well. As our users query Nova to schedule new VMs, they end up in a random API server (one of the 3 backend nova-api behind our haproxy, in our case), and then the VM are scheduled randomly on one of the 3 AZ, as expected.
Cheers,
Thomas Goirand (zigo)
Small follow up: And what can make things way worse in this scenario, if you have some trait requirement available only in some AZs (like GPUs). This becomes really, really painful then. On Tue, 24 Dec 2024, 12:23 Dmitriy Rabotyagov, <noonedeadpunk@gmail.com> wrote:
I would really argue how well it works, as it is a blind random method. There's no awareness about resource availability and it will work well only until you run out of resources in one of AZ, and then you need to track that externally and reconfigure the controller in advance to prevent random scheduling failures.
While I agree this works, there's quite some room for improvement. Though, this improvement is gonna be not an easy one for sure.
On Tue, 24 Dec 2024, 12:18 Thomas Goirand, <zigo@debian.org> wrote:
On 12/19/24 14:19, Arnaud Morin wrote:
Ok, thank you all for your answers. I see multiples options on the table here: - have my controllers equally split across AZ and set a different default_schedule_zone on each (sounds hacky to me)
That's exactly what Sylvain B. (PTL of Nova) suggested to me. We implemented it, and it works very well. As our users query Nova to schedule new VMs, they end up in a random API server (one of the 3 backend nova-api behind our haproxy, in our case), and then the VM are scheduled randomly on one of the 3 AZ, as expected.
Cheers,
Thomas Goirand (zigo)
participants (7)
-
Arnaud Morin
-
Damian Dabrowski
-
Dmitriy Rabotyagov
-
Kees Meijs | Nefos
-
Sylvain Bauza
-
Thomas Goirand
-
Tobias Urdin - Binero IT