<div dir="ltr">Bug report created at: <a href="https://bugs.launchpad.net/nova/+bug/2018318">https://bugs.launchpad.net/nova/+bug/2018318</a><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 26, 2023 at 12:23 PM Sylvain Bauza <<a href="mailto:sbauza@redhat.com">sbauza@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mer. 26 avr. 2023 à 14:46, Rafael Weingärtner <<a href="mailto:rafaelweingartner@gmail.com" target="_blank">rafaelweingartner@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Adding the response to this thread, as I replied in the wrong one. Sorry for the confusion.<br><br>Hello Sylvain Bauza and Sean Mooney,<br><br>The patch pointed out in [1]
is doing exactly what you said that we should not do. I mean, changing
the AZ in the request spec of the virtual machine (VM) that the user
created. The patch we propose in [2] is intended to avoid exactly that. <br><br>As
pointed out by Florian, the configuration "cross_az_attach" is a
constraint of the cloud environment, and as such it should be considered
when selecting hosts to execute VMs migrations. Therefore, for those
situations, and only for those (cross_az_attach=False), we send the
current AZ of the VM to placement, to enable it (placement) to filter
out hosts that are not from the current AZ of the VM.<br><br>Looking at
the patch suggested by you in [1], and later testing it, we can confirm
that the problem is still there in main/upstream. This happens because
the patch in [1] is only addressing the cases when the VM is created
based on volumes, and then it sets the AZ of the volumes in the request
spec of the VM. That is why everything works for the setups where
cross_az_attach=False. However, if we create a VM based on an image, and
then it (Nova) creates a new volume in Cinder, the AZ is not set in the
request spec (but it is used to execute the first call to placement to
select the hosts); thus, the issues described in [2] can still happen.<br><br>Anyways,
the proposal presented in [2] is simpler and works nicely. We can
discuss it further in the patchset then, if you guys think it is worth
it.<br><br></div></blockquote><div><br></div><div>As I replied in the gerrit change <a href="https://review.opendev.org/c/openstack/nova/+/864760/comments/4a302ce3_9805e7c6" target="_blank">https://review.opendev.org/c/openstack/nova/+/864760/comments/4a302ce3_9805e7c6</a> then you should create a Launchpad bug report but fwiw, you should also modify the implementation as it would rather do the same for image metadata that what we do for volumes with [1]</div><div><br></div><div>-Sylvain</div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">[1] <a href="https://review.opendev.org/c/openstack/nova/+/469675/12/nova/compute/api.py#1173" target="_blank">https://review.opendev.org/c/openstack/nova/+/469675/12/nova/compute/api.py#1173</a><br>[2] <a href="https://review.opendev.org/c/openstack/nova/+/864760" target="_blank">https://review.opendev.org/c/openstack/nova/+/864760</a></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 29, 2023 at 10:26 AM Nguyễn Hữu Khôi <<a href="mailto:nguyenhuukhoinw@gmail.com" target="_blank">nguyenhuukhoinw@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto">"If they *don't* provide this parameter, then depending on the default_schedule_zone config option, either the instance will eventually use a specific AZ (and then it's like if the enduser was asking for this AZ), or none of AZ is requested and then the instance can be created and moved between any hosts within *all* AZs."</div><div dir="auto"><br></div><div dir="auto">I ask aftet that, although without az when launch instances but they still have az. But i still mv to diffent host in diffent az when mirgrating or spawn which masakari. i am not clear, I tested.</div><div dir="auto"><br style="font-size:12.8px"></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 29, 2023, 7:38 PM Nguyễn Hữu Khôi <<a href="mailto:nguyenhuukhoinw@gmail.com" target="_blank">nguyenhuukhoinw@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Yes. Thanks, but the things I would like to know: after instances are created, how do we know if it was launched with specified AZ or without it? I mean the way to distinguish between specified instances and non specified instances?<div><br clear="all"><div><div dir="ltr"><div dir="ltr">Nguyen Huu Khoi<br></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 29, 2023 at 5:05 PM Sylvain Bauza <<a href="mailto:sbauza@redhat.com" rel="noreferrer" target="_blank">sbauza@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mer. 29 mars 2023 à 08:06, Nguyễn Hữu Khôi <<a href="mailto:nguyenhuukhoinw@gmail.com" rel="noreferrer" target="_blank">nguyenhuukhoinw@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello.<div>I have one question.</div><div>Follow this </div><div><br></div><div><a href="https://docs.openstack.org/nova/latest/admin/availability-zones.html" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/admin/availability-zones.html</a><br></div><div><br></div><div><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px">If the server was not created in a specific zone then it is free to be moved to other zones. but when I use </span><br></div><div><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px"><br></span></div><div>openstack server show [server id]<span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px"><br></span></div><div><br></div><div>I still see the "OS-EXT-AZ:availability_zone" value belonging to my instance.</div><div><br></div></div></blockquote><div><br></div><div>Correct, this is normal. If the operators creates some AZs, then the enduser should see where the instance in which AZ.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px">Could you tell the difference which causes "</span><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px">if the server was not created in a specific zone then it is free to be moved to other zones.</span><span style="color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px">"</span></div><div><br clear="all"></div></div></blockquote><div><br></div><div>To be clear, an operator can create Availability Zones. Those AZs can then be seen by an enduser using the os-availability-zones API [1]. Then, either the enduser wants to use a specific AZ for their next instance creation (and if so, he/she adds --availability-zone parameter to their instance creation client) or they don't want and then they don't provide this parameter.</div><div><br></div><div>If they provide this parameter, then the server will be created only in one host in the specific AZ and then when moving the instance later, it will continue to move to any host within the same AZ.</div><div>If they *don't* provide this parameter, then depending on the default_schedule_zone config option, either the instance will eventually use a specific AZ (and then it's like if the enduser was asking for this AZ), or none of AZ is requested and then the instance can be created and moved between any hosts within *all* AZs.</div><div><br></div><div>That being said, as I said earlier, the enduser can still verify the AZ from where the instance is by the server show parameter you told.<br></div><div><br></div><div>We also have a documentation explaining about Availability Zones, maybe this would help you more to understand about AZs : <a href="https://docs.openstack.org/nova/latest/admin/availability-zones.html" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/admin/availability-zones.html</a></div><div><br></div><div></div><div><br></div><div>[1] <a href="https://docs.openstack.org/api-ref/compute/#availability-zones-os-availability-zone" rel="noreferrer" target="_blank">https://docs.openstack.org/api-ref/compute/#availability-zones-os-availability-zone</a> (tbc, the enduser won't see the hosts, but they can see the list of existing AZs)<br></div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div dir="ltr"><div dir="ltr">Nguyen Huu Khoi<br></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 27, 2023 at 8:37 PM Nguyễn Hữu Khôi <<a href="mailto:nguyenhuukhoinw@gmail.com" rel="noreferrer" target="_blank">nguyenhuukhoinw@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello guys.</div><div><br></div><div>I just suggest to openstack nova works better. My story because </div><div><br></div><ol style="box-sizing:border-box;margin-top:0px;margin-bottom:10px;color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;font-size:14px"><li style="box-sizing:border-box"><p style="box-sizing:border-box;margin:0px 0px 10px">The server was created in a specific zone with the <code style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",monospace;padding:0px;color:rgb(0,0,0);background-color:transparent;border-radius:0px;font-weight:bold"><span style="box-sizing:border-box;font-family:"Courier New",Courier,monospace">POST</span> <span style="box-sizing:border-box;font-family:"Courier New",Courier,monospace">/servers</span></code> request containing the <code style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",monospace;padding:0px;color:rgb(0,0,0);background-color:transparent;border-radius:0px;font-weight:bold"><span style="box-sizing:border-box;font-family:"Courier New",Courier,monospace">availability_zone</span></code> parameter.</p></li></ol><div><font face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen-Sans, Ubuntu, Cantarell, Helvetica Neue, sans-serif" color="#333333"><span style="font-size:14px">It will be nice when we attach randow zone when we create instances then It will only move to the same zone when migrating or masakari ha.</span></font></div><div><font face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen-Sans, Ubuntu, Cantarell, Helvetica Neue, sans-serif" color="#333333"><span style="font-size:14px"><br></span></font></div><div><font face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen-Sans, Ubuntu, Cantarell, Helvetica Neue, sans-serif" color="#333333"><span style="font-size:14px">Currently we can force it to zone by default zone shedule in nova.conf.</span></font></div><div><font face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen-Sans, Ubuntu, Cantarell, Helvetica Neue, sans-serif" color="#333333"><span style="font-size:14px"><br></span></font></div><div><font face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen-Sans, Ubuntu, Cantarell, Helvetica Neue, sans-serif" color="#333333"><span style="font-size:14px">Sorry because I am new to Openstack and I am just an operator. I try to verify some real cases. </span></font></div><div><font face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen-Sans, Ubuntu, Cantarell, Helvetica Neue, sans-serif" color="#333333"><span style="font-size:14px"><br></span></font></div><div><font face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen-Sans, Ubuntu, Cantarell, Helvetica Neue, sans-serif" color="#333333"><span style="font-size:14px"> </span></font></div><div><font face="-apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen-Sans, Ubuntu, Cantarell, Helvetica Neue, sans-serif" color="#333333"><span style="font-size:14px"><br></span></font></div><div><div dir="ltr"><div dir="ltr">Nguyen Huu Khoi<br></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 27, 2023 at 7:43 PM Sylvain Bauza <<a href="mailto:sbauza@redhat.com" rel="noreferrer" target="_blank">sbauza@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le lun. 27 mars 2023 à 14:28, Sean Mooney <<a href="mailto:smooney@redhat.com" rel="noreferrer" target="_blank">smooney@redhat.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, 2023-03-27 at 14:06 +0200, Sylvain Bauza wrote:<br>
> Le lun. 27 mars 2023 à 13:51, Sean Mooney <<a href="mailto:smooney@redhat.com" rel="noreferrer" target="_blank">smooney@redhat.com</a>> a écrit :<br>
> <br>
> > On Mon, 2023-03-27 at 10:19 +0200, Sylvain Bauza wrote:<br>
> > > Le dim. 26 mars 2023 à 14:30, Rafael Weingärtner <<br>
> > > <a href="mailto:rafaelweingartner@gmail.com" rel="noreferrer" target="_blank">rafaelweingartner@gmail.com</a>> a écrit :<br>
> > > <br>
> > > > Hello Nguyễn Hũu Khôi,<br>
> > > > You might want to take a look at:<br>
> > > > <a href="https://review.opendev.org/c/openstack/nova/+/864760" rel="noreferrer noreferrer" target="_blank">https://review.opendev.org/c/openstack/nova/+/864760</a>. We created a<br>
> > patch<br>
> > > > to avoid migrating VMs to any AZ, once the VM has been bootstrapped in<br>
> > an<br>
> > > > AZ that has cross zone attache equals to false.<br>
> > > > <br>
> > > > <br>
> > > Well, I'll provide some comments in the change, but I'm afraid we can't<br>
> > > just modify the request spec like you would want.<br>
> > > <br>
> > > Anyway, if you want to discuss about it in the vPTG, just add it in the<br>
> > > etherpad and add your IRC nick so we could try to find a time where we<br>
> > > could be discussing it : <a href="https://etherpad.opendev.org/p/nova-bobcat-ptg" rel="noreferrer noreferrer" target="_blank">https://etherpad.opendev.org/p/nova-bobcat-ptg</a><br>
> > > Also, this kind of behaviour modification is more a new feature than a<br>
> > > bugfix, so fwiw you should create a launchpad blueprint so we could<br>
> > better<br>
> > > see it.<br>
> > <br>
> > i tought i left review feedback on that too that the approch was not<br>
> > correct.<br>
> > i guess i did not in the end.<br>
> > <br>
> > modifying the request spec as sylvain menthioned is not correct.<br>
> > i disucssed this topic on irc a few weeks back with mohomad for vxhost.<br>
> > what can be done is as follows.<br>
> > <br>
> > we can add a current_az field to the Destination object<br>
> > <br>
> > <a href="https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L1092-L1122" rel="noreferrer noreferrer" target="_blank">https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L1092-L1122</a><br>
> > The conductor can read the instance.AZ and populate it in that new field.<br>
> > We can then add a new weigher to prefer hosts that are in the same az.<br>
> > <br>
> > <br>
> <br>
> I tend to disagree this approach as people would think that the<br>
> Destination.az field would be related to the current AZ for an instance,<br>
> while we only look at the original AZ.<br>
> That being said, we could have a weigher that would look at whether the<br>
> host is in the same AZ than the instance.host.<br>
you miss understood what i wrote<br>
<br>
i suggested addint Destination.current_az to store teh curernt AZ of the instance before scheduling.<br>
<br>
so my proposal is if RequestSpec.AZ is not set and Destination.current_az is set then the new<br>
weigher would prefer hosts that are in the same az as Destination.current_az<br>
<br>
we coudl also call Destination.current_az Destination.prefered_az <br>
<br></blockquote><div><br></div><div>I meant, I think we don't need to provide a new field, we can already know about what host an existing instance uses if we want (using [1])</div><div>Anyway, let's stop to discuss about it here, we should rather review that for a Launchpad blueprint or more a spec.</div><div><br></div><div>-Sylvain<br></div><div><br></div><div>[1] <a href="https://github.com/openstack/nova/blob/b9a49ffb04cb5ae2d8c439361a3552296df02988/nova/scheduler/host_manager.py#L369-L370" rel="noreferrer" target="_blank">https://github.com/openstack/nova/blob/b9a49ffb04cb5ae2d8c439361a3552296df02988/nova/scheduler/host_manager.py#L369-L370</a></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> <br>
> <br>
> This will provide soft AZ affinity for the vm and preserve the fact that if<br>
> > a vm is created without sepcifying<br>
> > An AZ the expectaiton at the api level woudl be that it can migrate to any<br>
> > AZ.<br>
> > <br>
> > To provide hard AZ affintiy we could also add prefileter that would use<br>
> > the same data but instead include it in the<br>
> > placement query so that only the current AZ is considered. This would have<br>
> > to be disabled by default.<br>
> > <br>
> > <br>
> Sure, we could create a new prefilter so we could then deprecate the<br>
> AZFilter if we want.<br>
we already have an AZ prefilter and the AZFilter is deprecate for removal<br>
i ment to delete it in zed but did not have time to do it in zed of Antielope<br>
i deprecated the AZ| filter in <a href="https://github.com/openstack/nova/commit/7c7a2a142d74a7deeda2a79baf21b689fe32cd08" rel="noreferrer noreferrer" target="_blank">https://github.com/openstack/nova/commit/7c7a2a142d74a7deeda2a79baf21b689fe32cd08</a><br>
xena when i enabeld the az prefilter by default.<br>
<br></blockquote><div><br></div><div>Ah whoops, indeed I forgot the fact we already have the prefilter, so the hard support for AZ is already existing.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
i will try an delete teh AZ filter before m1 if others dont.<br></blockquote><div><br></div><div>OK.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> <br>
> <br>
> > That woudl allow operators to choose the desired behavior.<br>
> > curret behavior (disable weigher and dont enabel prefilter)<br>
> > new default, prefer current AZ (weigher enabeld prefilter disabled)<br>
> > hard affintiy(prefilter enabled.)<br>
> > <br>
> > there are other ways to approch this but updating the request spec is not<br>
> > one of them.<br>
> > we have to maintain the fact the enduser did not request an AZ.<br>
> > <br>
> > <br>
> Anyway, if folks want to discuss about AZs, this week is the good time :-)<br>
> <br>
> <br>
> > > <br>
> > > -Sylvain<br>
> > > <br>
> > > <br>
> > > <br>
> > > > On Sun, Mar 26, 2023 at 8:20 AM Nguyễn Hữu Khôi <<br>
> > <a href="mailto:nguyenhuukhoinw@gmail.com" rel="noreferrer" target="_blank">nguyenhuukhoinw@gmail.com</a>><br>
> > > > wrote:<br>
> > > > <br>
> > > > > Hello guys.<br>
> > > > > I playing with Nova AZ and Masakari<br>
> > > > > <br>
> > > > > <a href="https://docs.openstack.org/nova/latest/admin/availability-zones.html" rel="noreferrer noreferrer" target="_blank">https://docs.openstack.org/nova/latest/admin/availability-zones.html</a><br>
> > > > > <br>
> > > > > Masakari will move server by nova scheduler.<br>
> > > > > <br>
> > > > > Openstack Docs describe that:<br>
> > > > > <br>
> > > > > If the server was not created in a specific zone then it is free to<br>
> > be<br>
> > > > > moved to other zones, i.e. the AvailabilityZoneFilter<br>
> > > > > <<br>
> > <a href="https://docs.openstack.org/nova/latest/admin/scheduling.html#availabilityzonefilter" rel="noreferrer noreferrer" target="_blank">https://docs.openstack.org/nova/latest/admin/scheduling.html#availabilityzonefilter</a>><br>
> > is<br>
> > > > > a no-op.<br>
> > > > > <br>
> > > > > I see that everyone usually creates instances with "Any Availability<br>
> > > > > Zone" on Horzion and also we don't specify AZ when creating<br>
> > instances by<br>
> > > > > cli.<br>
> > > > > <br>
> > > > > By this way, when we use Masakari or we miragrated instances( or<br>
> > > > > evacuate) so our instance will be moved to other zones.<br>
> > > > > <br>
> > > > > Can we attach AZ to server create requests API based on Any<br>
> > > > > Availability Zone to limit instances moved to other zones?<br>
> > > > > <br>
> > > > > Thank you. Regards<br>
> > > > > <br>
> > > > > Nguyen Huu Khoi<br>
> > > > > <br>
> > > > <br>
> > > > <br>
> > > > --<br>
> > > > Rafael Weingärtner<br>
> > > > <br>
> > <br>
> > <br>
<br>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all"><br><span>-- </span><br><div dir="ltr"><div dir="ltr">Rafael Weingärtner</div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr">Rafael Weingärtner</div></div>