Open Stack

unfortnetly while i under stand the usecase this bug report is invlaid.

the behavior you are desciribe is the expected and intended behavior of nova.

if a vm does not specify a AZ when it is created and 
https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_schedule_zone
i not set then the expected behavior is that it can migrate acrros az

as a result using [cinder]/cross_az_attach=false without also seeting 
DEFAULT.default_schedule_zone is effectivly an invalid cofniguration.

it is only valid to use [cinder]/cross_az_attach=false and expect move operation to fucntion
if all vms have a AZ requested when they are created and that is the contindion that is not being met
in your current usecase.


i have discsued thsi a few times in the past.
a weigher (not a filter) could be created to give prefernce to an instance
currnent AZ but we cannot prevent the incance form being scduled to a diffent AZ.

This would be a new feature not a bug and should be discussed as such which is why i marked
tbe bug as invalid.


On Tue, 2023-05-02 at 13:12 -0300, Rafael Weingärtner wrote:
> Bug report created at: https://bugs.launchpad.net/nova/+bug/2018318
> 
> On Wed, Apr 26, 2023 at 12:23 PM Sylvain Bauza <sbauza at redhat.com> wrote:
> 
> > 
> > 
> > Le mer. 26 avr. 2023 à 14:46, Rafael Weingärtner <
> > rafaelweingartner at gmail.com> a écrit :
> > 
> > > Adding the response to this thread, as I replied in the wrong one. Sorry
> > > for the confusion.
> > > 
> > > Hello Sylvain Bauza and Sean Mooney,
> > > 
> > > The patch pointed out in [1] is doing exactly what you said that we
> > > should not do. I mean, changing the AZ in the request spec of the virtual
> > > machine (VM) that the user created. The patch we propose in [2] is intended
> > > to avoid exactly that.
> > > 
> > > As pointed out by Florian, the configuration "cross_az_attach" is a
> > > constraint of the cloud environment, and as such it should be considered
> > > when selecting hosts to execute VMs migrations. Therefore, for those
> > > situations, and only for those (cross_az_attach=False), we send the current
> > > AZ of the VM to placement, to enable it (placement) to filter out hosts
> > > that are not from the current AZ of the VM.
> > > 
> > > Looking at the patch suggested by you in [1], and later testing it, we
> > > can confirm that the problem is still there in main/upstream. This happens
> > > because the patch in [1] is only addressing the cases when the VM is
> > > created based on volumes, and then it sets the AZ of the volumes in the
> > > request spec of the VM. That is why everything works for the setups where
> > > cross_az_attach=False. However, if we create a VM based on an image, and
> > > then it (Nova) creates a new volume in Cinder, the AZ is not set in the
> > > request spec (but it is used to execute the first call to placement to
> > > select the hosts); thus, the issues described in [2] can still happen.
> > > 
> > > Anyways, the proposal presented in [2] is simpler and works nicely. We
> > > can discuss it further in the patchset then, if you guys think it is worth
> > > it.
> > > 
> > > 
> > As I replied in the gerrit change
> > https://review.opendev.org/c/openstack/nova/+/864760/comments/4a302ce3_9805e7c6
> > then you should create a Launchpad bug report but fwiw, you should also
> > modify the implementation as it would rather do the same for image metadata
> > that what we do for volumes with [1]
> > 
> > -Sylvain
> > 
> > [1]
> > > https://review.opendev.org/c/openstack/nova/+/469675/12/nova/compute/api.py#1173
> > > [2] https://review.opendev.org/c/openstack/nova/+/864760
> > > 
> > > On Wed, Mar 29, 2023 at 10:26 AM Nguyễn Hữu Khôi <
> > > nguyenhuukhoinw at gmail.com> wrote:
> > > 
> > > > "If they *don't* provide this parameter, then depending on the
> > > > default_schedule_zone config option, either the instance will eventually
> > > > use a specific AZ (and then it's like if the enduser was asking for this
> > > > AZ), or none of AZ is requested and then the instance can be created and
> > > > moved between any hosts within *all* AZs."
> > > > 
> > > > I ask aftet that, although without az when launch instances but they
> > > > still have az. But i still mv to diffent host in diffent az when mirgrating
> > > > or spawn which masakari. i am not clear, I tested.
> > > > 
> > > > 
> > > > On Wed, Mar 29, 2023, 7:38 PM Nguyễn Hữu Khôi <nguyenhuukhoinw at gmail.com>
> > > > wrote:
> > > > 
> > > > > Yes. Thanks, but the things I would like to know: after instances are
> > > > > created, how do we know if it was launched with specified AZ or without it?
> > > > > I mean the way to distinguish between specified instances and non specified
> > > > > instances?
> > > > > 
> > > > > Nguyen Huu Khoi
> > > > > 
> > > > > 
> > > > > On Wed, Mar 29, 2023 at 5:05 PM Sylvain Bauza <sbauza at redhat.com>
> > > > > wrote:
> > > > > 
> > > > > > 
> > > > > > 
> > > > > > Le mer. 29 mars 2023 à 08:06, Nguyễn Hữu Khôi <
> > > > > > nguyenhuukhoinw at gmail.com> a écrit :
> > > > > > 
> > > > > > > Hello.
> > > > > > > I have one question.
> > > > > > > Follow this
> > > > > > > 
> > > > > > > https://docs.openstack.org/nova/latest/admin/availability-zones.html
> > > > > > > 
> > > > > > > If the server was not created in a specific zone then it is free to
> > > > > > > be moved to other zones. but when I use
> > > > > > > 
> > > > > > > openstack server show [server id]
> > > > > > > 
> > > > > > > I still see the "OS-EXT-AZ:availability_zone" value belonging to my
> > > > > > > instance.
> > > > > > > 
> > > > > > > 
> > > > > > Correct, this is normal. If the operators creates some AZs, then the
> > > > > > enduser should see where the instance in which AZ.
> > > > > > 
> > > > > > 
> > > > > > > Could you tell the difference which causes "if the server was not
> > > > > > > created in a specific zone then it is free to be moved to other zones.
> > > > > > > "
> > > > > > > 
> > > > > > > 
> > > > > > To be clear, an operator can create Availability Zones. Those AZs can
> > > > > > then be seen by an enduser using the os-availability-zones API [1]. Then,
> > > > > > either the enduser wants to use a specific AZ for their next instance
> > > > > > creation (and if so, he/she adds --availability-zone parameter to their
> > > > > > instance creation client) or they don't want and then they don't provide
> > > > > > this parameter.
> > > > > > 
> > > > > > If they provide this parameter, then the server will be created only
> > > > > > in one host in the specific AZ and then when moving the instance later, it
> > > > > > will continue to move to any host within the same AZ.
> > > > > > If they *don't* provide this parameter, then depending on the
> > > > > > default_schedule_zone config option, either the instance will eventually
> > > > > > use a specific AZ (and then it's like if the enduser was asking for this
> > > > > > AZ), or none of AZ is requested and then the instance can be created and
> > > > > > moved between any hosts within *all* AZs.
> > > > > > 
> > > > > > That being said, as I said earlier, the enduser can still verify the
> > > > > > AZ from where the instance is by the server show parameter you told.
> > > > > > 
> > > > > > We also have a documentation explaining about Availability Zones,
> > > > > > maybe this would help you more to understand about AZs :
> > > > > > https://docs.openstack.org/nova/latest/admin/availability-zones.html
> > > > > > 
> > > > > > 
> > > > > > [1]
> > > > > > https://docs.openstack.org/api-ref/compute/#availability-zones-os-availability-zone
> > > > > > (tbc, the enduser won't see the hosts, but they can see the list of
> > > > > > existing AZs)
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > > Nguyen Huu Khoi
> > > > > > > 
> > > > > > > 
> > > > > > > On Mon, Mar 27, 2023 at 8:37 PM Nguyễn Hữu Khôi <
> > > > > > > nguyenhuukhoinw at gmail.com> wrote:
> > > > > > > 
> > > > > > > > Hello guys.
> > > > > > > > 
> > > > > > > > I just suggest to openstack nova works better. My story because
> > > > > > > > 
> > > > > > > > 
> > > > > > > >    1.
> > > > > > > > 
> > > > > > > >    The server was created in a specific zone with the POST /servers request
> > > > > > > >    containing the availability_zone parameter.
> > > > > > > > 
> > > > > > > > It will be nice when we attach randow zone when we create instances
> > > > > > > > then It will only move to the same zone when migrating or masakari ha.
> > > > > > > > 
> > > > > > > > Currently we can force it to zone by default zone shedule in
> > > > > > > > nova.conf.
> > > > > > > > 
> > > > > > > > Sorry because I am new to Openstack and I am just an operator. I try
> > > > > > > > to verify some real cases.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Nguyen Huu Khoi
> > > > > > > > 
> > > > > > > > 
> > > > > > > > On Mon, Mar 27, 2023 at 7:43 PM Sylvain Bauza <sbauza at redhat.com>
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Le lun. 27 mars 2023 à 14:28, Sean Mooney <smooney at redhat.com> a
> > > > > > > > > écrit :
> > > > > > > > > 
> > > > > > > > > > On Mon, 2023-03-27 at 14:06 +0200, Sylvain Bauza wrote:
> > > > > > > > > > > Le lun. 27 mars 2023 à 13:51, Sean Mooney <smooney at redhat.com>
> > > > > > > > > > a écrit :
> > > > > > > > > > > 
> > > > > > > > > > > > On Mon, 2023-03-27 at 10:19 +0200, Sylvain Bauza wrote:
> > > > > > > > > > > > > Le dim. 26 mars 2023 à 14:30, Rafael Weingärtner <
> > > > > > > > > > > > > rafaelweingartner at gmail.com> a écrit :
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Hello Nguyễn Hũu Khôi,
> > > > > > > > > > > > > > You might want to take a look at:
> > > > > > > > > > > > > > https://review.opendev.org/c/openstack/nova/+/864760. We
> > > > > > > > > > created a
> > > > > > > > > > > > patch
> > > > > > > > > > > > > > to avoid migrating VMs to any AZ, once the VM has been
> > > > > > > > > > bootstrapped in
> > > > > > > > > > > > an
> > > > > > > > > > > > > > AZ that has cross zone attache equals to false.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > Well, I'll provide some comments in the change, but I'm
> > > > > > > > > > afraid we can't
> > > > > > > > > > > > > just modify the request spec like you would want.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Anyway, if you want to discuss about it in the vPTG, just
> > > > > > > > > > add it in the
> > > > > > > > > > > > > etherpad and add your IRC nick so we could try to find a
> > > > > > > > > > time where we
> > > > > > > > > > > > > could be discussing it :
> > > > > > > > > > https://etherpad.opendev.org/p/nova-bobcat-ptg
> > > > > > > > > > > > > Also, this kind of behaviour modification is more a new
> > > > > > > > > > feature than a
> > > > > > > > > > > > > bugfix, so fwiw you should create a launchpad blueprint so
> > > > > > > > > > we could
> > > > > > > > > > > > better
> > > > > > > > > > > > > see it.
> > > > > > > > > > > > 
> > > > > > > > > > > > i tought i left review feedback on that too that the approch
> > > > > > > > > > was not
> > > > > > > > > > > > correct.
> > > > > > > > > > > > i guess i did not in the end.
> > > > > > > > > > > > 
> > > > > > > > > > > > modifying the request spec as sylvain menthioned is not
> > > > > > > > > > correct.
> > > > > > > > > > > > i disucssed this topic on irc a few weeks back with mohomad
> > > > > > > > > > for vxhost.
> > > > > > > > > > > > what can be done is as follows.
> > > > > > > > > > > > 
> > > > > > > > > > > > we can add a current_az field  to the Destination object
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L1092-L1122
> > > > > > > > > > > > The conductor can read the instance.AZ and populate it in that
> > > > > > > > > > new field.
> > > > > > > > > > > > We can then add a new weigher to prefer hosts that are in the
> > > > > > > > > > same az.
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > I tend to disagree this approach as people would think that the
> > > > > > > > > > > Destination.az field would be related to the current AZ for an
> > > > > > > > > > instance,
> > > > > > > > > > > while we only look at the original AZ.
> > > > > > > > > > > That being said, we could have a weigher that would look at
> > > > > > > > > > whether the
> > > > > > > > > > > host is in the same AZ than the instance.host.
> > > > > > > > > > you miss understood what i wrote
> > > > > > > > > > 
> > > > > > > > > > i suggested addint Destination.current_az to store teh curernt AZ
> > > > > > > > > > of the instance before scheduling.
> > > > > > > > > > 
> > > > > > > > > > so my proposal is if RequestSpec.AZ is not set and
> > > > > > > > > > Destination.current_az is set then the new
> > > > > > > > > > weigher would prefer hosts that are in the same az as
> > > > > > > > > > Destination.current_az
> > > > > > > > > > 
> > > > > > > > > > we coudl also call Destination.current_az Destination.prefered_az
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > I meant, I think we don't need to provide a new field, we can
> > > > > > > > > already know about what host an existing instance uses if we want (using
> > > > > > > > > [1])
> > > > > > > > > Anyway, let's stop to discuss about it here, we should rather
> > > > > > > > > review that for a Launchpad blueprint or more a spec.
> > > > > > > > > 
> > > > > > > > > -Sylvain
> > > > > > > > > 
> > > > > > > > > [1]
> > > > > > > > > https://github.com/openstack/nova/blob/b9a49ffb04cb5ae2d8c439361a3552296df02988/nova/scheduler/host_manager.py#L369-L370
> > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > This will provide soft AZ affinity for the vm and preserve the
> > > > > > > > > > fact that if
> > > > > > > > > > > > a vm is created without sepcifying
> > > > > > > > > > > > An AZ the expectaiton at the api level woudl be that it can
> > > > > > > > > > migrate to any
> > > > > > > > > > > > AZ.
> > > > > > > > > > > > 
> > > > > > > > > > > > To provide hard AZ affintiy we could also add prefileter that
> > > > > > > > > > would use
> > > > > > > > > > > > the same data but instead include it in the
> > > > > > > > > > > > placement query so that only the current AZ is considered.
> > > > > > > > > > This would have
> > > > > > > > > > > > to be disabled by default.
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > Sure, we could create a new prefilter so we could then deprecate
> > > > > > > > > > the
> > > > > > > > > > > AZFilter if we want.
> > > > > > > > > > we already have an AZ prefilter and the AZFilter is deprecate for
> > > > > > > > > > removal
> > > > > > > > > > i ment to delete it in zed but did not have time to do it in zed
> > > > > > > > > > of Antielope
> > > > > > > > > > i deprecated the AZ| filter in
> > > > > > > > > > https://github.com/openstack/nova/commit/7c7a2a142d74a7deeda2a79baf21b689fe32cd08
> > > > > > > > > > xena when i enabeld the az prefilter by default.
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > Ah whoops, indeed I forgot the fact we already have the prefilter,
> > > > > > > > > so the hard support for AZ is already existing.
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > > i will try an delete teh AZ filter before m1 if others dont.
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > OK.
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > > That woudl allow operators to choose the desired behavior.
> > > > > > > > > > > > curret behavior (disable weigher and dont enabel prefilter)
> > > > > > > > > > > > new default, prefer current AZ (weigher enabeld prefilter
> > > > > > > > > > disabled)
> > > > > > > > > > > > hard affintiy(prefilter enabled.)
> > > > > > > > > > > > 
> > > > > > > > > > > > there are other ways to approch this but updating the request
> > > > > > > > > > spec is not
> > > > > > > > > > > > one of them.
> > > > > > > > > > > > we have to maintain the fact the enduser did not request an AZ.
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > Anyway, if folks want to discuss about AZs, this week is the
> > > > > > > > > > good time :-)
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > > -Sylvain
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Sun, Mar 26, 2023 at 8:20 AM Nguyễn Hữu Khôi <
> > > > > > > > > > > > nguyenhuukhoinw at gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Hello guys.
> > > > > > > > > > > > > > > I playing with Nova AZ and Masakari
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 
> > > > > > > > > > https://docs.openstack.org/nova/latest/admin/availability-zones.html
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Masakari will move server by nova scheduler.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Openstack Docs describe that:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > If the server was not created in a specific zone then it
> > > > > > > > > > is free to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > moved to other zones, i.e. the AvailabilityZoneFilter
> > > > > > > > > > > > > > > <
> > > > > > > > > > > > 
> > > > > > > > > > https://docs.openstack.org/nova/latest/admin/scheduling.html#availabilityzonefilter
> > > > > > > > > > > 
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > a no-op.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I see that everyone usually creates instances with "Any
> > > > > > > > > > Availability
> > > > > > > > > > > > > > > Zone" on Horzion and also we don't specify AZ when
> > > > > > > > > > creating
> > > > > > > > > > > > instances by
> > > > > > > > > > > > > > > cli.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > By this way, when we use Masakari or we miragrated
> > > > > > > > > > instances( or
> > > > > > > > > > > > > > > evacuate) so our instance will be moved to other zones.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Can we attach AZ to server create requests API based on
> > > > > > > > > > Any
> > > > > > > > > > > > > > > Availability Zone to limit instances moved to other
> > > > > > > > > > zones?
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Thank you. Regards
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Nguyen Huu Khoi
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Rafael Weingärtner
> > > > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > 
> > > --
> > > Rafael Weingärtner
> > > 
> > 
>