Open Stack

Bug report created at: https://bugs.launchpad.net/nova/+bug/2018318

On Wed, Apr 26, 2023 at 12:23 PM Sylvain Bauza <sbauza at redhat.com> wrote:

>
>
> Le mer. 26 avr. 2023 à 14:46, Rafael Weingärtner <
> rafaelweingartner at gmail.com> a écrit :
>
>> Adding the response to this thread, as I replied in the wrong one. Sorry
>> for the confusion.
>>
>> Hello Sylvain Bauza and Sean Mooney,
>>
>> The patch pointed out in [1] is doing exactly what you said that we
>> should not do. I mean, changing the AZ in the request spec of the virtual
>> machine (VM) that the user created. The patch we propose in [2] is intended
>> to avoid exactly that.
>>
>> As pointed out by Florian, the configuration "cross_az_attach" is a
>> constraint of the cloud environment, and as such it should be considered
>> when selecting hosts to execute VMs migrations. Therefore, for those
>> situations, and only for those (cross_az_attach=False), we send the current
>> AZ of the VM to placement, to enable it (placement) to filter out hosts
>> that are not from the current AZ of the VM.
>>
>> Looking at the patch suggested by you in [1], and later testing it, we
>> can confirm that the problem is still there in main/upstream. This happens
>> because the patch in [1] is only addressing the cases when the VM is
>> created based on volumes, and then it sets the AZ of the volumes in the
>> request spec of the VM. That is why everything works for the setups where
>> cross_az_attach=False. However, if we create a VM based on an image, and
>> then it (Nova) creates a new volume in Cinder, the AZ is not set in the
>> request spec (but it is used to execute the first call to placement to
>> select the hosts); thus, the issues described in [2] can still happen.
>>
>> Anyways, the proposal presented in [2] is simpler and works nicely. We
>> can discuss it further in the patchset then, if you guys think it is worth
>> it.
>>
>>
> As I replied in the gerrit change
> https://review.opendev.org/c/openstack/nova/+/864760/comments/4a302ce3_9805e7c6
> then you should create a Launchpad bug report but fwiw, you should also
> modify the implementation as it would rather do the same for image metadata
> that what we do for volumes with [1]
>
> -Sylvain
>
> [1]
>> https://review.opendev.org/c/openstack/nova/+/469675/12/nova/compute/api.py#1173
>> [2] https://review.opendev.org/c/openstack/nova/+/864760
>>
>> On Wed, Mar 29, 2023 at 10:26 AM Nguyễn Hữu Khôi <
>> nguyenhuukhoinw at gmail.com> wrote:
>>
>>> "If they *don't* provide this parameter, then depending on the
>>> default_schedule_zone config option, either the instance will eventually
>>> use a specific AZ (and then it's like if the enduser was asking for this
>>> AZ), or none of AZ is requested and then the instance can be created and
>>> moved between any hosts within *all* AZs."
>>>
>>> I ask aftet that, although without az when launch instances but they
>>> still have az. But i still mv to diffent host in diffent az when mirgrating
>>> or spawn which masakari. i am not clear, I tested.
>>>
>>>
>>> On Wed, Mar 29, 2023, 7:38 PM Nguyễn Hữu Khôi <nguyenhuukhoinw at gmail.com>
>>> wrote:
>>>
>>>> Yes. Thanks, but the things I would like to know: after instances are
>>>> created, how do we know if it was launched with specified AZ or without it?
>>>> I mean the way to distinguish between specified instances and non specified
>>>> instances?
>>>>
>>>> Nguyen Huu Khoi
>>>>
>>>>
>>>> On Wed, Mar 29, 2023 at 5:05 PM Sylvain Bauza <sbauza at redhat.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> Le mer. 29 mars 2023 à 08:06, Nguyễn Hữu Khôi <
>>>>> nguyenhuukhoinw at gmail.com> a écrit :
>>>>>
>>>>>> Hello.
>>>>>> I have one question.
>>>>>> Follow this
>>>>>>
>>>>>> https://docs.openstack.org/nova/latest/admin/availability-zones.html
>>>>>>
>>>>>> If the server was not created in a specific zone then it is free to
>>>>>> be moved to other zones. but when I use
>>>>>>
>>>>>> openstack server show [server id]
>>>>>>
>>>>>> I still see the "OS-EXT-AZ:availability_zone" value belonging to my
>>>>>> instance.
>>>>>>
>>>>>>
>>>>> Correct, this is normal. If the operators creates some AZs, then the
>>>>> enduser should see where the instance in which AZ.
>>>>>
>>>>>
>>>>>> Could you tell the difference which causes "if the server was not
>>>>>> created in a specific zone then it is free to be moved to other zones.
>>>>>> "
>>>>>>
>>>>>>
>>>>> To be clear, an operator can create Availability Zones. Those AZs can
>>>>> then be seen by an enduser using the os-availability-zones API [1]. Then,
>>>>> either the enduser wants to use a specific AZ for their next instance
>>>>> creation (and if so, he/she adds --availability-zone parameter to their
>>>>> instance creation client) or they don't want and then they don't provide
>>>>> this parameter.
>>>>>
>>>>> If they provide this parameter, then the server will be created only
>>>>> in one host in the specific AZ and then when moving the instance later, it
>>>>> will continue to move to any host within the same AZ.
>>>>> If they *don't* provide this parameter, then depending on the
>>>>> default_schedule_zone config option, either the instance will eventually
>>>>> use a specific AZ (and then it's like if the enduser was asking for this
>>>>> AZ), or none of AZ is requested and then the instance can be created and
>>>>> moved between any hosts within *all* AZs.
>>>>>
>>>>> That being said, as I said earlier, the enduser can still verify the
>>>>> AZ from where the instance is by the server show parameter you told.
>>>>>
>>>>> We also have a documentation explaining about Availability Zones,
>>>>> maybe this would help you more to understand about AZs :
>>>>> https://docs.openstack.org/nova/latest/admin/availability-zones.html
>>>>>
>>>>>
>>>>> [1]
>>>>> https://docs.openstack.org/api-ref/compute/#availability-zones-os-availability-zone
>>>>> (tbc, the enduser won't see the hosts, but they can see the list of
>>>>> existing AZs)
>>>>>
>>>>>
>>>>>
>>>>>> Nguyen Huu Khoi
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 27, 2023 at 8:37 PM Nguyễn Hữu Khôi <
>>>>>> nguyenhuukhoinw at gmail.com> wrote:
>>>>>>
>>>>>>> Hello guys.
>>>>>>>
>>>>>>> I just suggest to openstack nova works better. My story because
>>>>>>>
>>>>>>>
>>>>>>>    1.
>>>>>>>
>>>>>>>    The server was created in a specific zone with the POST /servers request
>>>>>>>    containing the availability_zone parameter.
>>>>>>>
>>>>>>> It will be nice when we attach randow zone when we create instances
>>>>>>> then It will only move to the same zone when migrating or masakari ha.
>>>>>>>
>>>>>>> Currently we can force it to zone by default zone shedule in
>>>>>>> nova.conf.
>>>>>>>
>>>>>>> Sorry because I am new to Openstack and I am just an operator. I try
>>>>>>> to verify some real cases.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Nguyen Huu Khoi
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 27, 2023 at 7:43 PM Sylvain Bauza <sbauza at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le lun. 27 mars 2023 à 14:28, Sean Mooney <smooney at redhat.com> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> On Mon, 2023-03-27 at 14:06 +0200, Sylvain Bauza wrote:
>>>>>>>>> > Le lun. 27 mars 2023 à 13:51, Sean Mooney <smooney at redhat.com>
>>>>>>>>> a écrit :
>>>>>>>>> >
>>>>>>>>> > > On Mon, 2023-03-27 at 10:19 +0200, Sylvain Bauza wrote:
>>>>>>>>> > > > Le dim. 26 mars 2023 à 14:30, Rafael Weingärtner <
>>>>>>>>> > > > rafaelweingartner at gmail.com> a écrit :
>>>>>>>>> > > >
>>>>>>>>> > > > > Hello Nguyễn Hũu Khôi,
>>>>>>>>> > > > > You might want to take a look at:
>>>>>>>>> > > > > https://review.opendev.org/c/openstack/nova/+/864760. We
>>>>>>>>> created a
>>>>>>>>> > > patch
>>>>>>>>> > > > > to avoid migrating VMs to any AZ, once the VM has been
>>>>>>>>> bootstrapped in
>>>>>>>>> > > an
>>>>>>>>> > > > > AZ that has cross zone attache equals to false.
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > Well, I'll provide some comments in the change, but I'm
>>>>>>>>> afraid we can't
>>>>>>>>> > > > just modify the request spec like you would want.
>>>>>>>>> > > >
>>>>>>>>> > > > Anyway, if you want to discuss about it in the vPTG, just
>>>>>>>>> add it in the
>>>>>>>>> > > > etherpad and add your IRC nick so we could try to find a
>>>>>>>>> time where we
>>>>>>>>> > > > could be discussing it :
>>>>>>>>> https://etherpad.opendev.org/p/nova-bobcat-ptg
>>>>>>>>> > > > Also, this kind of behaviour modification is more a new
>>>>>>>>> feature than a
>>>>>>>>> > > > bugfix, so fwiw you should create a launchpad blueprint so
>>>>>>>>> we could
>>>>>>>>> > > better
>>>>>>>>> > > > see it.
>>>>>>>>> > >
>>>>>>>>> > > i tought i left review feedback on that too that the approch
>>>>>>>>> was not
>>>>>>>>> > > correct.
>>>>>>>>> > > i guess i did not in the end.
>>>>>>>>> > >
>>>>>>>>> > > modifying the request spec as sylvain menthioned is not
>>>>>>>>> correct.
>>>>>>>>> > > i disucssed this topic on irc a few weeks back with mohomad
>>>>>>>>> for vxhost.
>>>>>>>>> > > what can be done is as follows.
>>>>>>>>> > >
>>>>>>>>> > > we can add a current_az field  to the Destination object
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L1092-L1122
>>>>>>>>> > > The conductor can read the instance.AZ and populate it in that
>>>>>>>>> new field.
>>>>>>>>> > > We can then add a new weigher to prefer hosts that are in the
>>>>>>>>> same az.
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> >
>>>>>>>>> > I tend to disagree this approach as people would think that the
>>>>>>>>> > Destination.az field would be related to the current AZ for an
>>>>>>>>> instance,
>>>>>>>>> > while we only look at the original AZ.
>>>>>>>>> > That being said, we could have a weigher that would look at
>>>>>>>>> whether the
>>>>>>>>> > host is in the same AZ than the instance.host.
>>>>>>>>> you miss understood what i wrote
>>>>>>>>>
>>>>>>>>> i suggested addint Destination.current_az to store teh curernt AZ
>>>>>>>>> of the instance before scheduling.
>>>>>>>>>
>>>>>>>>> so my proposal is if RequestSpec.AZ is not set and
>>>>>>>>> Destination.current_az is set then the new
>>>>>>>>> weigher would prefer hosts that are in the same az as
>>>>>>>>> Destination.current_az
>>>>>>>>>
>>>>>>>>> we coudl also call Destination.current_az Destination.prefered_az
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I meant, I think we don't need to provide a new field, we can
>>>>>>>> already know about what host an existing instance uses if we want (using
>>>>>>>> [1])
>>>>>>>> Anyway, let's stop to discuss about it here, we should rather
>>>>>>>> review that for a Launchpad blueprint or more a spec.
>>>>>>>>
>>>>>>>> -Sylvain
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/openstack/nova/blob/b9a49ffb04cb5ae2d8c439361a3552296df02988/nova/scheduler/host_manager.py#L369-L370
>>>>>>>>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > This will provide soft AZ affinity for the vm and preserve the
>>>>>>>>> fact that if
>>>>>>>>> > > a vm is created without sepcifying
>>>>>>>>> > > An AZ the expectaiton at the api level woudl be that it can
>>>>>>>>> migrate to any
>>>>>>>>> > > AZ.
>>>>>>>>> > >
>>>>>>>>> > > To provide hard AZ affintiy we could also add prefileter that
>>>>>>>>> would use
>>>>>>>>> > > the same data but instead include it in the
>>>>>>>>> > > placement query so that only the current AZ is considered.
>>>>>>>>> This would have
>>>>>>>>> > > to be disabled by default.
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > Sure, we could create a new prefilter so we could then deprecate
>>>>>>>>> the
>>>>>>>>> > AZFilter if we want.
>>>>>>>>> we already have an AZ prefilter and the AZFilter is deprecate for
>>>>>>>>> removal
>>>>>>>>> i ment to delete it in zed but did not have time to do it in zed
>>>>>>>>> of Antielope
>>>>>>>>> i deprecated the AZ| filter in
>>>>>>>>> https://github.com/openstack/nova/commit/7c7a2a142d74a7deeda2a79baf21b689fe32cd08
>>>>>>>>> xena when i enabeld the az prefilter by default.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Ah whoops, indeed I forgot the fact we already have the prefilter,
>>>>>>>> so the hard support for AZ is already existing.
>>>>>>>>
>>>>>>>>
>>>>>>>>> i will try an delete teh AZ filter before m1 if others dont.
>>>>>>>>>
>>>>>>>>
>>>>>>>> OK.
>>>>>>>>
>>>>>>>>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > > That woudl allow operators to choose the desired behavior.
>>>>>>>>> > > curret behavior (disable weigher and dont enabel prefilter)
>>>>>>>>> > > new default, prefer current AZ (weigher enabeld prefilter
>>>>>>>>> disabled)
>>>>>>>>> > > hard affintiy(prefilter enabled.)
>>>>>>>>> > >
>>>>>>>>> > > there are other ways to approch this but updating the request
>>>>>>>>> spec is not
>>>>>>>>> > > one of them.
>>>>>>>>> > > we have to maintain the fact the enduser did not request an AZ.
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > Anyway, if folks want to discuss about AZs, this week is the
>>>>>>>>> good time :-)
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > > >
>>>>>>>>> > > > -Sylvain
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > > On Sun, Mar 26, 2023 at 8:20 AM Nguyễn Hữu Khôi <
>>>>>>>>> > > nguyenhuukhoinw at gmail.com>
>>>>>>>>> > > > > wrote:
>>>>>>>>> > > > >
>>>>>>>>> > > > > > Hello guys.
>>>>>>>>> > > > > > I playing with Nova AZ and Masakari
>>>>>>>>> > > > > >
>>>>>>>>> > > > > >
>>>>>>>>> https://docs.openstack.org/nova/latest/admin/availability-zones.html
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Masakari will move server by nova scheduler.
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Openstack Docs describe that:
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > If the server was not created in a specific zone then it
>>>>>>>>> is free to
>>>>>>>>> > > be
>>>>>>>>> > > > > > moved to other zones, i.e. the AvailabilityZoneFilter
>>>>>>>>> > > > > > <
>>>>>>>>> > >
>>>>>>>>> https://docs.openstack.org/nova/latest/admin/scheduling.html#availabilityzonefilter
>>>>>>>>> >
>>>>>>>>> > > is
>>>>>>>>> > > > > > a no-op.
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > I see that everyone usually creates instances with "Any
>>>>>>>>> Availability
>>>>>>>>> > > > > > Zone" on Horzion and also we don't specify AZ when
>>>>>>>>> creating
>>>>>>>>> > > instances by
>>>>>>>>> > > > > > cli.
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > By this way, when we use Masakari or we miragrated
>>>>>>>>> instances( or
>>>>>>>>> > > > > > evacuate) so our instance will be moved to other zones.
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Can we attach AZ to server create requests API based on
>>>>>>>>> Any
>>>>>>>>> > > > > > Availability Zone to limit instances moved to other
>>>>>>>>> zones?
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Thank you. Regards
>>>>>>>>> > > > > >
>>>>>>>>> > > > > > Nguyen Huu Khoi
>>>>>>>>> > > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > >
>>>>>>>>> > > > > --
>>>>>>>>> > > > > Rafael Weingärtner
>>>>>>>>> > > > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>>
>>>>>>>>>
>>
>> --
>> Rafael Weingärtner
>>
>

-- 
Rafael Weingärtner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230502/2faa63c5/attachment-0001.htm>