[openstack-dev] [nova] Interesting bug when unshelving an instance in an AZ and the AZ is gone

Jay Pipes jaypipes at gmail.com
Sun Oct 22 12:02:26 UTC 2017

On 10/16/2017 11:22 AM, Matt Riedemann wrote:
> This is interesting from the user point of view:
> https://bugs.launchpad.net/nova/+bug/1723880
> - The user creates an instance in a non-default AZ.
> - They shelve offload the instance.
> - The admin deletes the AZ that the instance was using, for whatever 
> reason.
> - The user unshelves the instance which goes back through scheduling and 
> fails with NoValidHost because the AZ on the original request spec no 
> longer exists.
> Now the question is what, if anything, do we do about this bug? Some notes:
> 1. How reasonable is it for a user to expect in a stable production 
> environment that AZs are going to be deleted from under them? We 
> actually have a spec related to this but with AZ renames:
> https://review.openstack.org/#/c/446446/

I don't think it's reasonable for a user to expect an AZ suddenly gets 
*deleted* from under them, no.

That said, I think it's reasonable for operators to want to *rename* an 
AZ. And because AZs in Nova aren't really *things* [1], attempting to 
change the name of an AZ involves a bunch of nasty DB updates (including 
shadow tables). [2]

> 2. Should we null out the instance.availability_zone when it's shelved 
> offloaded like we do for the instance.host and instance.node attributes? 
> Similarly, we would not take into account the 
> RequestSpec.availability_zone when scheduling during unshelve. I tend to 
> prefer this option because once you unshelve offload an instance, it's 
> no longer associated with a host and therefore no longer associated with 
> an AZ. However, is it reasonable to assume that the user doesn't care 
> that the instance, once unshelved, is no longer in the originally 
> requested AZ? Probably not a safe assumption.

Yeah, I don't think this is appropriate.

> 3. When a user unshelves, they can't propose a new AZ (and I don't think 
> we want to add that capability to the unshelve API). So if the original 
> AZ is gone, should we automatically remove the 
> RequestSpec.availability_zone when scheduling? I tend to not like this 
> as it's very implicit and the user could see the AZ on their instance 
> change before and after unshelve and be confused.

I don't think this is something we should add to the public API (for 
reasons Matt stated in a followup email to Dean). Instead, I think the 
"rename AZ" functionality should do the needful DB-related tasks to 
change the instance.availability_zone for shelved instances to the new 
AZ name...

> 4. We could simply do nothing about this specific bug and assert the 
> behavior is correct. The user requested an instance in a specific AZ, 
> shelved that instance and when they wanted to unshelve it, it's no 
> longer available so it fails. The user would have to delete the instance 
> and create a new instance from the shelve snapshot image in a new AZ. If 
> we implemented Sylvain's spec in #1 above, maybe we don't have this 
> problem going forward since you couldn't remove/delete an AZ when there 
> are even shelved offloaded instances still tied to it.

I think it's reasonable to prevent deletion of an AZ (whatever that 
actually means... see [1]) when the AZ "has instances in it" (whatever 
that means... see [1])


> Other options?

[1] AZs in Nova are just metadata key/values on aggregates and string 
values in the instance.availability_zone DB table field that have no FK 
relationship to said metadata key/values

[2] Note that, as I've said before, the entire concept of an 
availability zone in Nova/Cinder/Neutron is completely fictional and 
improperly pretending to be an AWS EC2 availability zone. AZs in Nova 
pretend to be failure domains. They are not anything of the sort.

More information about the OpenStack-dev mailing list