how to update existing vm instances 'availability_zone' field in nova database 'instances' table
Hi, How to update existing vm instances 'availability_zone' field in nova database 'instances' table? It turns out there are many VMs with default 'nova' availability zone, while the hypervisors are with another different target aggregate/availability zone name, thus live migration for these VMs are failing. The strange thing is that the 'OS-EXT-AZ:availability_zone' output from 'openstack server show VM_NAME' is showing these VMs are already with the target availibity_zone name. No idea why we have this inconsistency. Would it be ok to just update database table entry directly to set 'availility_zone' to different value for these VMs? Or there might be better alternatives I am not aware of? Thanks, Hai
It seems this 'AvailabilityZoneFilter' is really buggy. The 'instances' table 'availability_zone' field was already updated to match the target desired zone name, but 'AvailabilityZoneFilter' kept rejecting the request. I already restarted nova-api, nova-conductor, nova-scheduler services after updating this database field value, but that's not helpful. In the source code in file '/usr/lib/python3/dist-packages/nova/scheduler/filters/availability_zone_filter.py', how does 'spec_obj' got its availability_zone value? I don't understand this particular code. This issue is really frustrating .. def host_passes(self, host_state, spec_obj): availability_zone = spec_obj.availability_zone On Wed, Aug 4, 2021 at 5:11 PM hai wu <haiwu.us@gmail.com> wrote:
Hi,
How to update existing vm instances 'availability_zone' field in nova database 'instances' table? It turns out there are many VMs with default 'nova' availability zone, while the hypervisors are with another different target aggregate/availability zone name, thus live migration for these VMs are failing.
The strange thing is that the 'OS-EXT-AZ:availability_zone' output from 'openstack server show VM_NAME' is showing these VMs are already with the target availibity_zone name. No idea why we have this inconsistency.
Would it be ok to just update database table entry directly to set 'availility_zone' to different value for these VMs? Or there might be better alternatives I am not aware of?
Thanks, Hai
Hello, The availability_zone field in the instances object is a read-only field that doesn’t have anything to do with the actual scheduling of instances. This has been covered numerous of times on the mailing list here. The request_specs table in the nova API database contains the specs for when an instance was launched and the JSON blob there (in the spec field) includes the field availability_zone which affects scheduling. Please not that it’s not supported or recommend to update that field and if you do it should be done using oslo_versionedobjects even though updating it directly in the database will make it appear to work but might not be without bugs. Best regards
On 6 Aug 2021, at 21:24, hai wu <haiwu.us@gmail.com> wrote:
It seems this 'AvailabilityZoneFilter' is really buggy. The 'instances' table 'availability_zone' field was already updated to match the target desired zone name, but 'AvailabilityZoneFilter' kept rejecting the request. I already restarted nova-api, nova-conductor, nova-scheduler services after updating this database field value, but that's not helpful.
In the source code in file '/usr/lib/python3/dist-packages/nova/scheduler/filters/availability_zone_filter.py', how does 'spec_obj' got its availability_zone value? I don't understand this particular code. This issue is really frustrating .. def host_passes(self, host_state, spec_obj): availability_zone = spec_obj.availability_zone
On Wed, Aug 4, 2021 at 5:11 PM hai wu <haiwu.us@gmail.com> wrote:
Hi,
How to update existing vm instances 'availability_zone' field in nova database 'instances' table? It turns out there are many VMs with default 'nova' availability zone, while the hypervisors are with another different target aggregate/availability zone name, thus live migration for these VMs are failing.
The strange thing is that the 'OS-EXT-AZ:availability_zone' output from 'openstack server show VM_NAME' is showing these VMs are already with the target availibity_zone name. No idea why we have this inconsistency.
Would it be ok to just update database table entry directly to set 'availility_zone' to different value for these VMs? Or there might be better alternatives I am not aware of?
Thanks, Hai
Thanks Tobias! I was wondering about this 'request_specs' table before, and 'availability_zone' in its blob for those VMs were NULL value IIRC, and I don't know how to update it before. I am not sure how to use oslo_versionedobjects to update it. So basically there would be no way for us to use new 'availability_zone' aggregate name later for hypervisor hosts? I finally worked around this issue by removing hosts from new availability zone name, which by default put those hosts back to `nova` zone, and that ensured the live migration process working again. This is not ideal, but is now working. Thanks, Hai On Sun, Aug 8, 2021 at 3:44 AM Tobias Urdin <tobias.urdin@binero.com> wrote:
Hello,
The availability_zone field in the instances object is a read-only field that doesn’t have anything to do with the actual scheduling of instances. This has been covered numerous of times on the mailing list here.
The request_specs table in the nova API database contains the specs for when an instance was launched and the JSON blob there (in the spec field) includes the field availability_zone which affects scheduling.
Please not that it’s not supported or recommend to update that field and if you do it should be done using oslo_versionedobjects even though updating it directly in the database will make it appear to work but might not be without bugs.
Best regards
On 6 Aug 2021, at 21:24, hai wu <haiwu.us@gmail.com> wrote:
It seems this 'AvailabilityZoneFilter' is really buggy. The 'instances' table 'availability_zone' field was already updated to match the target desired zone name, but 'AvailabilityZoneFilter' kept rejecting the request. I already restarted nova-api, nova-conductor, nova-scheduler services after updating this database field value, but that's not helpful.
In the source code in file '/usr/lib/python3/dist-packages/nova/scheduler/filters/availability_zone_filter.py', how does 'spec_obj' got its availability_zone value? I don't understand this particular code. This issue is really frustrating .. def host_passes(self, host_state, spec_obj): availability_zone = spec_obj.availability_zone
On Wed, Aug 4, 2021 at 5:11 PM hai wu <haiwu.us@gmail.com> wrote:
Hi,
How to update existing vm instances 'availability_zone' field in nova database 'instances' table? It turns out there are many VMs with default 'nova' availability zone, while the hypervisors are with another different target aggregate/availability zone name, thus live migration for these VMs are failing.
The strange thing is that the 'OS-EXT-AZ:availability_zone' output from 'openstack server show VM_NAME' is showing these VMs are already with the target availibity_zone name. No idea why we have this inconsistency.
Would it be ok to just update database table entry directly to set 'availility_zone' to different value for these VMs? Or there might be better alternatives I am not aware of?
Thanks, Hai
I looked further into oslo_versionedobjects. It seems it is some other library, which is not part of openstacksdk. I have no idea on how to use oslo_versionedobjects to update request_spec field for certain VM instance. The only sample code I could find online is this line: versions = ovo_base.obj_tree_get_versions('RequestSpec') Do you have any sample code to show how to use oslo_versionedobjects to update a test VM instance request_spec? On Mon, Aug 9, 2021 at 8:50 AM hai wu <haiwu.us@gmail.com> wrote:
Thanks Tobias! I was wondering about this 'request_specs' table before, and 'availability_zone' in its blob for those VMs were NULL value IIRC, and I don't know how to update it before. I am not sure how to use oslo_versionedobjects to update it.
So basically there would be no way for us to use new 'availability_zone' aggregate name later for hypervisor hosts?
I finally worked around this issue by removing hosts from new availability zone name, which by default put those hosts back to `nova` zone, and that ensured the live migration process working again. This is not ideal, but is now working.
Thanks, Hai
On Sun, Aug 8, 2021 at 3:44 AM Tobias Urdin <tobias.urdin@binero.com> wrote:
Hello,
The availability_zone field in the instances object is a read-only field that doesn’t have anything to do with the actual scheduling of instances. This has been covered numerous of times on the mailing list here.
The request_specs table in the nova API database contains the specs for when an instance was launched and the JSON blob there (in the spec field) includes the field availability_zone which affects scheduling.
Please not that it’s not supported or recommend to update that field and if you do it should be done using oslo_versionedobjects even though updating it directly in the database will make it appear to work but might not be without bugs.
Best regards
On 6 Aug 2021, at 21:24, hai wu <haiwu.us@gmail.com> wrote:
It seems this 'AvailabilityZoneFilter' is really buggy. The 'instances' table 'availability_zone' field was already updated to match the target desired zone name, but 'AvailabilityZoneFilter' kept rejecting the request. I already restarted nova-api, nova-conductor, nova-scheduler services after updating this database field value, but that's not helpful.
In the source code in file '/usr/lib/python3/dist-packages/nova/scheduler/filters/availability_zone_filter.py', how does 'spec_obj' got its availability_zone value? I don't understand this particular code. This issue is really frustrating .. def host_passes(self, host_state, spec_obj): availability_zone = spec_obj.availability_zone
On Wed, Aug 4, 2021 at 5:11 PM hai wu <haiwu.us@gmail.com> wrote:
Hi,
How to update existing vm instances 'availability_zone' field in nova database 'instances' table? It turns out there are many VMs with default 'nova' availability zone, while the hypervisors are with another different target aggregate/availability zone name, thus live migration for these VMs are failing.
The strange thing is that the 'OS-EXT-AZ:availability_zone' output from 'openstack server show VM_NAME' is showing these VMs are already with the target availibity_zone name. No idea why we have this inconsistency.
Would it be ok to just update database table entry directly to set 'availility_zone' to different value for these VMs? Or there might be better alternatives I am not aware of?
Thanks, Hai
participants (2)
-
hai wu
-
Tobias Urdin