Re: [Openstack-operators] Fwd: Nova hypervisor uuid
Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me. I am sure kvm nodes names are note changed. Tables where uuid are duplicated are: dataresource_providers in nova_api db compute_nodes in nova db Regards Ignazio Il 28/Nov/2018 11:09 AM, "Gianpiero Ardissono" <gp.ardissono@gmail.com> ha scritto:
---------- Forwarded message --------- From: Matt Riedemann <mriedemos@gmail.com> Date: mar 27 nov 2018, 19:03 Subject: Re: [Openstack-operators] Nova hypervisor uuid To: <openstack-operators@lists.openstack.org>
On 11/27/2018 11:32 AM, Ignazio Cassano wrote:
Hi All, Please anyone know where hypervisor uuid is retrived? Sometime updating kmv nodes with yum update it changes and in nova database 2 uuids are assigned to the same node. regards Ignazio
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
To be clear, do you mean the computes_nodes.uuid column value in the cell database? Which is also used for the GET /os-hypervisors response 'id' value if using microversion >= 2.53. If so, that is generated randomly* when the compute_nodes table record is created:
https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d60...
https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d60...
When you hit this problem, are you sure the hostname on the compute host is not changing? Because when nova-compute starts up, it should look for the existing compute node record by host name and node name, which for the libvirt driver should be the same. That lookup code is here:
https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d60...
So the only way nova-compute should create a new compute_nodes table record for the same host is if the host/node name changes during the upgrade. Is the deleted value in the database the same (0) for both of those records?
* The exception to this is for the ironic driver which re-uses the ironic node uuid as of this change: https://review.openstack.org/#/c/571535/
--
Thanks,
Matt
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
On 11/28/2018 4:19 AM, Ignazio Cassano wrote:
Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me. I am sure kvm nodes names are note changed. Tables where uuid are duplicated are: dataresource_providers in nova_api db compute_nodes in nova db Regards Ignazio
It would be easier if you simply dumped the result of a select query on the compute_nodes table where the duplicate nodes exist (you said duplicate UUIDs but I think you mean duplicate host/node names with different UUIDs, correct?). There is a unique constraint on host/hypervisor_hostname (nodename)/deleted: schema.UniqueConstraint( 'host', 'hypervisor_hostname', 'deleted', name="uniq_compute_nodes0host0hypervisor_hostname0deleted"), So I'm wondering if the deleted field is not 0 on one of those because if one is marked as deleted, then the compute service will create a new compute_nodes table record on startup (and associated resource provider). -- Thanks, Matt _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Hello Mattm Yes I mean sometimes I have same host/node names with different uuid in compute_nodes table in nova database I must delete nodes with uuid those not match with nova-hypervisor list command. At this time I have the following: MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes; +---------------------+--------------------------------------+---------+ | hypervisor_hostname | uuid | deleted | +---------------------+--------------------------------------+---------+ | tst2-kvm02 | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d | 0 | | tst2-kvm01 | ce27803b-06cd-44a7-b927-1fa42c813b0f | 0 | +---------------------+--------------------------------------+---------+ 2 rows in set (0,00 sec) But sometimes old uuid are inserted in the table . I deleted again them. I restarted kvm nodes and now the table is ok. I also restarded each controller and the tables is ok. I do not know because 3 days ago I had same compute nodes names with different uuids. Thanks and Regards Ignazio Il giorno mer 28 nov 2018 alle ore 17:54 Matt Riedemann <mriedemos@gmail.com> ha scritto:
On 11/28/2018 4:19 AM, Ignazio Cassano wrote:
Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me. I am sure kvm nodes names are note changed. Tables where uuid are duplicated are: dataresource_providers in nova_api db compute_nodes in nova db Regards Ignazio
It would be easier if you simply dumped the result of a select query on the compute_nodes table where the duplicate nodes exist (you said duplicate UUIDs but I think you mean duplicate host/node names with different UUIDs, correct?).
There is a unique constraint on host/hypervisor_hostname (nodename)/deleted:
schema.UniqueConstraint( 'host', 'hypervisor_hostname', 'deleted', name="uniq_compute_nodes0host0hypervisor_hostname0deleted"),
So I'm wondering if the deleted field is not 0 on one of those because if one is marked as deleted, then the compute service will create a new compute_nodes table record on startup (and associated resource provider).
--
Thanks,
Matt
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
On 11/29/2018 12:49 AM, Ignazio Cassano wrote:
Hello Mattm Yes I mean sometimes I have same host/node names with different uuid in compute_nodes table in nova database I must delete nodes with uuid those not match with nova-hypervisor list command. At this time I have the following: MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes; +---------------------+--------------------------------------+---------+ | hypervisor_hostname | uuid | deleted | +---------------------+--------------------------------------+---------+ | tst2-kvm02 | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d | 0 | | tst2-kvm01 | ce27803b-06cd-44a7-b927-1fa42c813b0f | 0 | +---------------------+--------------------------------------+---------+ 2 rows in set (0,00 sec)
But sometimes old uuid are inserted in the table . I deleted again them. I restarted kvm nodes and now the table is ok. I also restarded each controller and the tables is ok. I do not know because 3 days ago I had same compute nodes names with different uuids.
Thanks and Regards Ignazio
OK I guess if it happens again, please get the host/hypervisor_hostname/uuid/deleted values from the compute_nodes table before you cleanup any entries. Also, when you're deleting the resources from the DB, are you doing it in the DB directly or via the DELETE /os-services/{service_id} API? Because the latter cleans up other related resources to the nova-compute service (the services table record, the compute_nodes table record, the related resource_providers table record in placement, and the host_mappings table record in the nova API DB). The resource provider/host mappings cleanup when deleting a compute service is a more recent bug fix though which depending on your release you might not have: https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80 -- Thanks, Matt
Hi Matt, I did in the DB directly. I am using queens now. Any python client command to delete hold records or I must use api ? Thanks & Regards Ignazio Il giorno gio 29 nov 2018 alle ore 16:28 Matt Riedemann <mriedemos@gmail.com> ha scritto:
On 11/29/2018 12:49 AM, Ignazio Cassano wrote:
Hello Mattm Yes I mean sometimes I have same host/node names with different uuid in compute_nodes table in nova database I must delete nodes with uuid those not match with nova-hypervisor list command. At this time I have the following: MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes; +---------------------+--------------------------------------+---------+ | hypervisor_hostname | uuid | deleted | +---------------------+--------------------------------------+---------+ | tst2-kvm02 | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d | 0 | | tst2-kvm01 | ce27803b-06cd-44a7-b927-1fa42c813b0f | 0 | +---------------------+--------------------------------------+---------+ 2 rows in set (0,00 sec)
But sometimes old uuid are inserted in the table . I deleted again them. I restarted kvm nodes and now the table is ok. I also restarded each controller and the tables is ok. I do not know because 3 days ago I had same compute nodes names with different uuids.
Thanks and Regards Ignazio
OK I guess if it happens again, please get the host/hypervisor_hostname/uuid/deleted values from the compute_nodes table before you cleanup any entries.
Also, when you're deleting the resources from the DB, are you doing it in the DB directly or via the DELETE /os-services/{service_id} API? Because the latter cleans up other related resources to the nova-compute service (the services table record, the compute_nodes table record, the related resource_providers table record in placement, and the host_mappings table record in the nova API DB). The resource provider/host mappings cleanup when deleting a compute service is a more recent bug fix though which depending on your release you might not have:
https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80
--
Thanks,
Matt
On 11/29/2018 10:27 AM, Ignazio Cassano wrote:
I did in the DB directly. I am using queens now. Any python client command to delete hold records or I must use api ?
You can use the CLI: https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-servi... https://docs.openstack.org/python-openstackclient/latest/cli/command-objects... -- Thanks, Matt
Many thks Matt. If the issue will happen again I hope openstack commands will show duplicate entries and let me clean them. When happened nova-hypervisor list command did not show duplcated entries but I saw them only in the database. Regards Ignazio Il giorno Gio 29 Nov 2018 17:28 Matt Riedemann <mriedemos@gmail.com> ha scritto:
On 11/29/2018 10:27 AM, Ignazio Cassano wrote:
I did in the DB directly. I am using queens now. Any python client command to delete hold records or I must use api ?
You can use the CLI:
https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-servi...
https://docs.openstack.org/python-openstackclient/latest/cli/command-objects...
--
Thanks,
Matt
participants (2)
-
Ignazio Cassano
-
Matt Riedemann