New nova compute nodes not correctly adding into service
Hi All, I was wondering if anyone could point me in the right direction with an issue we are having adding in new compute nodes to our platform. We have been trying to add a few new compute nodes to increase our capacity and upon configuring them with OSA and them appearing to be in service we have noticed that no new instances are provisioning onto them. We did a test by disabling all compute nodes except for the new ones and an instance launch failed with "No valid compute hosts were found" We have enabled debug mode on Nova and also looked at the nova-api containers ( scheduler, compute etc ) and it appears there is not even an attempt to try and launch an instance on the new hosts from what we can see. The only thing we have noticed on the newly added hosts is that the "privsep-helper" doesn't appear to be started whereas on a working node it is running. We have not changed the software versions of OpenStack or OSA, we are running Queens. The newly added compute node is reporting itself back fine from what we can see: Total usable vcpus: 72, total allocated vcpus: 0 _report_final_resource_view /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/nova/compute/resource_tracker.py:827 2020-12-24 16:26:32.895 6491 INFO nova.compute.resource_tracker [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Final resource view: name=compute-27.openstack.local phys_ram=257826MB used_ram=2048MB phys_disk=30166GB used_disk=0GB total_vcpus=72 used_vcpus=0 pci_stats=[] 2020-12-24 16:26:33.022 6491 DEBUG nova.compute.resource_tracker [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Compute_service record updated for compute-27:compute-27.openstack.local _update_available_resource /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/nova/compute/resource_tracker.py:767 2020-12-24 16:26:33.023 6491 DEBUG oslo_concurrency.lockutils [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.334s inner /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 2020-12-24 16:26:35.001 6491 DEBUG oslo_service.periodic_task [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 Does anyone know of any other logs we can check / enable that might lead us in the right direction to figure out why the new hosts are not adding in? Any help would be greatly appreciated as we are a bit stumped at the moment. Many thanks, -- Grant Morley Site Reliability Engineer, Civo Ltd Unit H-K, Gateway 1000, Whittle Way Stevenage, Herts, SG1 2FP, UK Visit us at www.civo.com <https://www.civo.com/> Signup for an account now > <https://www.civo.com/signup>
Hi, Check the nova scheduler log. You will see what filter is failing. That's a good starting point. Maybe you have an aggregate group and you need to add the new compute node into the group. On Thu, 24 Dec 2020 at 18:38 Grant Morley <grant@civo.com> wrote:
Hi All,
I was wondering if anyone could point me in the right direction with an issue we are having adding in new compute nodes to our platform. We have been trying to add a few new compute nodes to increase our capacity and upon configuring them with OSA and them appearing to be in service we have noticed that no new instances are provisioning onto them. We did a test by disabling all compute nodes except for the new ones and an instance launch failed with "No valid compute hosts were found"
We have enabled debug mode on Nova and also looked at the nova-api containers ( scheduler, compute etc ) and it appears there is not even an attempt to try and launch an instance on the new hosts from what we can see.
The only thing we have noticed on the newly added hosts is that the "privsep-helper" doesn't appear to be started whereas on a working node it is running.
We have not changed the software versions of OpenStack or OSA, we are running Queens.
The newly added compute node is reporting itself back fine from what we can see:
Total usable vcpus: 72, total allocated vcpus: 0 _report_final_resource_view /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/nova/compute/resource_tracker.py:827 2020-12-24 16:26:32.895 6491 INFO nova.compute.resource_tracker [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Final resource view: name=compute-27.openstack.local phys_ram=257826MB used_ram=2048MB phys_disk=30166GB used_disk=0GB total_vcpus=72 used_vcpus=0 pci_stats=[] 2020-12-24 16:26:33.022 6491 DEBUG nova.compute.resource_tracker [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Compute_service record updated for compute-27:compute-27.openstack.local _update_available_resource /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/nova/compute/resource_tracker.py:767 2020-12-24 16:26:33.023 6491 DEBUG oslo_concurrency.lockutils [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.334s inner /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 2020-12-24 16:26:35.001 6491 DEBUG oslo_service.periodic_task [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
Does anyone know of any other logs we can check / enable that might lead us in the right direction to figure out why the new hosts are not adding in?
Any help would be greatly appreciated as we are a bit stumped at the moment.
Many thanks,
-- Grant Morley Site Reliability Engineer, Civo Ltd Unit H-K, Gateway 1000, Whittle Way Stevenage, Herts, SG1 2FP, UK Visit us at www.civo.com Signup for an account now > <https://www.civo.com/signup>
-- Ionut Biru - https://fleio.com
The nova.filter log entry for the nova-scheduler should tell you why computes are being filtered out. You can also check the flavors you are using to see if it's targeting a host-aggregate. On Thu, Dec 24, 2020 at 11:40 AM Grant Morley <grant@civo.com> wrote:
Hi All,
I was wondering if anyone could point me in the right direction with an issue we are having adding in new compute nodes to our platform. We have been trying to add a few new compute nodes to increase our capacity and upon configuring them with OSA and them appearing to be in service we have noticed that no new instances are provisioning onto them. We did a test by disabling all compute nodes except for the new ones and an instance launch failed with "No valid compute hosts were found"
We have enabled debug mode on Nova and also looked at the nova-api containers ( scheduler, compute etc ) and it appears there is not even an attempt to try and launch an instance on the new hosts from what we can see.
The only thing we have noticed on the newly added hosts is that the "privsep-helper" doesn't appear to be started whereas on a working node it is running.
We have not changed the software versions of OpenStack or OSA, we are running Queens.
The newly added compute node is reporting itself back fine from what we can see:
Total usable vcpus: 72, total allocated vcpus: 0 _report_final_resource_view /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/nova/compute/resource_tracker.py:827 2020-12-24 16:26:32.895 6491 INFO nova.compute.resource_tracker [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Final resource view: name=compute-27.openstack.local phys_ram=257826MB used_ram=2048MB phys_disk=30166GB used_disk=0GB total_vcpus=72 used_vcpus=0 pci_stats=[] 2020-12-24 16:26:33.022 6491 DEBUG nova.compute.resource_tracker [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Compute_service record updated for compute-27:compute-27.openstack.local _update_available_resource /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/nova/compute/resource_tracker.py:767 2020-12-24 16:26:33.023 6491 DEBUG oslo_concurrency.lockutils [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker._update_available_resource" :: held 0.334s inner /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285 2020-12-24 16:26:35.001 6491 DEBUG oslo_service.periodic_task [req-6c58a97e-b7cb-4987-a219-da5b1ce156f8 - - - - -] Running periodic task ComputeManager._poll_rebooting_instances run_periodic_tasks /openstack/venvs/nova-17.1.2/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
Does anyone know of any other logs we can check / enable that might lead us in the right direction to figure out why the new hosts are not adding in?
Any help would be greatly appreciated as we are a bit stumped at the moment.
Many thanks, -- Grant Morley Site Reliability Engineer, Civo Ltd Unit H-K, Gateway 1000, Whittle Way Stevenage, Herts, SG1 2FP, UK Visit us at www.civo.com Signup for an account now > <https://www.civo.com/signup>
participants (3)
-
Grant Morley
-
Ionut Biru
-
Laurent Dumont