<div dir="auto">Restart the libvirtd. It should sort out<div dir="auto"><br></div><div dir="auto"><br><div dir="auto"><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 23 Aug 2023, 02:32 Thomas Goirand, <<a href="mailto:thomas@goirand.fr">thomas@goirand.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 8/22/23 17:01, Gk Gk wrote:<br>
> Hi All,<br>
> <br>
> We have an openstack ansible stein version setup. We had a vm on it <br>
> which we deleted. The associated neutron port was also deleted and the <br>
> vm uuid is gone from the nova as well. It had a provider network ip. <br>
> Even after deleting the vm, we are still finding that ssh port 22 on <br>
> that IP is responding. So I logged into the hypervisor (compute node) on <br>
> which that vm was there before deletion and found that the libvirtd <br>
> service was is error state with the following logs in the nova-compute <br>
> service:<br>
> <br>
> ---------<br>
> 2023-08-18 10:08:51.463 12587 INFO nova.service [-] Starting compute <br>
> node (version 19.3.2)<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host <br>
> self.force_reraise()<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host <br>
> six.reraise(self.type_, self.value, self.tb)<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 442, in _get_connection<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host <br>
> self._wrapped_conn = self._get_new_connection()<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 396, in _get_new_connection<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host <br>
> wrapped_conn = self._connect(self._uri, self._read_only)<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 252, in _connect<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host <br>
> libvirt.openAuth, uri, auth, flags)<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host rv = <br>
> execute(f, *args, **kwargs)<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host <br>
> six.reraise(c, e, tb)<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host rv = <br>
> meth(*args, **kwargs)<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/libvirt.py", <br>
> line 104, in openAuth<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host if ret is <br>
> None:raise libvirtError('virConnectOpenAuth() failed')<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host libvirtError: <br>
> Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection <br>
> refused<br>
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service <br>
> [req-4108e9e2-2afd-472e-a4d9-fcccffc380dc - - - - -] Error starting <br>
> thread.: HypervisorUnavailable: Connection to the hypervisor is broken <br>
> on host<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service Traceback (most <br>
> recent call last):<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_service/service.py", line 810, in run_service<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service service.start()<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/service.py", line 176, in start<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service <br>
> self.manager.init_host()<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/compute/manager.py", line 1225, in init_host<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service <br>
> self.driver.init_host(host=self.host)<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 503, in init_host<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service <br>
> self._do_quality_warnings()<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 471, in _do_quality_warnings<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service caps = <br>
> self._host.get_capabilities()<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 664, in get_capabilities<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service xmlstr = <br>
> self.get_connection().getCapabilities()<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File <br>
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 482, in get_connection<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service raise <br>
> exception.HypervisorUnavailable()<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service <br>
> HypervisorUnavailable: Connection to the hypervisor is broken on host<br>
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service<br>
> 2023-08-22 14:28:07.678 12587 INFO nova.virt.libvirt.driver [-] <br>
> Connection event '0' reason 'Failed to connect to libvirt: Failed to <br>
> connect socket to '/var/run/libvirt/libvirt-sock': Connection refused'<br>
> -----<br>
> <br>
> However that particular neutron port for that vm is also deleted. We <br>
> cant find it in the neutron db.<br>
> <br>
> So how is it possible for that VM IP to be responding on port 22 even <br>
> after it is deleted. Please help me here.<br>
<br>
Your above log says that libvirt is down. Therefore, Nova cannot ask it <br>
to shutdown the VM. Restart libvirt, and see what's going on...<br>
<br>
Cheers,<br>
<br>
Thomas Goirand (zigo)<br>
<br>
<br>
</blockquote></div>