On 8/22/23 17:01, Gk Gk wrote:

Hi All,

We have an openstack ansible stein version setup. We had a vm on it which we deleted. The associated neutron port was also deleted and the vm uuid is gone from the nova as well. It had a provider network ip. Even after deleting the vm, we are still finding that ssh port 22 on that IP is responding. So I logged into the hypervisor (compute node) on which that vm was there before deletion and found that the libvirtd service was is error state with the following logs in the nova-compute service:

--------- 2023-08-18 10:08:51.463 12587 INFO nova.service [-] Starting compute node (version 19.3.2) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host self.force_reraise() 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host six.reraise(self.type_, self.value, self.tb) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 442, in _get_connection 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host self._wrapped_conn = self._get_new_connection() 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 396, in _get_new_connection 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 252, in _connect 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     rv = execute(f, *args, **kwargs) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host six.reraise(c, e, tb) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     rv = meth(*args, **kwargs) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/libvirt.py", line 104, in openAuth 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     if ret is None:raise libvirtError('virConnectOpenAuth() failed') 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service [req-4108e9e2-2afd-472e-a4d9-fcccffc380dc - - - - -] Error starting thread.: HypervisorUnavailable: Connection to the hypervisor is broken on host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service Traceback (most recent call last): 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_service/service.py", line 810, in run_service 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     service.start() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/service.py", line 176, in start 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service self.manager.init_host() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/compute/manager.py", line 1225, in init_host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service self.driver.init_host(host=self.host) 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 503, in init_host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service self._do_quality_warnings() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 471, in _do_quality_warnings 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     caps = self._host.get_capabilities() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 664, in get_capabilities 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 482, in get_connection 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     raise exception.HypervisorUnavailable() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service HypervisorUnavailable: Connection to the hypervisor is broken on host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service 2023-08-22 14:28:07.678 12587 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused' -----

However that particular neutron port for that vm is also deleted. We cant find it in the neutron db.

So how is it possible for that VM IP to be responding on port 22 even after it is deleted. Please help me here.

Your above log says that libvirt is down. Therefore, Nova cannot ask it to shutdown the VM. Restart libvirt, and see what's going on... Cheers, Thomas Goirand (zigo)

Hey, I assume that port is being listened, because VM is still running in QEMU, since nova-compute failed to connect to libvirtd to delete the instance. So you should somehow map instance id in libvirtd with one in nova. IIRC, nova will drop OS-EXT-SRV-ATTR:instance_name from the database once it deletes instance (but not 100% sure - maybe it stays somewhere in database). So you need to: 1. Restart libvirtd and ensure that nova-compute can connect to it 2. Just in case restart nova-compute. This might not be needed 3. In case VM is still pingable - get list of instance names in libvirt and compare it with ones in `virsh list`. Eventually, nova-compute should take care of this, but in case it didn't have connection to libvirt - that would explain the behaviour. вт, 22 авг. 2023 г. в 17:09, Gk Gk <ygk.kmr@gmail.com>:

Hi All,

We have an openstack ansible stein version setup. We had a vm on it which we deleted. The associated neutron port was also deleted and the vm uuid is gone from the nova as well. It had a provider network ip. Even after deleting the vm, we are still finding that ssh port 22 on that IP is responding. So I logged into the hypervisor (compute node) on which that vm was there before deletion and found that the libvirtd service was is error state with the following logs in the nova-compute service:

--------- 2023-08-18 10:08:51.463 12587 INFO nova.service [-] Starting compute node (version 19.3.2) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host self.force_reraise() 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host six.reraise(self.type_, self.value, self.tb) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 442, in _get_connection 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host self._wrapped_conn = self._get_new_connection() 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 396, in _get_new_connection 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 252, in _connect 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host six.reraise(c, e, tb) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/libvirt.py", line 104, in openAuth 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed') 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service [req-4108e9e2-2afd-472e-a4d9-fcccffc380dc - - - - -] Error starting thread.: HypervisorUnavailable: Connection to the hypervisor is broken on host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service Traceback (most recent call last): 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_service/service.py", line 810, in run_service 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service service.start() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/service.py", line 176, in start 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service self.manager.init_host() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/compute/manager.py", line 1225, in init_host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service self.driver.init_host(host=self.host) 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 503, in init_host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service self._do_quality_warnings() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 471, in _do_quality_warnings 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service caps = self._host.get_capabilities() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 664, in get_capabilities 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service xmlstr = self.get_connection().getCapabilities() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 482, in get_connection 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service raise exception.HypervisorUnavailable() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service HypervisorUnavailable: Connection to the hypervisor is broken on host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service 2023-08-22 14:28:07.678 12587 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused' -----

However that particular neutron port for that vm is also deleted. We cant find it in the neutron db.

So how is it possible for that VM IP to be responding on port 22 even after it is deleted. Please help me here.

Thanks Kumar