Open Stack

On 8/22/23 17:01, Gk Gk wrote:
> Hi All,
> 
> We have an openstack ansible stein version setup. We had a vm on it 
> which we deleted. The associated neutron port was also deleted and the 
> vm uuid is gone from the nova as well. It had a provider network ip.  
> Even after deleting the vm, we are still finding that ssh port 22 on 
> that IP is responding. So I logged into the hypervisor (compute node) on 
> which that vm was there before deletion and found that the libvirtd 
> service was is error state with the following logs in the nova-compute 
> service:
> 
> ---------
> 2023-08-18 10:08:51.463 12587 INFO nova.service [-] Starting compute 
> node (version 19.3.2)
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     
> self.force_reraise()
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     
> six.reraise(self.type_, self.value, self.tb)
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 442, in _get_connection
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     
> self._wrapped_conn = self._get_new_connection()
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 396, in _get_new_connection
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     
> wrapped_conn = self._connect(self._uri, self._read_only)
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 252, in _connect
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     
> libvirt.openAuth, uri, auth, flags)
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     rv = 
> execute(f, *args, **kwargs)
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     
> six.reraise(c, e, tb)
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     rv = 
> meth(*args, **kwargs)
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/libvirt.py", 
> line 104, in openAuth
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     if ret is 
> None:raise libvirtError('virConnectOpenAuth() failed')
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host libvirtError: 
> Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection 
> refused
> 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service 
> [req-4108e9e2-2afd-472e-a4d9-fcccffc380dc - - - - -] Error starting 
> thread.: HypervisorUnavailable: Connection to the hypervisor is broken 
> on host
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service Traceback (most 
> recent call last):
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_service/service.py", line 810, in run_service
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     service.start()
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/service.py", line 176, in start
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     
> self.manager.init_host()
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/compute/manager.py", line 1225, in init_host
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     
> self.driver.init_host(host=self.host)
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 503, in init_host
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     
> self._do_quality_warnings()
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 471, in _do_quality_warnings
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     caps = 
> self._host.get_capabilities()
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 664, in get_capabilities
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     xmlstr = 
> self.get_connection().getCapabilities()
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File 
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 482, in get_connection
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     raise 
> exception.HypervisorUnavailable()
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service 
> HypervisorUnavailable: Connection to the hypervisor is broken on host
> 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service
> 2023-08-22 14:28:07.678 12587 INFO nova.virt.libvirt.driver [-] 
> Connection event '0' reason 'Failed to connect to libvirt: Failed to 
> connect socket to '/var/run/libvirt/libvirt-sock': Connection refused'
> -----
> 
> However that particular neutron port for that vm is also deleted. We 
> cant find it in the neutron db.
> 
> So how is it possible for that VM IP to be responding on port 22 even 
> after it is deleted. Please help me here.

Your above log says that libvirt is down. Therefore, Nova cannot ask it 
to shutdown the VM. Restart libvirt, and see what's going on...

Cheers,

Thomas Goirand (zigo)