Open Stack

Restart the libvirtd. It should sort out




On Wed, 23 Aug 2023, 02:32 Thomas Goirand, <thomas at goirand.fr> wrote:

> On 8/22/23 17:01, Gk Gk wrote:
> > Hi All,
> >
> > We have an openstack ansible stein version setup. We had a vm on it
> > which we deleted. The associated neutron port was also deleted and the
> > vm uuid is gone from the nova as well. It had a provider network ip.
> > Even after deleting the vm, we are still finding that ssh port 22 on
> > that IP is responding. So I logged into the hypervisor (compute node) on
> > which that vm was there before deletion and found that the libvirtd
> > service was is error state with the following logs in the nova-compute
> > service:
> >
> > ---------
> > 2023-08-18 10:08:51.463 12587 INFO nova.service [-] Starting compute
> > node (version 19.3.2)
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host
> > self.force_reraise()
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_utils/excutils.py",
> line 196, in force_reraise
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host
> > six.reraise(self.type_, self.value, self.tb)
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py",
> line 442, in _get_connection
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host
> > self._wrapped_conn = self._get_new_connection()
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py",
> line 396, in _get_new_connection
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host
> > wrapped_conn = self._connect(self._uri, self._read_only)
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py",
> line 252, in _connect
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host
> > libvirt.openAuth, uri, auth, flags)
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py",
> line 148, in proxy_call
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     rv =
> > execute(f, *args, **kwargs)
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py",
> line 129, in execute
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host
> > six.reraise(c, e, tb)
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py",
> line 83, in tworker
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     rv =
> > meth(*args, **kwargs)
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host   File
> > "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/libvirt.py",
> > line 104, in openAuth
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host     if ret is
> > None:raise libvirtError('virConnectOpenAuth() failed')
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host libvirtError:
> > Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection
> > refused
> > 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service
> > [req-4108e9e2-2afd-472e-a4d9-fcccffc380dc - - - - -] Error starting
> > thread.: HypervisorUnavailable: Connection to the hypervisor is broken
> > on host
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service Traceback (most
> > recent call last):
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_service/service.py",
> line 810, in run_service
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service
> service.start()
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/service.py",
> line 176, in start
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service
> > self.manager.init_host()
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/compute/manager.py",
> line 1225, in init_host
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service
> > self.driver.init_host(host=self.host)
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> line 503, in init_host
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service
> > self._do_quality_warnings()
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> line 471, in _do_quality_warnings
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     caps =
> > self._host.get_capabilities()
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py",
> line 664, in get_capabilities
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     xmlstr =
> > self.get_connection().getCapabilities()
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service   File
> >
> "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py",
> line 482, in get_connection
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service     raise
> > exception.HypervisorUnavailable()
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service
> > HypervisorUnavailable: Connection to the hypervisor is broken on host
> > 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service
> > 2023-08-22 14:28:07.678 12587 INFO nova.virt.libvirt.driver [-]
> > Connection event '0' reason 'Failed to connect to libvirt: Failed to
> > connect socket to '/var/run/libvirt/libvirt-sock': Connection refused'
> > -----
> >
> > However that particular neutron port for that vm is also deleted. We
> > cant find it in the neutron db.
> >
> > So how is it possible for that VM IP to be responding on port 22 even
> > after it is deleted. Please help me here.
>
> Your above log says that libvirt is down. Therefore, Nova cannot ask it
> to shutdown the VM. Restart libvirt, and see what's going on...
>
> Cheers,
>
> Thomas Goirand (zigo)
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230823/fe26819d/attachment-0001.htm>