August 2023 - openstack-discuss - lists.openstack.org

Need assistance
by Gk Gk 22 Aug '23

22 Aug '23

Hi All, We have an openstack ansible stein version setup. We had a vm on it which we deleted. The associated neutron port was also deleted and the vm uuid is gone from the nova as well. It had a provider network ip. Even after deleting the vm, we are still finding that ssh port 22 on that IP is responding. So I logged into the hypervisor (compute node) on which that vm was there before deletion and found that the libvirtd service was is error state with the following logs in the nova-compute service: --------- 2023-08-18 10:08:51.463 12587 INFO nova.service [-] Starting compute node (version 19.3.2) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host self.force_reraise() 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host six.reraise(self.type_, self.value, self.tb) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 442, in _get_connection 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host self._wrapped_conn = self._get_new_connection() 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 396, in _get_new_connection 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 252, in _connect 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host six.reraise(c, e, tb) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs) 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/libvirt.py", line 104, in openAuth 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed') 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused 2023-08-22 14:28:07.670 12587 ERROR nova.virt.libvirt.host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service [req-4108e9e2-2afd-472e-a4d9-fcccffc380dc - - - - -] Error starting thread.: HypervisorUnavailable: Connection to the hypervisor is broken on host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service Traceback (most recent call last): 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/oslo_service/service.py", line 810, in run_service 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service service.start() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/service.py", line 176, in start 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service self.manager.init_host() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/compute/manager.py", line 1225, in init_host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service self.driver.init_host(host=self.host) 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 503, in init_host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service self._do_quality_warnings() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 471, in _do_quality_warnings 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service caps = self._host.get_capabilities() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 664, in get_capabilities 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service xmlstr = self.get_connection().getCapabilities() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service File "/openstack/venvs/nova-stein-em/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 482, in get_connection 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service raise exception.HypervisorUnavailable() 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service HypervisorUnavailable: Connection to the hypervisor is broken on host 2023-08-22 14:28:07.676 12587 ERROR oslo_service.service 2023-08-22 14:28:07.678 12587 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused' ----- However that particular neutron port for that vm is also deleted. We cant find it in the neutron db. So how is it possible for that VM IP to be responding on port 22 even after it is deleted. Please help me here. Thanks Kumar

4 3

vPTG October 2023 Teams List
by Kendall Nelson 22 Aug '23

22 Aug '23

Hello Everyone! The October 2023 Project Teams List is official! Projects + Teams - OpenInfra Foundation Teams - - Diversity & Inclusion Working Group - OpenInfra Edge Computing Group - Environmental Sustainability Working Group - StarlingX - OpenStack - - Services - - Blazar - Cinder - CloudKitty - Designate - Glance - Horizon - Ironic - Kolla - Manila - Neutron - Nova - Octavia - OpenStack Charms - OpenStack-Helm - Quality Assurance - OpenStackSDK/CLI - Swift - Tacker - Other OpenStack Teams - - Interoperability Working Group - OpenStack Security SIG - PublicCloud SIG If your team was planning to meet and isn’t on this list, please contact ptg(a)openinfra.dev IMMEDIATELY. Soon I will be contacting moderators to sign up for time via the PTGBot[1] once I have it configured for the teams that are signed up to attend the event. Otherwise, please don't forget to register[2] — It's FREE! As usual, feel free to let us know if you have any questions! Thanks! -Kendall (diablo_rojo) [1] PTGBot Docs: https://github.com/openstack/ptgbot#open-infrastructure-ptg-bot [2] PTG Registration: http://ptg2023.openinfra.dev/

1 0

Collaborating Without Boundaries Working Group - Meeting #2 Tomorrow at 1500 UTC
by Helena Spease 22 Aug '23

22 Aug '23

Hi everyone! Tomorrow at 1500 UTC the OpenInfra Collaborating Without Boundaries Working Group is meeting! If you are a university or organization interested in OpenStack training come learn from Seongsoo Cho, how the Korean User Group is partnering with local universities to provide training opportunities to students. Etherpad : https://meetpad.opendev.org/etherpad/p/cwb <https://meetpad.opendev.org/etherpad/p/cwb> Meetpad: https://meetpad.opendev.org/CWB <https://meetpad.opendev.org/CWB> Cheers, Helena helena(a)openinfra.dev Marketing & Community Coordinator The OpenInfra Foundation

1 0

[tc] Technical Committee Next Weekly Meeting, Today on August 22, 2023
by Nikolla, Kristi 22 Aug '23

22 Aug '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held Today, on Tuesday, 22 August 2023 at 1800 UTC on #openstack-tc on OFTC IRC. The agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting and attached below. • Roll call • Follow up on past action items • No action items • Gate health check • Documenting implementation processes for Unmaintained • With the resolution being merged, the next step is defining and documenting the processes and implementing the required tooling. • Open Discussion and Reviews • https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla

1 0

[dev][glance_storage][s3] improving download speed from s3 backend
by Gökhan Koçak 22 Aug '23

22 Aug '23

Hello everyone, I did not find any option to use something like multipart downloads (only uploads) or concurrency when downloading from s3 storage. So my first question is if this is correct? Second, I see that boto3 is being used: https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_… and I wonder if it is possible to implement using the concurrency settings from boto: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customiza… ? Best, Gökhan

1 0

[nova] Slow nvme performance for local storage instances
by Jan Wasilewski 22 Aug '23

22 Aug '23

Hi, I am reaching out to inquire about the performance of our local storage setup. Currently, I am conducting tests using NVMe disks; however, the results appear to be underwhelming. In terms of my setup, I have recently incorporated two NVMe disks into my compute node. These disks have been configured as RAID1 under md127 and subsequently mounted at /var/lib/nova/instances [1]. During benchmarking using the fio tool within this directory, I am achieving approximately 160,000 IOPS [2]. This figure serves as a satisfactory baseline and reference point for upcoming VM tests. As the next phase, I have established a flavor that employs a root disk for my virtual machine [3]. Regrettably, the resulting performance yields around 18,000 IOPS, which is nearly ten times poorer than the compute node results [4]. While I expected some degradation, a tenfold decrease seems excessive. Realistically, I anticipated no more than a twofold reduction compared to the compute node's performance. Hence, I am led to ask: what should be configured to enhance performance? I have already experimented with the settings recommended on the Ceph page for image properties [5]; however, these changes did not yield the desired improvements. In addition, I attempted to modify the CPU architecture within the nova.conf file, switching to Cascade Lake architecture, yet this endeavor also proved ineffective. For your convenience, I have included a link to my current dumpxml results [6]. Your insights and guidance would be greatly appreciated. I am confident that there is a solution to this performance disparity that I may have overlooked. Thank you in advance for your help. /Jan Wasilewski *References:* *[1] nvme allocation and raid configuration: https://paste.openstack.org/show/bMMgGqu5I6LWuoQWV7TV/ <https://paste.openstack.org/show/bMMgGqu5I6LWuoQWV7TV/>* *[2] fio performance inside compute node: https://paste.openstack.org/show/bcMi4zG7QZwuJZX8nyct/ <https://paste.openstack.org/show/bcMi4zG7QZwuJZX8nyct/>* *[3] Flavor configuration: https://paste.openstack.org/show/b7o9hCKilmJI3qyXsP5u/ <https://paste.openstack.org/show/b7o9hCKilmJI3qyXsP5u/>* *[4] fio performance inside VM: https://paste.openstack.org/show/bUjqxfU4nEtSFqTlU8oH/ <https://paste.openstack.org/show/bUjqxfU4nEtSFqTlU8oH/>* *[5] image properties: https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties <https://docs.ceph.com/en/pacific/rbd/rbd-openstack/#image-properties>* *[6] dumpxml of vm: https://paste.openstack.org/show/bRECcaSMqa8TlrPp0xrT/ <https://paste.openstack.org/show/bRECcaSMqa8TlrPp0xrT/>*

5 18

[sunbeam] Juju 3.2 stable release break
by James Page 22 Aug '23

22 Aug '23

Morning The Juju project popped a new stable release of the 3.2 series we currently use for Sunbeam; this has created a problem with the mysql-k8s charm that we rely on for Sunbeam deployment and as such any new deployments will fail. The team is focussed on resolving this ASAP as this is breaking all of our check/gates right now as well. Regards James

1 0

[elections][tc] Self-nominate for Technical Committee
by Dmitriy Rabotyagov 21 Aug '23

21 Aug '23

Hello everyone! I want to nominate myself for the second term on a Technical Committee. During the first term, I actively participated in various TC activities, including a change of the platform for PTI, better defining expectations on projects for removing support for older Python versions, and others. While it took me time to get up to speed and understand better how things are done, I feel being efficient since then as well as bringing value to the table during discussions. Apart from TC activities, I am currently a Core Reviewer and Project Team Leader of OpenStack-Ansible and Vitrage projects. I am a strong advocate for diversity and inclusion, which now is even more important than ever before when corporations are diverting from their commitments to open-source software. I believe that diverse and community-driven projects better represent the real needs of end users and operators. Throughout my career, I have been dealing mostly with Public Clouds, which have a way wider variety of use cases and perspectives than any single corporate solution can have. With that in mind, I want to continue supporting an inclusive environment for all community members as well as connecting operators and developers worlds during the next term of the Technical Committee member. Thanks for your consideration!

3 2

swift_objects are not getting saved in container
by Gaurav Oberai 21 Aug '23

21 Aug '23

Hi Team, We have a setup running with OpenStack Swift setup and a proxy server. We have also used s3api middleware in proxy-server configuration to store objects like AWS s3. Now we are facing issues like objects are not getting stored inside the container instead we are able to store the objects on storage nodes successfully and get a successful response for every request. We have verified the same by listing the objects using the aws-s3 command line utility. However, when tried with OpenStack swift command buckets and objects are not visible inside the container. My concern is how objects are saved if the container is not saving them and how can we fix this issue to store all objects inside the container? Could you please help me with this? Thanks & Regards, Gaurav

2 1

[barbican][cinder][cloudkitty][freezer][heat][ironic][keystone][mistral][monasca][murano][nova][OpenStackSDK][oslo][sahara][Telemetry][zaqar][zun] broken gate on master branch
by Elõd Illés 21 Aug '23

21 Aug '23

Hi $teams_in_subject, I've generated test patches to see gate health (to projects' for cycle-with-intermediary deliverables, that have not merged any patches recently) and the patches show [1] that you have broken CI jobs on your gate. Could the teams please check what the issue / issues are and fix as soon as possible? Thanks in advance! (Note: multiple times I see that the failure in docs job is: Handler <function html_page_context at 0x7f86178f55a0> for event 'html-page-context' threw an exception (exception: object of type 'PosixPath' has no len()) so several gates can be fixed with a fix to this issue) [1] https://review.opendev.org/q/topic:release-health-check-cwi-bobcat+is:open Thanks, Előd Illés irc: elodilles @ #openstack-release

3 3