December 2024 - openstack-discuss

[openstack-ansible] Weekly meetings on Dec 24 and Dec 31 cancelled
by Dmitriy Rabotyagov 17 Dec '24

17 Dec '24

Hi, Last week we decided to cancel the weekly meetings on December 24 and December 31 2024 due to Christmas holidays as many people will not be around for the meeting. This means that the next meeting will happen only on January 7 2025. With that being said, I want to congratulate the community with the upcoming Christmas and New Year and wish to have plenty of fun during holidays! And thank for all the hard work that was put in during 2024.

1 0

[neutron] End of year meeting schedule
by Brian Haley 17 Dec '24

17 Dec '24

Hello Neutrinos: Just wanted to make sure everyone was aware of our meeting schedule for the rest of the year. As discussed in our meeting Tuesday, next week will be the last week of meetings for the year, specifically: December 13th - Drivers meeting cancelled December 16th - Neutron CI meeting December 17th - Neutron meeting December 20th - Drivers meeting cancelled December 23rd - Neutron CI meeting (?) will leave up to Yatin Meetings the week of December 23rd and December 30th are cancelled, starting up again the week of January 6th. As usual, reach out on IRC/gerrit/email with questions for me or the core team, but assume a potentially slow turn-around. I will be offline from December 19th to January 8th, checking-in occasionally to make sure there are no fires, Miguel has offered to lead the team meeting on January 7th in my absence (thanks Miguel!). -Brian, Neutron PTL

2 2

Vault cannot authorize approle
by Andy Speagle 17 Dec '24

17 Dec '24

Hey Team, I'm working with charmed openstack and it looks like I let my certs expire stupidly... and now I'm seeing this from vault: Vault cannot authorize approle Further info from the logs indicate this: unit-vault-0: 08:42:04 WARNING unit.vault/0.juju-log InternalServerError: Unable to authorize approle. This may indicate failure to communicate with the database unit-vault-0: 08:42:04 ERROR unit.vault/0.juju-log Traceback (most recent call last): File "/var/lib/juju/agents/unit-vault- 0/charm/reactive/vault_handlers.py", line 896, in client_approle_authorized vault.get_local_client() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 339, in wrapped_f return self(f, *args, **kw) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 430, in __call__ do = self.iter(retry_state=retry_state) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 378, in iter raise retry_exc.reraise() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 206, in reraise raise self.last_attempt.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 433, in __call__ result = fn(*args, **kwargs) File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 258, in get_local_client client.auth_approle(app_role_id) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/v1/__init__.py", line 2072, in auth_approle return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/v1/__init__.py", line 1726, in auth return self._adapter.auth( File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 159, in auth response = self.post(url, **kwargs).json() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 103, in post return self.request('post', url, **kwargs) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 233, in request utils.raise_for_error(response.status_code, text, errors=errors) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/utils.py", line 39, in raise_for_error raise exceptions.InternalServerError(message, errors=errors) hvac.exceptions.InternalServerError: internal error The database seems to be up and accessible... so, I gotta believe it's a cert issue... so, I feel like this might be a chicken-egg issue where it can't connect to the DB due to certs... but then can't issue certs because it can't connect to the DB. Any thoughts? ----- Andy Speagle

2 4

[neutron] bug deputy report for week of 2024-12-09+
by Ihar Hrachyshka 16 Dec '24

16 Dec '24

Hi all, Bugs ------- Gate issues ========= 1. fullstack failures in l3ha agent, placeholder: https://bugs.launchpad.net/neutron/+bug/2091351 May be related to https://bugs.launchpad.net/neutron/+bug/2083609 There's a fix for the latter: https://review.opendev.org/c/openstack/neutron/+/937758 2. functional suite OOMing when CPU number is low: https://bugs.launchpad.net/neutron/+bug/2091855 There's a fix to split the suite in serial runs: https://review.opendev.org/c/openstack/neutron/+/937759 The bug alludes that memory consumption could be improved; I'd recommend to limit the scope of this bug to fixing the CI failure and let the memory concerns be addressed elsewhere / by another owner if needed. 3. tempest failure in stable branch in psql job: https://bugs.launchpad.net/neutron/+bug/2091711 No fix; no owner. API / policy ========= All assigned to Slawek; not sure if he really wants to own all of them. 1. no quota on the number of tags breaks api: https://bugs.launchpad.net/neutron/+bug/2091410 DoS when too many tags are created; CVE adjacent. Need quotas for tags? 2. policy mechanism allows to work around some limitations on external connectivity when fwaas is used: https://bugs.launchpad.net/neutron/+bug/2091536 Not sure if it has a security angle or affects anything but fwaas. 3. custom policy rule failing due to missing fields: https://bugs.launchpad.net/neutron/+bug/2091493 Needs owner ========== 1. Unicast leakage on flat network: https://bugs.launchpad.net/neutron/+bug/2091211 The next step would be to set up a 2-compute devenv setup with flat and see if ml2/ovs openvswitch driver can reproduce. Additional tests to functional / fullstack / tempest validating leakage is not happening could be helpful too. Already fixed ========== 1. 500 when backend doesn't support vlan-transparent: https://bugs.launchpad.net/neutron/+bug/2091613 2. pylint change broke ovn agent delete: https://bugs.launchpad.net/neutron/+bug/2091071 New RFEs -------------- 1. use unicast mode for keepalived to avoid vxlan flooding when mcast is off: https://bugs.launchpad.net/neutron/+bug/2091599 Ihar

1 0

[Neutron] DHCP port status inactive
by Vinicius Afonso 16 Dec '24

16 Dec '24

Hello, I am new to Openstack environment and I need help with a Neutron problem. When I create a new network, the DHCP port launches with status inactive. Even though I tried to create a VM, the DHCP generate a port for the machine, However it takes a few minutes to give an error: "Build of instance ed6c8835-f2d4-4730-9638-74ded388361b aborted: Failed to allocate the network(s), not rescheduling." The only alternative solution to overcome this issue is to change the DHCP agent of the network. If I change it, the new DHCP port will have an active status and then I can launch a VM. Nothing conclusive was found in the logs, just the error to bind the port. It doesn't say anything to why the port is inactive, just that the port was created and with status active. Things that I already tried: -> See nova and neutron logs -> Restart neutron-server, neutron-dhcp-agent, neutron-openvswitch-agent -> Change manually the DHCP agent Anyone had the same problem as this one? Any tips to troubleshoot? Any help is welcome! Thanks.

2 1

Dalmatian: Dimensions of instance console
by berndbausch＠gmail.com 16 Dec '24

16 Dec '24

This is a Dalmatian-based Devstack. The instance console's height is too large for my screen. Documentation claims that console dimensions can be configured in a file named _detail_vnc.html [1], but there is no file with this name anywhere in the filesystem. I just submitted a bug [2] but wonder if somebody can help me out right now. How can the console dimensions be configured? Thanks! Bernd Bausch Instructor and tinkerer Here is my bug description: Under Frequently Asked Questions (https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#fre quently-asked-questions), the answer to Q: How do I adjust the dimensions of the VNC window image in the OpenStack dashboard? is incorrect. My Devstack installation contains no file named _detail_vnc.html. I found a file with the interesting name _detail_console.html under /opt/stack/horizon/openstack_dashboard/dashboards/project/instances/template s/instances, but it doesn't contain the answer's code for width and height. [1] https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#freq uently-asked-questions [2] https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#freq uently-asked-questions

1 0

[publiccloud-sig] Reminder: next meeting December 18th - 0800 UTC
by Felix Kronlage-Dammers 16 Dec '24

16 Dec '24

Hi everyone, This upcoming Wednesday (December 18th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See ya in our pre-christmas special edition of the public cloud SIG :) felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-15)
by Goutham Pacha Ravi 15 Dec '24

15 Dec '24

Hello Stackers, We're 15 weeks away from the 2025.1 ("Epoxy") release date (2024-04-02) [1]. During the last week, the Technical Committee continued discussing the state of "unmaintained" code branches across OpenStack repositories and election officials proposed dates for the combined TC+PTL elections for the next release cycle [2]. We have decided to cancel the weekly meetings previously scheduled for 24th December and 31st December in lieu of the holidays. === Weekly Meeting === The OpenStack Technical Committee met over IRC on 2024-12-10 [3]. We revisited challenges with using CentOS 10 in the community CI system due to lack of AVX/AVX2 CPU flags on current cloud providers. Discussions revolved around finding providers with suitable hardware or reducing reliance on CentOS testing. We then revisited the state of "unmaintained" branches across OpenStack repositories. The oldest "unmaintained" branch tracks the "victoria" release (GA: 2020-10-14). The consensus amongst TC members was to enforce the policy to default old branches to EOL unless explicitly opted-in with required maintenance assurances. To do this, we need tooling to automate the transition, and a way to track and validate volunteer opt-ins for unmaintained branches. Volunteers stepped up to transition branches up until the 2023.1 ("Antelope") release to "end-of-life", beginning with the Victoria release branch [4]. If you're interested in keeping a particular branch open, for any repository, please express your interest directly on the gerrit change. All EOL transition patches will be merged after 1 month of community review. The next meeting of the OpenStack Technical Committee is on 2024-12-17 at 1800 UTC. This meeting will take place in the #openstack-tc channel on OFTC's IRC server. Please find the meeting agenda on the wiki [5]. Please remember that you can propose topics for upcoming TC meetings by directly editing the wiki. If you're unable to do so, please holler at me (IRC nick: gouthamr) or in the TC's IRC channel (#openstack-tc). === Governance Proposals === ==== Merged ==== - Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 ==== Open for Review ==== - Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 - Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 - Allow more than 2 weeks for elections | https://review.opendev.org/c/openstack/governance/+/937741 === How to Contact the TC === You can reach the TC in several ways: - Email: send an email with the tag [tc] on this mailing list. - IRC: Ping us using the tc-members keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] 2025.2/"F" elections proposal: https://review.opendev.org/c/openstack/election/+/937408 [3] TC Meeting IRC Log 2024-12-10: https://meetings.opendev.org/meetings/tc/2024/tc.2024-12-10-18.01.log.html [4] Transition unmaintained/victoria to EOL: https://review.opendev.org/c/openstack/releases/+/937515 [5] TC Meeting Agenda, 2024-12-17: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[nova] Next 3 meetings are CANCELLED
by Sylvain Bauza 15 Dec '24

15 Dec '24

Hi folks, Apologies for this email, but I was needing to travel this week so I can't be around for this week's meeting and then I will be on PTO for Dec 24 and 31. Thanks, Sylvain

1 0

Diagnosing Sluggish Instance Boot Performance in OpenStack Cluster
by Abhijit Singh Anand 14 Dec '24

14 Dec '24

Dear All, Problem Statement: Instances remain stuck in the “Block Device Mapping” state for an extended period (~9 minutes) before eventually booting. This issue is observed consistently in one of my OpenStack clusters, while another cluster built on similar hardware and configuration spins up instances much faster. Environment Details: 1. Cluster Configuration: * Deployed using Kolla-Ansible on similar hardware specifications. * Both clusters connect to a common Ceph SSD-backed storage cluster. * Connection: 20 Gbps bonded interface over Cisco Nexus switches. * Same volumes used for images, VMs, backups. * Ceph keyrings and configurations are identical. 2. Performance Comparison: * “Good” cluster: Instances (even with large images >20 GB) boot to a login prompt in ~1-2 minutes. * “Bad” cluster: Instances with the same images and size take ~8-9 minutes. 3. Storage Backend Architecture: * “Good” cluster: Cinder-volume service is co-resident on compute nodes. * “Bad” cluster: Cinder-volume service runs only on controller nodes. 4. Instance Images: * Performance issue occurs with both RAW and QCOW2 images. * Example: A Windows 10 Pro RAW image boots in 1 min 40 sec in the "Good" cluster and takes 8 min 45 sec in the "Bad" cluster. What I’ve Tried So Far: * Configuration Matching: Reviewed and aligned as many configuration parameters as possible between the two clusters. * Resource Monitoring: Verified CPU and memory usage on nodes during instance launches—no significant resource contention observed. * Ceph Performance: Checked Ceph cluster performance; it seems healthy with no latency or throughput issues. * Network Testing: Verified 20 Gbps bond interfaces on both clusters; no observed bottlenecks. Hypothesis: I suspect the cinder-volume architecture could be contributing to the issue. In the "Good" cluster, cinder-volume is co-located with compute nodes, possibly optimizing data transfer paths. In contrast, the "Bad" cluster relies solely on controller-hosted cinder-volume services, which might introduce delays in data availability to compute nodes. Request for Community Input: 1. Does the difference in the placement of the cinder-volume service (co-resident vs. controller-only) align with the observed performance discrepancy? 2. Could there be other subtle configuration or architectural nuances that might explain the delays in the "Bad" cluster? 3. What specific logs or metrics would be most insightful in diagnosing this issue further? * I’ve reviewed Nova and Cinder logs but could dive deeper if pointed in the right direction. 4. Are there recommended best practices for tuning Cinder, Nova, or Ceph for environments with high storage performance requirements? Thank You! I appreciate any insights, suggestions, or pointers you can provide to help narrow down the root cause of this frustrating performance discrepancy. Regards, Abhijit S Anand

2 2