- openstack-discuss - lists.openstack.org

Vault cannot authorize approle
by Andy Speagle 17 Dec '24

17 Dec '24

Hey Team, I'm working with charmed openstack and it looks like I let my certs expire stupidly... and now I'm seeing this from vault: Vault cannot authorize approle Further info from the logs indicate this: unit-vault-0: 08:42:04 WARNING unit.vault/0.juju-log InternalServerError: Unable to authorize approle. This may indicate failure to communicate with the database unit-vault-0: 08:42:04 ERROR unit.vault/0.juju-log Traceback (most recent call last): File "/var/lib/juju/agents/unit-vault- 0/charm/reactive/vault_handlers.py", line 896, in client_approle_authorized vault.get_local_client() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 339, in wrapped_f return self(f, *args, **kw) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 430, in __call__ do = self.iter(retry_state=retry_state) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 378, in iter raise retry_exc.reraise() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 206, in reraise raise self.last_attempt.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 433, in __call__ result = fn(*args, **kwargs) File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 258, in get_local_client client.auth_approle(app_role_id) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/v1/__init__.py", line 2072, in auth_approle return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/v1/__init__.py", line 1726, in auth return self._adapter.auth( File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 159, in auth response = self.post(url, **kwargs).json() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 103, in post return self.request('post', url, **kwargs) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 233, in request utils.raise_for_error(response.status_code, text, errors=errors) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/utils.py", line 39, in raise_for_error raise exceptions.InternalServerError(message, errors=errors) hvac.exceptions.InternalServerError: internal error The database seems to be up and accessible... so, I gotta believe it's a cert issue... so, I feel like this might be a chicken-egg issue where it can't connect to the DB due to certs... but then can't issue certs because it can't connect to the DB. Any thoughts? ----- Andy Speagle

2 4

[neutron] bug deputy report for week of 2024-12-09+
by Ihar Hrachyshka 16 Dec '24

16 Dec '24

Hi all, Bugs ------- Gate issues ========= 1. fullstack failures in l3ha agent, placeholder: https://bugs.launchpad.net/neutron/+bug/2091351 May be related to https://bugs.launchpad.net/neutron/+bug/2083609 There's a fix for the latter: https://review.opendev.org/c/openstack/neutron/+/937758 2. functional suite OOMing when CPU number is low: https://bugs.launchpad.net/neutron/+bug/2091855 There's a fix to split the suite in serial runs: https://review.opendev.org/c/openstack/neutron/+/937759 The bug alludes that memory consumption could be improved; I'd recommend to limit the scope of this bug to fixing the CI failure and let the memory concerns be addressed elsewhere / by another owner if needed. 3. tempest failure in stable branch in psql job: https://bugs.launchpad.net/neutron/+bug/2091711 No fix; no owner. API / policy ========= All assigned to Slawek; not sure if he really wants to own all of them. 1. no quota on the number of tags breaks api: https://bugs.launchpad.net/neutron/+bug/2091410 DoS when too many tags are created; CVE adjacent. Need quotas for tags? 2. policy mechanism allows to work around some limitations on external connectivity when fwaas is used: https://bugs.launchpad.net/neutron/+bug/2091536 Not sure if it has a security angle or affects anything but fwaas. 3. custom policy rule failing due to missing fields: https://bugs.launchpad.net/neutron/+bug/2091493 Needs owner ========== 1. Unicast leakage on flat network: https://bugs.launchpad.net/neutron/+bug/2091211 The next step would be to set up a 2-compute devenv setup with flat and see if ml2/ovs openvswitch driver can reproduce. Additional tests to functional / fullstack / tempest validating leakage is not happening could be helpful too. Already fixed ========== 1. 500 when backend doesn't support vlan-transparent: https://bugs.launchpad.net/neutron/+bug/2091613 2. pylint change broke ovn agent delete: https://bugs.launchpad.net/neutron/+bug/2091071 New RFEs -------------- 1. use unicast mode for keepalived to avoid vxlan flooding when mcast is off: https://bugs.launchpad.net/neutron/+bug/2091599 Ihar

1 0

[Neutron] DHCP port status inactive
by Vinicius Afonso 16 Dec '24

16 Dec '24

Hello, I am new to Openstack environment and I need help with a Neutron problem. When I create a new network, the DHCP port launches with status inactive. Even though I tried to create a VM, the DHCP generate a port for the machine, However it takes a few minutes to give an error: "Build of instance ed6c8835-f2d4-4730-9638-74ded388361b aborted: Failed to allocate the network(s), not rescheduling." The only alternative solution to overcome this issue is to change the DHCP agent of the network. If I change it, the new DHCP port will have an active status and then I can launch a VM. Nothing conclusive was found in the logs, just the error to bind the port. It doesn't say anything to why the port is inactive, just that the port was created and with status active. Things that I already tried: -> See nova and neutron logs -> Restart neutron-server, neutron-dhcp-agent, neutron-openvswitch-agent -> Change manually the DHCP agent Anyone had the same problem as this one? Any tips to troubleshoot? Any help is welcome! Thanks.

2 1

Dalmatian: Dimensions of instance console
by berndbausch＠gmail.com 16 Dec '24

16 Dec '24

This is a Dalmatian-based Devstack. The instance console's height is too large for my screen. Documentation claims that console dimensions can be configured in a file named _detail_vnc.html [1], but there is no file with this name anywhere in the filesystem. I just submitted a bug [2] but wonder if somebody can help me out right now. How can the console dimensions be configured? Thanks! Bernd Bausch Instructor and tinkerer Here is my bug description: Under Frequently Asked Questions (https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#fre quently-asked-questions), the answer to Q: How do I adjust the dimensions of the VNC window image in the OpenStack dashboard? is incorrect. My Devstack installation contains no file named _detail_vnc.html. I found a file with the interesting name _detail_console.html under /opt/stack/horizon/openstack_dashboard/dashboards/project/instances/template s/instances, but it doesn't contain the answer's code for width and height. [1] https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#freq uently-asked-questions [2] https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#freq uently-asked-questions

1 0

[publiccloud-sig] Reminder: next meeting December 18th - 0800 UTC
by Felix Kronlage-Dammers 16 Dec '24

16 Dec '24

Hi everyone, This upcoming Wednesday (December 18th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See ya in our pre-christmas special edition of the public cloud SIG :) felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-15)
by Goutham Pacha Ravi 15 Dec '24

15 Dec '24

Hello Stackers, We're 15 weeks away from the 2025.1 ("Epoxy") release date (2024-04-02) [1]. During the last week, the Technical Committee continued discussing the state of "unmaintained" code branches across OpenStack repositories and election officials proposed dates for the combined TC+PTL elections for the next release cycle [2]. We have decided to cancel the weekly meetings previously scheduled for 24th December and 31st December in lieu of the holidays. === Weekly Meeting === The OpenStack Technical Committee met over IRC on 2024-12-10 [3]. We revisited challenges with using CentOS 10 in the community CI system due to lack of AVX/AVX2 CPU flags on current cloud providers. Discussions revolved around finding providers with suitable hardware or reducing reliance on CentOS testing. We then revisited the state of "unmaintained" branches across OpenStack repositories. The oldest "unmaintained" branch tracks the "victoria" release (GA: 2020-10-14). The consensus amongst TC members was to enforce the policy to default old branches to EOL unless explicitly opted-in with required maintenance assurances. To do this, we need tooling to automate the transition, and a way to track and validate volunteer opt-ins for unmaintained branches. Volunteers stepped up to transition branches up until the 2023.1 ("Antelope") release to "end-of-life", beginning with the Victoria release branch [4]. If you're interested in keeping a particular branch open, for any repository, please express your interest directly on the gerrit change. All EOL transition patches will be merged after 1 month of community review. The next meeting of the OpenStack Technical Committee is on 2024-12-17 at 1800 UTC. This meeting will take place in the #openstack-tc channel on OFTC's IRC server. Please find the meeting agenda on the wiki [5]. Please remember that you can propose topics for upcoming TC meetings by directly editing the wiki. If you're unable to do so, please holler at me (IRC nick: gouthamr) or in the TC's IRC channel (#openstack-tc). === Governance Proposals === ==== Merged ==== - Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 ==== Open for Review ==== - Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 - Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 - Allow more than 2 weeks for elections | https://review.opendev.org/c/openstack/governance/+/937741 === How to Contact the TC === You can reach the TC in several ways: - Email: send an email with the tag [tc] on this mailing list. - IRC: Ping us using the tc-members keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] 2025.2/"F" elections proposal: https://review.opendev.org/c/openstack/election/+/937408 [3] TC Meeting IRC Log 2024-12-10: https://meetings.opendev.org/meetings/tc/2024/tc.2024-12-10-18.01.log.html [4] Transition unmaintained/victoria to EOL: https://review.opendev.org/c/openstack/releases/+/937515 [5] TC Meeting Agenda, 2024-12-17: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[nova] Next 3 meetings are CANCELLED
by Sylvain Bauza 15 Dec '24

15 Dec '24

Hi folks, Apologies for this email, but I was needing to travel this week so I can't be around for this week's meeting and then I will be on PTO for Dec 24 and 31. Thanks, Sylvain

1 0

Diagnosing Sluggish Instance Boot Performance in OpenStack Cluster
by Abhijit Singh Anand 14 Dec '24

14 Dec '24

Dear All, Problem Statement: Instances remain stuck in the “Block Device Mapping” state for an extended period (~9 minutes) before eventually booting. This issue is observed consistently in one of my OpenStack clusters, while another cluster built on similar hardware and configuration spins up instances much faster. Environment Details: 1. Cluster Configuration: * Deployed using Kolla-Ansible on similar hardware specifications. * Both clusters connect to a common Ceph SSD-backed storage cluster. * Connection: 20 Gbps bonded interface over Cisco Nexus switches. * Same volumes used for images, VMs, backups. * Ceph keyrings and configurations are identical. 2. Performance Comparison: * “Good” cluster: Instances (even with large images >20 GB) boot to a login prompt in ~1-2 minutes. * “Bad” cluster: Instances with the same images and size take ~8-9 minutes. 3. Storage Backend Architecture: * “Good” cluster: Cinder-volume service is co-resident on compute nodes. * “Bad” cluster: Cinder-volume service runs only on controller nodes. 4. Instance Images: * Performance issue occurs with both RAW and QCOW2 images. * Example: A Windows 10 Pro RAW image boots in 1 min 40 sec in the "Good" cluster and takes 8 min 45 sec in the "Bad" cluster. What I’ve Tried So Far: * Configuration Matching: Reviewed and aligned as many configuration parameters as possible between the two clusters. * Resource Monitoring: Verified CPU and memory usage on nodes during instance launches—no significant resource contention observed. * Ceph Performance: Checked Ceph cluster performance; it seems healthy with no latency or throughput issues. * Network Testing: Verified 20 Gbps bond interfaces on both clusters; no observed bottlenecks. Hypothesis: I suspect the cinder-volume architecture could be contributing to the issue. In the "Good" cluster, cinder-volume is co-located with compute nodes, possibly optimizing data transfer paths. In contrast, the "Bad" cluster relies solely on controller-hosted cinder-volume services, which might introduce delays in data availability to compute nodes. Request for Community Input: 1. Does the difference in the placement of the cinder-volume service (co-resident vs. controller-only) align with the observed performance discrepancy? 2. Could there be other subtle configuration or architectural nuances that might explain the delays in the "Bad" cluster? 3. What specific logs or metrics would be most insightful in diagnosing this issue further? * I’ve reviewed Nova and Cinder logs but could dive deeper if pointed in the right direction. 4. Are there recommended best practices for tuning Cinder, Nova, or Ceph for environments with high storage performance requirements? Thank You! I appreciate any insights, suggestions, or pointers you can provide to help narrow down the root cause of this frustrating performance discrepancy. Regards, Abhijit S Anand

2 2

[manila] Epoxy Bugsquash: Dec 17th to 19th
by Carlos Silva 13 Dec '24

13 Dec '24

Hello zorillas and interested stackers, The first bugsquash of this cycle is right around the corner! Our bug backlog has been increasing for a while and we are seeing a trend where bugs are staying in progress for some time. For that reason, we will focus on reviews and following up on changes for the next week's bugsquash. Vida has kindly helped us put a list of bugs together [1]. We will be meeting at 15 UTC on Tuesday using meetpad [2], as agreed and we will use the next week's weekly meeting as a checkpoint for the bugsquash, where we'll discuss the progress of the bugs. [1] https://ethercalc.net/a4lehhy9ilp5 [2] https://meetpad.opendev.org/manila-epoxy-bugsquash I'm looking forward to it! carloss

1 0

[release] Release countdown for week R-15
by Thierry Carrez 13 Dec '24

13 Dec '24

Development Focus ----------------- The Epoxy-2 milestone will happen next month, on January 9th, 2025. 2025.1 "Epoxy"-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/epoxy/schedule.html for details. General Information ------------------- Please remember that libraries need to be released at least once per milestone period. At milestone 2, the release team will propose releases for any library that has not been otherwise released since milestone 1. Other non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. At milestone-2 we also freeze the contents of the final release. If you have a new deliverable that should be included in the final release, you should make sure it has a deliverable file in: https://opendev.org/openstack/releases/src/branch/master/deliverables/epoxy You should request a beta release (or intermediary release) for those new deliverables by milestone-2. We understand some may not be quite ready for a full release yet, but if you have something minimally viable to get released it would be good to do a 0.x release to exercise the release tooling for your deliverables. See the MembershipFreeze description for more details: https://releases.openstack.org/epoxy/schedule.html#e-mf Finally, now may be a good time for teams to check on any stable releases that need to be done for your deliverables. If you have bugfixes that have been backported, but no stable release getting those. If you are unsure what is out there committed but not released in the openstack/releases repo, running the command "tools/list_stable_unreleased_changes.sh <cycle_name>" gives a nice report. Upcoming Deadlines & Dates -------------------------- Epoxy-2 Milestone: January 9th, 2025 2025.1 Epoxy final release: April 2nd, 2025 -- Thierry Carrez (ttx)

1 0