September 2024 - openstack-discuss

Can only live migrate between hypervisors with same kernel
by sinan86polat＠gmail.com 10 Sep '24

10 Sep '24

Is it normal behavior that instances can only be live migrated between hypervisors running the same kernel? Like this I am not able to patch my hypervisors without downtime on the instances. When I live migrate all instances from hypervisor A to hypervisor B, patch hypervisor A, I cannot migrate the instances back to hypervisor A since it is running a newer kernel. The error is: guest CPU doesn't match specification: missing features: xsaves My source and target hypervisors have both the same CPU. Running OpenStack Rocky.

4 3

Clarification on Custom Role and Sub-User Management in OpenStack
by Thamanna Farhath 10 Sep '24

10 Sep '24

Dear OpenStack Community, I hope this message finds you well. I am reaching out to seek clarification and guidance on a scenario involving custom roles and sub-user management within OpenStack. In our deployment, we have the standard admin role as well as a custom role called super_user, which grants privileges for CRUD operations on projects, role assignments, and user management. We currently have two users with the super_user role, named Alice and John. Each of these super_user accounts manages their own sub-users: Alice manages the ABC sub-users, and John manages the XYZ sub-users. I would like to implement a restriction where Alice, as a super_user, can only manage the A,B,C sub-users and should not be able to delete or manage John's X,Y,Z sub-users. Similarly, John should not be able to manage or delete Alice's A,B,C sub-users. My questions are as follows: How can I identify and ensure that ABC sub-users belong to Alice and X,Y,Z sub-users belong to John? What is the best approach to implement this restriction within OpenStack, preventing cross-management of sub-users between Alice and John? Will using the multi-domain concept help me achieve this? Any insights, best practices, or suggestions on how to configure this within OpenStack would be greatly appreciated. Thank you for your assistance and support. Thanks & Regards, Thamanna Farhath | Associate Engineer-R&D Ph No: (+91) 9344093591 email: mailto:thamanna.f@zybisys.com Zybisys Consulting LLP | NO.1B 2nd floor, NSA Tower, Akila Nagar, Main Rd, Ganapathy Nagar, Thiruvanaikoil, Tiruchirappalli, Tamil Nadu 620005 zybisys.com Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 0

[swift] Ideas: Object Locking
by David Zanetti 10 Sep '24

10 Sep '24

Hello everyone! We've added an entry to the Swift ideas page about implementing Object Locks. The full description of what this would be and how we expect it to work is in the ideas page here: https://wiki.openstack.org/wiki/Swift/ideas/object-lock I have two questions of the community. The first is whether work on this would be accepted by the Swift project, as we wouldn't want to expend efforts on something we'd have to carry as our own patches to upstream. The second is if there is any interest from other people to collaborate on design/implementation of the locking feature. Happy to take replies off-list if people prefer, and thanks for reading this far! (Ob-reply: Yes, we know Ceph RGW does this already. We use Swift for Object for $REASONS, which we'd like to keep doing.) -- David Zanetti Chief Technology Officer [1]Aotearoa New Zealand's cloud provider e david.zanetti(a)catalystcloud.nz w catalystcloud.nz Follow us on LinkedIn [2] Level 9, 33 Federal Street, Auckland 1010 [1] https://catalystcloud.nz/ [2] Follow us on LinkedIn https://www.linkedin.com/company/catalyst-cloud-limited/

2 1

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-3)
by Goutham Pacha Ravi 10 Sep '24

10 Sep '24

Hello Stackers, Projects following a cycle-with-rc release model are hard at work alongside the release team to tag the first version of the 2024.2 release candidates (RC) this week [1]. The community would appreciate testing of these RC builds so we can weed out any critical issues prior to the coordinated release of this software in three weeks (2024-10-04). The OpenStack Technical Committee Election is ongoing. If you are a Foundation Individual Member who has been an Active Contributor during the past year, you should have received an email ballot with the details. If you haven't, please pop into #openstack-election and let us know. I hope you'll be able to make your vote count. === Weekly Meeting === The OpenStack Technical Committee met simultaneously over video [2] and IRC [3] last week. We took note of the smooth kick-off to the community elections and appreciated the effort the election officials have put in thus far. With the retirement of several inactive project teams and some focused outreach, we had far fewer projects without PTL candidates this time [4]. I will keep you posted on the TC's course of action in this matter over the next few weeks. At the meeting, we brainstormed reasons why these project teams had no PTL candidates, and whether Distributed Project Leadership is an appropriate choice for relatively stable project teams. We reflected on the community's experience with leadership appointments in the past where the appointed individuals couldn't sustain the project. These past experiences will help shape the TC's decisions. We noted that the maintenance phase transition for the stable/2023.1 ("Antelope") branch is coming up. The release team estimates transitioning the "stable/2023.1" branch across OpenStack repositories to "unmaintained/2023.1" [5] around 2024-10-30. After this transition, no further releases will be produced from these branches. If you are relying on this branch, please help project teams maintain testing and continuous integration for it. However, note that the release would be 19 months old at the time of this transition; consider upgrading the software to newer releases instead. The TC is considering a change to the OpenStack CI policy of re-enqueuing changes through Zuul when approved patches fail to get merged. Currently, if a change that fails "gate" jobs is re-enqueued, "check" jobs are re-run if new changes are detected in the repository. A suggestion was made to alter this policy so that only the "gate" jobs are repeated. We decided to defer this discussion to the upcoming PTG in October. We concluded the meeting by acknowledging the contributions Jay Faulkner (JayF) and Dan Smith (dansmith) made during their time on the TC. The elections will conclude on 2024-09-18. Again, please vote if you haven't already! :) The next meeting of the OpenStack Technical Committee is today: 2024-09-10. It will be hosted on OFTC's #openstack-tc channel at 1800 UTC. Please find the agenda on the meeting's wiki page [6]. I hope you'll be able to join. Please remember that you can propose discussion items to the agenda. === Governance Proposals === ==== Merged ==== - Fix typo in tested runtimes documents | https://review.opendev.org/c/openstack/governance/+/928603 ==== Open for review ==== - Add goal to migrate CI/CD jobs to Ubuntu Noble | https://review.opendev.org/c/openstack/governance/+/928078 - Adding Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 - Charms: retire unmaintained charms | https://review.opendev.org/c/openstack/governance/+/926793 - Retire all single charm repositories | https://review.opendev.org/c/openstack/governance/+/903490 - Retire kuryr-kubernetes and kuryr-tempest-plugin | https://review.opendev.org/c/openstack/governance/+/922507 === Upcoming Events === - 2024-09-10: OpenInfra Board Meeting: https://board.openinfra.dev/ - 2024-09-18: OpenStack 2025.1 Election Polling Ends - 2024-10-15: OpenInfra Days NA, Indianapolis: https://ittraining.iu.edu/explore-topics/titles/oid-iu/ - 2024-10-21: OpenInfra Project Teams Gathering https://openinfra.dev/ptg/ === How to Contact the TC === You can reach out to the TC in multiple ways: 1. **Email**: Send an email with the tag [tc] on this email list. 2. **IRC**: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel. 3. **Weekly Meeting**: The Technical Committee meets every week on Tuesdays at 1800 UTC [6]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] TC Meeting Video Recording, 2024-09-03: https://youtu.be/22oyI9vuvhM [3] TC Meeting IRC Log, 2024-09-03: https://meetings.opendev.org/meetings/tc/2024/tc.2024-09-03-18.00.log.html [4] "Leaderless" projects for the 2025.1 release cycle: https://etherpad.opendev.org/p/2025.1-leaderless [5] "Unmaintained" branches: https://docs.openstack.org/project-team-guide/stable-branches.html#unmainta… [6] TC Meeting Agenda, 2024-08-27: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[neutron] bug deputy report for week of 2024-09-02+
by Ihar Hrachyshka 10 Sep '24

10 Sep '24

Hi, Processed everything up to end-of-Mon (Sep 9). --- Gate failures: - https://bugs.launchpad.net/neutron/+bug/2078846: gate failure that happened to me, some race (?) in provisioning blocks / ovn port UP status propagation. Couldn't figure out the cause completely. Would benefit from a second pair of eyes. No patch. - https://bugs.launchpad.net/neutron/+bug/2079047: gate issue with one test in openvswitch-iptables_hybrid-distributed-dhcp flavor (not regular one!) No patch. - https://bugs.launchpad.net/neutron/+bug/2079048: gate failure in functional, sporadic, metadata/l3 related. No patch. - https://bugs.launchpad.net/neutron/+bug/2079831: tempest issue. Patch posted (WIP): https://review.opendev.org/c/openstack/tempest/+/928471 Actionable, need an owner: - https://bugs.launchpad.net/neutron/+bug/2078849: designate / S-RBAC integration changes needed. - https://bugs.launchpad.net/neutron/+bug/1995078: OVN: order of external chassis / gateway chassis lists should be stable. There's a WIP from Rodolfo but AFAIU someone else should take it over. - https://bugs.launchpad.net/neutron/+bug/2066989: SR-IOV; reproducer info was now attached. - https://bugs.launchpad.net/neutron/+bug/2079996: idea to restart haproxy only when needed. No patch. Marked as Low but isn't it a regression / datapath impact? Waiting for info from reporters: - https://bugs.launchpad.net/neutron/+bug/2059405: old bug with DNS, confirmed by the user with a non-.local. domain, asked to attach logs etc and then move to New; - https://bugs.launchpad.net/neutron/+bug/2078856: ovn related, waiting for logs / db dump for ovn to triage VPNaaS: - https://bugs.launchpad.net/neutron/+bug/2078845: vpnaas, need someone from the project owners to take a look. - https://bugs.launchpad.net/bugs/2080072: vpnaas, sqlalchemy transaction issue. Affected gate. No patch yet. FWaaS: - https://bugs.launchpad.net/neutron/+bug/2078918: fwaas, TypeError, patch posted. - https://launchpad.net/bugs/2078930: fwaas, patch posted. I hope it's helpful, Ihar

1 0

[nova][ci] cyborg-tempest (non-voting) job is failing
by melanie witt 10 Sep '24

10 Sep '24

Hey everyone, I noticed that the cyborg-tempest (non-voting) CI job began failing on the master branch sometime on 2024-09-03: https://zuul.opendev.org/t/openstack/builds?job_name=cyborg-tempest&project… The errors I see in the test result report are policy related, for example: Details: {'error_message': '{"faultcode": "Server", "faultstring": "\\"cyborg:device_profile:create\\": \\"rule:system_admin_api\\" requires a scope of [\'system\'], request was made with project scope.", "debuginfo": null}'} I'm guessing this is related to the recent "oslo.policy 4.4.0 enable the RBAC new defaults by default" change [1]: https://review.opendev.org/c/openstack/requirements/+/925464 The job is non-voting so it's not blocking anything, but it would be great if someone knows the best way to fix it. -melwitt [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…

2 1

Request objects
by engineer2024 09 Sep '24

09 Sep '24

How long do nova request objects live when a user request comes in ? Do they persist until the result is returned by the server ? Please throw some light on this... Thanks

1 0

[openstack][glance] Cannot rescue instance if image boot was deleted
by Nguyễn Hữu Khôi 09 Sep '24

09 Sep '24

Hi folks, I have this problem, It is glad if I had some guides for this situation. Openstack Version: 2023.1 OS: Ubuntu 22.04 Deployment Tool: Kolla-Ansible When I try to rescue an instance that booted from a volume, I cannot do so if the image used for the volume boot has been deleted This is an error message: Instance 6deeed71-c701-42a3-a8f9-d89048a47c11 cannot be rescued: Driver Error: Block Device Mapping is Invalid: failed to get image 844ea429-23a5-43d4-8f65-19c879d63921. Thank you. Nguyen Huu Khoi

4 6

Using diskimage-creator for vhd format
by Tony Lombardo 06 Sep '24

06 Sep '24

Hi all. As an example I was trying to run something like this command: sudo tox -e venv -- disk-image-create -o ubuntu-min-gold.vhd vm block-device-gpt ubuntu-minimal -t vhd However, I do get the error message that vhd-util isn't available. "vhd output format specified but no vhd-util executable found." I read the commit where this was deprecated in favor of qemu-img https://opendev.org/openstack/diskimage-builder/commit/a3e9e7f89e1059ae0929… I can't seem to get the syntax for the command to work, not sure what I am missing but any help would be greatly appreciated. QEMU emulator version 6.2.0 Diskimage-builder 3.33.1 I'm running this in WSL Ubuntu, it seems to work. I was able to create an image - just not the vhd format I need. Thanks in advance for any advice. -Tony -- Tony Lombardo Systems Administrator Division of Public Safety Access Control Department Cornell University

4 5

Struggling to rejion a node into rabbitmq cluster of Openstack-on-Ansible environment
by sudeb ghosh 06 Sep '24

06 Sep '24

Hi Team, We are operating an OpenStack environment managed by Ansible with 3 controller nodes. Openstack Version: Ussuri(21.2.1) OS: CentOS8 Recently, Controller-2 encountered a network partition with rabbitmq cluster issue that we couldn't resolve, leading us to remove it from the rabbitmq cluster and do rabbitmqctl reset. Since then, the RabbitMQ cluster node removal, We are currently unable to rejoin this node to the RabbitMQ cluster. The following error occurs when attempting to join the cluster: [Controller-2]# rabbitmqctl join_cluster rabbit@ctrl-1-rabbit-mq-container Clustering node rabbit@ctrl-2 with rabbit@ctrl-1 Error: {:failed_to_cluster_with, [:"rabbit@ctrl-3", :"rabbit@ctrl-1"], 'Mnesia could not connect to any nodes.'} We have verified that Controller-2 can communicate with other nodes in the cluster and it responds with a 'pong' for "rabbitmqctl eval 'net_adm:ping('\''rrabbit@ctrl-1'\'').'" There are no additional error logs indicating the root of the problem. We would appreciate any assistance in resolving this issue. Regards, Sudeb Ghosh 7044064878 9332034788

2 1

[all][tc][ptls] Eventlet: Python 3.13 - Attach your seatbelt
by Herve Beraud 06 Sep '24

06 Sep '24

Hi all, As Python 3.13 is close to become a reality, new concerning incompatibilities between this new CPython version and Eventlet also become a reality: - https://github.com/eventlet/eventlet/issues/964 Fedora Python maintainers proposed a patch, but for now the problem remains unsolved. Also, many people reported bugs related to incompatibility between python3.12/dnspython/eventlet/pip-system-cert. There is an example who summarize well the problem https://github.com/eventlet/eventlet/issues/957 Do you have some visibility concerning the adoption of Python 3.13 into Openstack? Turbulences ahead... -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

4 5

[release] Release countdown for week R-3, Sep 09-13
by Elõd Illés 06 Sep '24

06 Sep '24

Development Focus ----------------- The Release Candidate (RC) deadline is next Friday, September 13th, 2024. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/2024.2 branch will be cut. This branch will track the 2024.2 Dalmatian release. Once stable/2024.2 has been created, the master branch will be ready to switch to 2025.1 Epoxy development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing 2024.2 Dalmatian plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/2024.2 branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/2024.2 branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be 2025.1 Epoxy) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any 2024.2 Dalmatian release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: September 13th, 2024 (R-3 week) Final RC deadline: September 26th, 2024 (R-1 week) Final 2024.2 Dalmatian release: October 2nd, 2024 Előd Illés irc: elodilles @ #openstack-release

1 0

[releases][tempest] Tempest 39.1.0 release
by Rodolfo Alonso Hernandez 06 Sep '24

06 Sep '24

Hello all: Because I think I'm having some communication problems about [1], I would like to write it down in a mail. This release patch for 2024.2 is pointing to [2]. However, the requirements branch used in tempest was tagged as 2024.1. That means the tempest release for 2024.2 will use the requirements from 2024.1. There are other examples done in previous releases that support what I'm saying: * 2023.2: ** https://review.opendev.org/c/openstack/releases/+/896158 ** https://github.com/openstack/tempest/commit/3926608631111a0ec93b3c44129d0cb… * 2023.1: ** https://review.opendev.org/c/openstack/releases/+/877840 ** https://github.com/openstack/tempest/commit/7c7d0fdd67022ad48ed128c997dfcc3… BTW, despite not having the stable 2024.2 branch in "requirements", the link https://releases.openstack.org/constraints/upper/2024.2 is valid and points to master (once the stable branch is created, that will point to 2024.2). Regards. [1]https://review.opendev.org/c/openstack/releases/+/927833 [2]https://review.opendev.org/c/openstack/tempest/+/927831

2 2

[manila] Dalmatian bugsquash - Sep 9th - 10th
by Carlos Silva 06 Sep '24

06 Sep '24

Hello, Zorillas and interested stackers! We'll have the second bugsquash of the Dalmatian cycle next week. As we have agreed a couple of weeks ago, it will be a two day event (Sep 9 and 10th). We will have a kick-off meeting on Monday, 15 UTC and we will use a meetpad [1] for the meeting. We will be focusing on bugs that are in progress and try to get them delivered as part of the 2024.2 Dalmatian release. Please check the ethercalc with the list of bugs [2] and add the bug you are willing to work on over the next week if it is not there yet. I'm looking forward to it :) [1] https://meetpad.opendev.org/manila-dalmatian-m3-bugsquash [2] https://ethercalc.net/mgstvday5vq0 Regards, carloss

1 0

FIPs broken when used with tunneled router gateway networks
by Sesterhenn, Maximilian 06 Sep '24

06 Sep '24

Hey all, I am currently facing an issue with OpenStack Neutron and OVN and could need some help here. Maybe someone else is doing something similar and can report if it works for them. TLDR: FIPs are broken for us when used together with routers which use a tunneled network as router gateway network. ovn_router_indirect_snat is not used and both OVN 23.09 and 24.03 have been tested. FIPs work fine when used with an external provider network. We have deployed three OpenStack projects, let's just call the left, middle and right for simplicity. The middle project has a network with a large subnet. No special configuration here, just a big subnet which is transported by OVN through Geneve. The idea is to use this subnet as an internal transfer network between different projects / tenants, which means we connect routers with SNAT and FIP configuration. Left and right are potentially the same subnet, that's why we cannot simply route the traffic. The network is shared with "access_as_external/shared" RBAC rules with both the left and right tenant, so it can be selected as the external gateway network for routers. I'm aware that until very recently, neutron was unable to assign a chassis to routers which use gateway networks which are tunneled. Thats why we are currently running the latest neutron code from the master branch where this feature was added. Let's say we want to connect from left to right and we have a router in each project that has this large network from the middle project as the external gateway network. An instance in the right tenant has a FIP to be reachable. Another instance in the left tenant is masqueraded by the SNAT functionality of the router. The router in the left project performs SNAT and the router in the right project performs the FIP DNAT. While the instance from the left project can ping the internal and external interfaces of its local router, the FIP is not reachable. The external interface of the remote router (from the right project) can be pinged as well. When capturing packets on the underlying hosts we can see the packet going from the compute host of the left instance to one of the network nodes. SNAT happens there, DNAT is not visible, this could be due to the packet capture scenario. However, we see ARP requests coming from the external interface of the router on the right project for the FIP address. We see the same ARP requests on a second network node. To me it would make more sense if the router from the left project would ARP for the FIP address. This ARP request is not being answered. We're using enable_distributed_floating_ip / DVR, so the FIP is actually configured onto the compute node. We're not using ovn_router_indirect_snat, here are issues with that. We even tried that; it makes no difference. OVN 23.09 (tested 24.03 as well) OpenStack 2024.1 (neutron master) deployed using kolla-ansible. BR Maximilian

2 2

[charms] Proposing Brian Haley and Nicolas Bock for OpenStack Charms core
by Edward Hope-Morley 05 Sep '24

05 Sep '24

Hi everyone, I would like to propose Brian Haley and Nicolas Bock for the OpenStack Charms core development team. Both Brian and Nicolas have been members of the stable charms core team since 2023 and been hugely helpful in that role, providing insightful feedback to reviews as well as contributions of their own. Nicolas has also been a contributor to the OpenStack charms for many years and Brian is well versed in the ways of upstream OpenStack maintenance through his role as Neutron PTL so I have full confidence that they will prove valuable additions to the charms-core team. Thank you both for your contributions to the project! Regards, Ed

3 2

[Kolla] [LetsEncrypt] Internal/External/Backend Certificates
by jjjamesg＠proton.me 05 Sep '24

05 Sep '24

Hi all, another one here that can't seem to get kolla running with letsencrypt certs, just curious as to what settings are required for kolla to pull internal/external/backend certs from letsencrypt? I've looked at the docs and have tried a few settings but keep getting errors, sure i had this working last week before i tore it down ready to deploy for production and now i just cant get it to complete the deployment because of certs, only thing i can think of is i need to run kolla-ansible -i multinode certificates but i thought that was just for self made certs? Also, what are best practices for backend tls certs, self signed or using the internal cert etc?? Is it possible to use letsencrypt at all for backend? Cheers all

3 6

[neutron] OpenStack OVS DVR setup in large deployments
by frantisek.reznicek.szn＠gmail.com 05 Sep '24

05 Sep '24

Dear all, we are running several OpenStack instances in Czechia. New generation of clouds we launched are deployed OpenStack Yoga on top of k8s via openstack-helm, using Neutron OVS DVR. Very recently we noticed "wasting" public IP addresses for "network:floatingip_agent_gateway ports" as briefly discussed earlier here [1] while we were scaling number of nodes up (currently around 40, final state around 300). The linked post[1] suggests to use Service subnets and let allocate "network:floatingip_agent_gateway ports" from internal address range. The service subnets documentation page[2] comes with example where "network:floatingip_agent_gateway ports" being allocated from bogon ip range 198.51.100.0/24. I want to achieve state where no allow allocation of "network:floatingip_agent_gateway ports" is allocated from public IP range. I'd preferably stay using OVS+DVR if possible. 1] Could you possibly confirm that DVR setup will accept "network:floatingip_agent_gateway ports" being allocated from smaller network in private ipv4 class A range 10.0.0.0/8? At the moment "network:floatingip_agent_gateway ports" are the ports where all FIP trafic goes, so it is kind of surprising that private addresses can be used there. Could you elaborate more in detail on that? 2] We also use multiple physical provider network segments within one logical openstack network. Can you comment on whether Service subnets feature and multi-segment feature are compatible a can be used together? I believe service subnet for "network:floatingip_agent_gateway ports" has to be routed as segment too, is that right? 3] What steps I would need to do to start using OVS w/o DVR apart from disabling distributed routing in neutron and in neutron-ovs-agent and in L3 agent and restart? Is there any special procedure? Thanks upfront for your comments... Kind Regards, František [1] https://lists.openstack.org/pipermail/openstack-dev/2016-June/096384.html [2] https://docs.openstack.org/neutron/yoga/admin/config-service-subnets.html#e…

2 2

[cinder] Feature Freeze Exception Request
by Zong, Yian 05 Sep '24

05 Sep '24

Dear All, I'd like to request a feature freeze exception for dalmatian-3 milestone regarding: 907133: Dell PowerStore driver: Add Cinder active-active support | https://review.opendev.org/c/openstack/cinder/+/907133 This change adds cinder active-active support for Dell PowerStore driver and allows users to configure PowerStore backends in cinder clustered environment. Please let me know if you have any concern. Thank you! BR, Yian Internal Use - Confidential

1 0

[all][elections][ptl][tc] Conbined PTL/TC 2025.1 cycle Election Voting Kickoff
by Slawomir Kaplonski 05 Sep '24

05 Sep '24

Poll for TC elections is now open and will remain open for you to cast your vote until Sep 18, 2024 23:45 UTC. We are selecting 4 TC members. Please rank all candidates in your order of preference. You are eligible to vote in the TC election if you are a Foundation individual member[0] that also has committed to any official project team's deliverable repositories[1] over the Sep 15, 2023 00:00 UTC - Aug 28, 2024 00:00 UTC timeframe (2024.1 to 2024.2) or if you are in the list of extra-acs[2] for any official project team. If you are eligible to vote in an election, you should find your email with a link to the Condorcet Internet Voting Service (CIVS) page to cast your vote in the inbox of your Gerrit preferred email[3]. Emails should be already sent to everyone on Aug 30. If you have already voted using the CIVS link from that email, you don't need to submit your vote again now. What to do if you don't see the email and have a commit in at least one of the projects having an election: * due to a new CIVS policy, to get the email from poll supervisors, you must opt in to email communication opt in with your Gerrit email address at: https://civs1.civs.us/cgi-bin/opt_in.pl * eligible voters who opt in to email communication late can see a pending poll invitation and vote until Sep 18, 2024 23:45 UTC. * check the trash or spam folders of your Gerrit Preferred Email address, in case it went into trash or spam * wait a bit and check again, in case your email server is a bit slow * find the sha of at least one commit from the project's deliverable repos[0] and email the election officials[4]. If we can confirm that you are entitled to vote, we will add you to the voters list for the appropriate election. Our democratic process is important to the health of OpenStack, please exercise your right to vote! Candidate statements/platforms can be found linked to Candidate names on this page: https://governance.openstack.org/election/ Happy voting, [0] https://www.openstack.org/community/members/ [1] The list of the repositories eligible for electoral status: https://opendev.org/openstack/governance/src/tag/to-be-released/reference/p… [2] Look for the extra-acs element in [1] [3] Sign into review.openstack.org: Go to Settings > Contact Information. Look at the email listed as your preferred email. That is where the ballot has been sent. [4] https://governance.openstack.org/election/#election-officials -- Best regards Slawek Kaplonski skaplons(a)redhat.com

1 0

[cinder] Offering local storage using the device mapper clone target
by Jan Horstmann 04 Sep '24

04 Sep '24

Hello cinder team and other interested parties, I have recently started thinking about ways to offer local storage as part of the OpenStack block storage service. Naturally any solution would require to transparently migrate volumes to and with the instances they are attached to, so that the user experience of being able to access their storage from anywhere would not be degraded. Fortunately, with the device mapper clone target ([1]), a mechanism is available in the linux kernel which allows to do exactly that: Create copies of devices (even network attached ones) while allowing to immediately use the target device. I have detailed the motivation and some implications of having local storage in the block storage service in a blog post [2], which I do not want to repeat here in order to not make this mail longer than it needs to be. The blog post also outlines a way to implement a cinder driver leveraging this mechanism, which is loosely based on the way cinder migrates volumes between backends, i.e.: When a volume is going to be attached, a new volume is created on the instance's host, which is backed by a device mapper clone target with the original volume as its source; Then the volume objects are switched and a local connector is returned, which may be used immediately. Since then I went ahead and created a proof of concept cinder driver ([3]), which roughly follows what is outlined in the post (some lessons were already learned during implementation). The driver is based on the in-tree lvm driver for basic volume management and can do basic things like create/delete, attach/detach and also handles live migration and rebuilding of instances. There is nothing currently regarding snapshots and volume migrations will definitly not work as expected yet, but it should be sufficient to get a feeling for the general idea. I am currently looking for feedback on the idea and the implementation and would like to know if this is of interest to others. I am happpy to answer any questions or present this in more detail. In case anyone wants to take the driver for a spin I am going to leave some instructions on how to set up the environment in [4]. Regards, Jan [1] https://docs.kernel.org/admin-guide/device-mapper/dm-clone.html [2] https://scs.community/2024/07/12/local-block-storage-via-dm-clone/ [3] https://github.com/janhorstmann/openstack-cinder/tree/59e1808a24d112d2e8d08… [4] - Set up an OpenStack environment - Create an lvm2 volume group and run a cinder volume service on every hypervisor - Configure a new backend on every volume service with the same `volume_backend_name` Configure it like you would an lvm backend, but with `volume_driver = cinder.volume.drivers.dmclone.DMCloneVolumeDriver`, e.g.: [local-volumes] volume_driver = cinder.volume.drivers.dmclone.DMCloneVolumeDriver volume_group = local volume_backend_name = local-volumes target_helper = tgtadm target_protocol = iscsi lvm_type = default I have tested with the values above, but any target driver should work -- Jan Horstmann Senior Cloud Engineer Mail: horstmann(a)osism.tech Web: https://osism.tech OSISM GmbH Talweg 8 / 75417 Mühlacker / Deutschland Geschäftsführer: Christian Berendt Unternehmenssitz: Mühlacker Amtsgericht Mannheim, HRB 750852

1 1

[OSSA-2024-003] OpenStack Ironic: Unvalidated image data passed to qemu-img (CVE-2024-44082)
by Brian Rosmaita 04 Sep '24

04 Sep '24

======================================================== OSSA-2024-003: Unvalidated image data passed to qemu-img ======================================================== :Date: September 04, 2024 :CVE: CVE-2024-44082 Affects ~~~~~~~ - Ironic: <21.4.3, >=22.0.0 <23.0.2, >=23.1.0 <24.1.2, >=25.0.0 <26.0.1 - Ironic-python-agent: <9.4.2, >=9.5.0 <9.7.1, >=9.8.0 <9.11.1, >=9.12.0 <9.13.1 Description ~~~~~~~~~~~ Dan Smith and Julia Kreger of Red Hat and Jay Faulkner of G-Research noticed a vulnerability in image processing for Ironic, in which a specially crafted image could be used by an authenticated user to exploit undesired behaviors in qemu-img, including possible unauthorized access to potentially sensitive data. Mitigation ---------- The attached patches contain code allowing Ironic and the Ironic-Python-Agent (IPA) to pre-screen images before passing them to qemu-img. You should patch both of them because in some popular deployment configurations, it is possible for images to bypass the Ironic conductor for more efficient image downloads. In situations where it is not possible to patch IPA, there is a new configuration option, ``[conductor]conductor_always_validates_images``. When this option is True, all image downloads are forced to occur through the Ironic conductor, where they are validated. However, this may incur a significant performance hit; hence our recommendation to patch both Ironic and IPA. Patches are available for both Ironic and IPA for all maintained branches, namely, the Dalmatian development branch (current master) through Antelope. For these branches, ``conductor_always_validates_images`` defaults to False. For the unmaintained branches (Zed through Victoria), patches are not available for IPA due to a significant risk of regressions. For these branches, the Ironic patches set ``conductor_always_validates_images`` to True, as this is the only safe way to run Ironic with an un-patched IPA. Cached images ------------- Images in the Ironic image cache should be purged. To purge the cache, stop the Ironic conductor and remove the files in the ``[pxe]/instance_master_path`` directory on that conductor node. Supported image formats ----------------------- A new configuration option, ``[conductor]permitted_image_formats`` controls what image formats will be accepted by Ironic. By default, the allowed formats are 'raw' and 'qcow2', which are the only formats that Ironic is tested with. In previous releases, it was possible to use images of other, unsupported formats with Ironic, but this is now blocked by default due to security issues arising from the image conversion process. It is possible, though not recommended, to expand the list of permitted formats. Alternatively, users requiring the use of an image in an unsupported format can convert the image offline into a supported format before uploading the image to Glance or otherwise making the image available to Ironic. Consult the Ironic documentation for details. Warning ------- As is well-documented, the OpenStack Ironic project does not support the use of ironic-lib for non-Ironic use cases. In its normal supported context, ironic-lib assumes that images have been pre-screened before ironic-lib comes in contact with them. Thus, unsupported use of ironic-lib leaves you vulnerable to the exploit outlined in this OSSA. The Ironic project intends eventually to explicitly remove the vulnerable methods in ironic-lib, but reminds the reader that independent use of ironic-lib is not supported. Bugfix branches --------------- Patches are available for maintained bugfix branches. These patches will be merged to the maintained branches, but no new releases will be triggered from those branches. Patches ~~~~~~~ - https://review.opendev.org/927972 (2023.1/antelope(ironic)) - https://review.opendev.org/927979 (2023.1/antelope(ironic-python-agent)) - https://review.opendev.org/927970 (2023.2/bobcat(ironic)) - https://review.opendev.org/927978 (2023.2/bobcat(ironic-python-agent)) - https://review.opendev.org/927968 (2024.1/caracal(ironic)) - https://review.opendev.org/927976 (2024.1/caracal(ironic-python-agent)) - https://review.opendev.org/927965 (2024.2/dalmatian(ironic)) - https://review.opendev.org/927974 (2024.2/dalmatian(ironic-python-agent)) - https://review.opendev.org/927969 (Bugfix/24.0 (ironic)) - https://review.opendev.org/927967 (Bugfix/25.0 (ironic)) - https://review.opendev.org/927966 (Bugfix/26.0 (ironic)) - https://review.opendev.org/927983 (Bugfix/9.12 (ironic-python-agent)) - https://review.opendev.org/927981 (Bugfix/9.13 (ironic-python-agent)) - https://review.opendev.org/927984 (Bugfix/9.9 (ironic-python-agent)) - https://review.opendev.org/927982 (Unmaintained/victoria(ironic)) - https://review.opendev.org/927980 (Unmaintained/wallaby(ironic)) - https://review.opendev.org/927977 (Unmaintained/xena(ironic)) - https://review.opendev.org/927975 (Unmaintained/yoga(ironic)) - https://review.opendev.org/927973 (Unmaintained/zed(ironic)) Credits ~~~~~~~ - Dan Smith from Red Hat (CVE-2024-44082) - Jay Faulkner from G-Research (CVE-2024-44082) - Julia Kreger from Red Hat (CVE-2024-44082) References ~~~~~~~~~~ - https://launchpad.net/bugs/2071740 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-44082 Notes ~~~~~ - For more information about the unadvisability of calling 'qemu-img info' on untrusted images, see CVE-2024-04467. - The ironic unmaintained/* branches will receive no point releases, but patches for them are provided as a courtesy. - The ironic-python-agent unmaintained/* branches will receive no point releases, nor are patches for them available. - The ironic and ironic-python-agent bugfix/* branches will receive no point releases, but patches for them are provided as a courtesy. -- Brian Rosmaita OpenStack Vulnerability Management Team https://security.openstack.org/vmt.html

1 0

[keystoneauth] OpenID Connect support for Onelogin broke Okta
by Pavlo Shchelokovskyy 04 Sep '24

04 Sep '24

Hi all, I'd like to bring your attention and discuss the following bug https://bugs.launchpad.net/keystoneauth/+bug/2078437 When onboarding Caracal release internally, we've noticed a regression in keystoneauth, namely that openstack CLI using v3OidcPassword auth no longer works with Okta provider. Okta returns the following error: HTTP 401 {"error":"invalid_request","error_description":"Cannot supply multiple client credentials. Use one of the following: credentials in the Authorization header, credentials in the post body, or a client_assertion in the post body." It looks like this is the aftermath of the following patch https://review.opendev.org/c/openstack/keystoneauth/+/881969 that presumably aimed to fix the Onelogin provider. I've prepared a patch that should fix both my Okta use case and, to my understanding, also address the original Onelogin concern, but I do not obviously have the use case and Onelogin access the original author of the patch had, and also it seems the author has changed jobs and is no longer active in OpenStack. Maybe somebody still has such an access to try and validate the original Onelogin use case using my patch? Or at least validate my conclusions I've outlined in the bug comments on why this should work and satisfy both cases: https://review.opendev.org/c/openstack/keystoneauth/+/927581 CC Graeme Moss who has ported that original Onelogin patch to Antelope and Zed and thus might have some additional perspective. Best regards, -- Dr. Pavlo Shchelokovskyy Principal Software Engineer Mirantis Inc www.mirantis.com

1 0

[largescale-sig] Next meeting: August 21, 9utc
by Thierry Carrez 04 Sep '24

04 Sep '24

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 9UTC. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20240821T09 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

3 2

Subject: Best Deployment Method for Large-Scale Production OpenStack
by Thamanna Farhath 04 Sep '24

04 Sep '24

Hello OpenStack Community, We are preparing for a large-scale OpenStack deployment in production and currently use Kolla Ansible. Before proceeding, we want to know if Kolla Ansible is the best choice for production or if there are better alternatives for large-scale deployments. Any recommendations or insights on the best deployment methods for production environments would be greatly appreciated. Thank you! Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

4 3

[neutron] "eventlet" deprecation status
by Rodolfo Alonso Hernandez 04 Sep '24

04 Sep '24

Hello Neutrinos: During the last week we have started the "journey" to deprecate "eventlet" in Neutron and the stadium projects. We have two fronts to deal with: the Neutron API and the Neutron agents. The Neutron agents are spawned using "oslo.service" ``ServiceLauncher``, based on "eventlet". It is first needed the deprecation of "eventlet" in this project. The Neutron API was still using, by default, the "eventlet" server. The migration to the WSGI module, to the detriment of the "eventlet" implementation, was required. Most of the main project APIs have already migrated. The Neutron API using ML2/OVS was already migrated to WSGI but it was not fully functional. The periodic workers were not running, at least when spawned with devstack: * https://review.opendev.org/c/openstack/devstack/+/922125 It was also noticeable the instability of some ML2/OVS tempest CI jobs. That was caused by the RPC event handler implementation in the Neutron API: * https://review.opendev.org/c/openstack/neutron/+/926922 The Neutron API using ML2/OVN was not working at all. It was first needed to refactor the way to initialize the OVN plugin: * https://review.opendev.org/c/openstack/neutron/+/922470 ML2/OVN was also missing the maintenance worker, that was not running when using WSGI: * https://review.opendev.org/c/openstack/devstack/+/922611 * https://review.opendev.org/c/openstack/neutron/+/922074 With ML2/OVN, that usually doesn't need RPC communication, was a problem with the RPC worker (my bad for so many iterations...): * https://review.opendev.org/c/openstack/devstack/+/924473 * https://review.opendev.org/c/openstack/devstack/+/924693 * https://review.opendev.org/c/openstack/devstack/+/925412 The TLS CI job running with ML2/OVN was failing too: * https://review.opendev.org/c/openstack/neutron/+/925376/ The next steps: * To test the Neutron API using ML2/OVN (that doesn't use oslo.messaging for RPC) without any "eventlet" call (at least not calling the monkey patch method). That will be a POC patch only, but enough for testing. * Once we have new versions of the oslo libraries, we'll start with the deprecation in the agents. Regards.

1 0

[all][rbac][tacker][horizon][neutron][octavia] oslo.policy 4.4.0 enable the RBAC new defaults by default
by Ghanshyam Mann 04 Sep '24

04 Sep '24

Hi All, You might have seen some fixes and testing changes on your project to test the oslo.policy 4.4.0. oslo.policy 4.4.0 enables the RBAC new defaults by default, which means those will be enabled for all the OpenStack services unless they have disabled them by overriding the default value. It enables both enforce_scope as well enforce_new_defaults. The requirement change[1] adds the oslo.policy 4.4.0 in upper-constraints which will enable the latest oslo.policy in upstream CI. Along with jobs failing on requirement change, I tested more integration jobs and fixing the failures[2]. Most of the fixes have been merged, and a few of them are up for review. Below is the list of fixes up for review, please merge them before the requirement change is merged. If there is something else I missed to test and failed, let me know on IRC or here. In-progress fixes: - Horizon: - https://review.opendev.org/c/openstack/horizon/+/927571 - Tacker: - https://review.opendev.org/c/openstack/tacker/+/926089/5 - Neutron + Designates job (making it non-voting until we figure out the root cause): - https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/927648 - Octavia (Michael is working on more modifications on this ): - https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/926867 [1] https://review.opendev.org/c/openstack/requirements/+/925464 [2] https://review.opendev.org/q/topic:%22secure-rbac%22+owner:gmann@ghanshyamm… -gmann

2 4

[ops][kolla][kolla-ansible] Lets Encrypt Deployment on 2024.1/v18
by Forrest Fuqua 03 Sep '24

03 Sep '24

The documentation seems a little incomplete when it comes to deploying a stack with let's encrypt The attached config is after trying a few settings, but the lets encrypt container errors out every time during deployment causing haproxy to not get a proper ssl certificate the services.d entries are showing its ignoring kolla_internal_fqdn_cert and still using kolla_internal.pem in most services causing the following deployment error TASK [service-ks-register : keystone | Creating services] **************************************************************************************************************************************************** FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (5 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (4 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (3 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (2 retries left). FAILED - RETRYING: [100.70.0.1]: keystone | Creating services (1 retries left). failed: [100.70.0.1] (item=keystone (identity)) => {"action": "os_keystone_service", "ansible_loop_var": "item", "attempts": 5, "changed": false, "item": {"description": "Openstack Identity Service", "endpoints": [{"interface": "internal", "url": "https://openstack.cyberrange.rit.edu:5000"}, {"interface": "public", "url": "https://openstack.cyberrange.rit.edu:5000"}], "name": "keystone", "type": "identity"}, "module_stderr": "Failed to discover available identity versions when contacting https://openstack.cyberrange.rit.edu:5000. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 715, in urlopen\n httplib_response = self._make_request(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 404, in _make_request\n self._validate_conn(conn)\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 1058, in _validate_conn\n conn.connect()\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connection.py\", line 419, in connect\n self.sock = ssl_wrap_socket(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/util/ssl_.py\", line 449, in ssl_wrap_socket\n ssl_sock = _ssl_wrap_socket_impl(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/util/ssl_.py\", line 493, in _ssl_wrap_socket_impl\n return ssl_context.wrap_socket(sock, server_hostname=server_hostname)\n File \"/usr/lib/python3.10/ssl.py\", line 513, in wrap_socket\n return self.sslsocket_class._create(\n File \"/usr/lib/python3.10/ssl.py\", line 1100, in _create\n self.do_handshake()\n File \"/usr/lib/python3.10/ssl.py\", line 1371, in do_handshake\n self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/requests/adapters.py\", line 486, in send\n resp = conn.urlopen(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 799, in urlopen\n retries = retries.increment(\n File \"/opt/ansible/lib/python3.10/site-packages/urllib3/util/retry.py\", line 592, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1021, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/requests/adapters.py\", line 517, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py\", line 133, in _do_create_plugin\n disc = self.get_discovery(session,\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 605, in get_discovery\n return discover.get_discovery(session=session, url=url,\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/discover.py\", line 1459, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/discover.py\", line 539, in __init__\n self._data = get_version_data(session, url,\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/discover.py\", line 106, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1154, in get\n return self.request(url, 'GET', **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 930, in request\n resp = send(**kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1025, in _send_request\n raise exceptions.SSLError(msg)\nkeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://openstack.cyberrange.rit.edu:5000: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/tmp/ansible-tmp-1720019606.1431425-10481-181523778594420/AnsiballZ_catalog_service.py\", line 107, in <module>\n _ansiballz_main()\n File \"/tmp/ansible-tmp-1720019606.1431425-10481-181523778594420/AnsiballZ_catalog_service.py\", line 99, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/tmp/ansible-tmp-1720019606.1431425-10481-181523778594420/AnsiballZ_catalog_service.py\", line 47, in invoke_module\n runpy.run_module(mod_name='ansible_collections.openstack.cloud.plugins.modules.catalog_service', init_globals=dict(_module_fqn='ansible_collections.openstack.cloud.plugins.modules.catalog_service', _modlib_path=modlib_path),\n File \"/usr/lib/python3.10/runpy.py\", line 224, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib/python3.10/runpy.py\", line 96, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 211, in <module>\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 207, in main\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 415, in __call__\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 113, in run\n File \"/tmp/ansible_os_keystone_service_payload_m68qs7ok/ansible_os_keystone_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 175, in _find\n File \"/opt/ansible/lib/python3.10/site-packages/openstack/service_description.py\", line 89, in __get__\n proxy = self._make_proxy(instance)\n File \"/opt/ansible/lib/python3.10/site-packages/openstack/service_description.py\", line 289, in _make_proxy\n found_version = temp_adapter.get_api_major_version()\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/adapter.py\", line 352, in get_api_major_version\n return self.session.get_api_major_version(auth or self.auth, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/session.py\", line 1289, in get_api_major_version\n return auth.get_api_major_version(self, **kwargs)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 497, in get_api_major_version\n data = get_endpoint_data(discover_versions=discover_versions)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 268, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/base.py\", line 131, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py\", line 203, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/opt/ansible/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py\", line 155, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://openstack.cyberrange.rit.edu:5000: HTTPSConnectionPool(host='openstack.cyberrange.rit.edu', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} #config excerpts from globals.yml kolla_base_distro: "ubuntu" kolla_internal_vip_address: "100.70.0.20" # /etc/hosts updated on all hosts to have openstack.cyberrange.rit.edu point to the internal IP kolla_internal_fqdn: "openstack.cyberrange.rit.edu" kolla_external_vip_address: "129.21.246.130" kolla_external_fqdn: "openstack.cyberrange.rit.edu" kolla_enable_tls_internal: "yes" kolla_enable_tls_external: "yes" kolla_certificates_dir: "/etc/kolla/certificates" kolla_external_fqdn_cert: "{{ kolla_certificates_dir }}/haproxy.pem" kolla_internal_fqdn_cert: "{{ kolla_certificates_dir }}/haproxy.pem" #kolla_admin_openrc_cacert: "" kolla_copy_ca_into_containers: "yes" haproxy_backend_cacert: "{{ 'ca-certificates.crt' if kolla_base_distro in ['debian', 'ubuntu'] else 'ca-bundle.trust.crt' }}" haproxy_backend_cacert_dir: "/etc/ssl/certs" letsencrypt_email: "fffics(a)rit.edu" enable_letsencrypt: yes letsencrypt_cert_server: "https://acme-v02.api.letsencrypt.org/directory" # attempt to renew Let's Encrypt certificate every 12 hours letsencrypt_cron_renew_schedule: "0 */12 * * *"

3 2

Best Deployment Method for Large-Scale Production OpenStack
by Thamanna Farhath 03 Sep '24

03 Sep '24

Hello OpenStack Community, We are preparing for a large-scale OpenStack deployment in production and currently use Kolla Ansible. Before proceeding, we want to know if Kolla Ansible is the best choice for production or if there are better alternatives for large-scale deployments. Any recommendations or insights on the best deployment methods for production environments would be greatly appreciated. Thank you! Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 1

[release][PTLs] 2024.2 / Dalmatian Cycle Highlights
by Elõd Illés 03 Sep '24

03 Sep '24

Hi, This is a reminder, that *this week* is the week for Cycle highlights [1][2]! They need to be added to deliverable yamls so that they can be included in release marketing preparations. (See the details about how to add them at the project team guide [3]) [1] https://releases.openstack.org/dalmatian/schedule.html [2] https://releases.openstack.org/dalmatian/schedule.html#d-cycle-highlights [3] https://docs.openstack.org/project-team-guide/release-management.html#cycle… Thanks, Előd Illés irc: elodilles @ #openstack-release

1 0

[openstack-ansible] ussuri to victoria major upgrade
by Ram Ram 03 Sep '24

03 Sep '24

Hi Team I am planning openstack-ansible major upgrade from current version Ussuri to next sequential version Victoria. I am checking below link for the same. https://docs.openstack.org/openstack-ansible/victoria/admin/upgrades/major-… And I need support regarding below points: 1. What are the pre-requisite backups to be taken before the upgrade? 2. What is the impact of the upgrade , are all openstack services upgraded parallel or one by one. Example: there are 3 control nodes having each openstack control service lxc container (neutron, nova-api, cinder) running, so at a time one service of openstack is upgraded on each control node example nova (control-1)--->nova(control02)--->nova(control03) or all together. 3. What is the downtime expected in this upgrade? 4. How are openstack services upgraded on the compute nodes? What is the impact on VM running on the compute node that is being upgraded? 5. Any suggestion or experience which can be helpful for the upgrade. 6. What should be the rollback plan in case some issue occurs? Regards Rambo

2 1

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-4)
by Goutham Pacha Ravi 02 Sep '24

02 Sep '24

Hello Stackers, Last week was the feature freeze [1] within OpenStack projects that follow a "cycle-with-rc" release model [2]. These project teams are currently closing out a few feature freeze exceptions and working on bug fixes and documentation so they can ship their 2024.2 "Dalmatian" release candidates next week. When this happens, these projects can begin merging feature changes to the trunk branches again as they target the next release, 2025.1 "Epoxy." The release schedule for Epoxy has been published [3]. Members of the OpenInfra community are gathering in Suwon, South Korea, for the OpenInfra Summit on the 3rd and 4th of September 2024 [4]. This turned out to be a "sold-out" event; we're excited to hear about this event from folks participating in person. It's election season in the OpenStack community. We're electing four members to the Technical Committee [5]. The polling window officially opens on 2024-09-04. If you've been an individual member of the OpenInfra Foundation and have contributed to any OpenStack project within the past year, you should have received a ballot via email. If you have not, please send the election officials an email [6], or hop onto the #openstack-election channel on OFTC and let them know. During the past week, the TC merged an update to the tested runtimes for the 2025.1 release, which is due on 2025-04-04 [3]. === Weekly Meeting === The OpenStack Technical Committee meeting last week was held over IRC, and it was well attended [7]. We began brainstorming our agenda for various upcoming events, including the OpenInfra Summit in Suwon [4], the OpenInfra Days in North America (2024-10-15 to 2024-10-16), and the OpenInfra Virtual Project Teams Gathering (2024-10-21 to 2024-10-25). There will be some current TC members and candidates at these in-person events; Amy Marrich (spotz) will be hosting a meet-and-greet. Please stop by and say hello. The team also discussed the health of the OpenStack Zuul Gate jobs. Ghanshyam Mann (gmann) brought up the requirements bump to oslo.policy 4.4.0. Most projects have addressed incompatibilities; however, there is still some ongoing work [8]. One of the two donated ARM64 clouds was lost recently, affecting Gate jobs running on ARM64. The remaining cloud (osuosl) is slower than expected because of this consolidation. Projects like Kolla had to stop running image build jobs for ARM64. There was discussion on the importance and relevance of ARM support, with some members noting continued interest in ARM usage, especially in AI/ML. Jeremy Stanley (fungi) initiated a discussion [9] on improving communication patterns around code review and contributions, highlighting the need for more engagement from TC members. We also discussed that project teams, including the Technical Committee, must vet the questions in the User Survey and weed out questions or answer choices that are no longer relevant. The next meeting of the OpenStack Technical Committee is on 2024-09-03. This meeting will be held simultaneously over video and IRC. Please find the agenda for the meeting and the details to join within the TC Meeting Wiki [10]. I'm looking forward to your participation. === Governance Proposals === ==== Merged ==== Define testing runtime for 2025.1 release | https://review.opendev.org/c/openstack/governance/+/926150 ==== Open for review ==== charms: retire unmaintained charms | https://review.opendev.org/c/openstack/governance/+/926793 Retire all single charm repositories | https://review.opendev.org/c/openstack/governance/+/903490 Retire kuryr-kubernetes and kuryr-tempest-plugin | https://review.opendev.org/c/openstack/governance/+/922507 === Upcoming Events === 2024-09-03: OpenInfra Summit Asia, Suwon 2024-09-06: OpenInfra Days, Tokyo 2024-09-10: OpenInfra Board Meeting: https://board.openinfra.dev/ 2024-09-18: OpenStack 2025.1 Election Polling Ends 2024-10-15: OpenInfra Days NA, Indianapolis: https://ittraining.iu.edu/explore-topics/titles/oid-iu/ Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] Cycle-with-RC release model: https://releases.openstack.org/reference/release_models.html#cycle-with-rc [3] 2025.1 Release Schedule: https://releases.openstack.org/epoxy/schedule.html [4] OpenInfra Summit 2024, Suwon: https://2024.openinfraasia.org/a/schedule [5] 2025.1 OpenStack Technical Committee Election: https://governance.openstack.org/election/#tc-candidates [6] 2025.1 OpenStack Election Officials: https://governance.openstack.org/election/#election-officials [7] TC Meeting IRC Log, 2024-08-27: https://meetings.opendev.org/meetings/tc/2024/tc.2024-08-27-18.00.log.html [8] oslo.policy 4.4.0 enables the RBAC new defaults by default https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [9] Bridging the gap between the community and contributing orgs: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [10] TC Meeting Agenda, 2024-09-03: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda

1 0

Can not work on Dashboard(throwing 504-Timed out) on Openestack-on-ansible environment
by sudeb ghosh 02 Sep '24

02 Sep '24

Hi Team , We have an 'Openstack on Ansible' Private cloud environment with 3 controllers & 35 Compute host nodes. Age of this environment is around 4 yrs. DISTRIB_RELEASE: 21.2.1 DISTRIB_CODENAME: Ussuri DISTRIB_DESCRIPTION: Openstack-Ansible Top of this Linux containers are there where the services are running inside such as neutron, galera, horizon, keystone, etcd, nova-api, utility etc for all 3 controllers. Current Set Up: We find in haproxy config file only controller2 & controller3 are configured. So the galera container along with Mariadb Services are running fine within controller2(Primary) & controller3. But Galera container in controller1 is stopped & mariadb service not running here. Issue Started with: All of suddent we faced, user not able to log in dashboard 5 days back. The dashboard page is not getting loaded. Action Taken: We find Maria-DB in controller2 is non-primary & DB on controller-3 is down. Made Maria-DB in controller2 primary & DB on controller-3 up which resulted to load the page & log in. Current Issue: 1) But the Dashboard page is too slow to work. Sometime throwing 504-Timed out error 2) VM Console(VNC) not working for any VM 3) NOt able to create any VM(Showing 'Scheduling' continuously) 4) 25/35 Hypervisons are showing down in Dashboard within 'Hypervisor list' tab but those are up as per CLI 5) Not able to create any volume Request you to go through the Problem description & Can anybody help me out providing the solution for the same? Pl do intimate me if any more info is required. Regards, Sudeb Ghosh sudeb_ece(a)yahoo.co.in BR// Sudeb Ghosh 7044064878 9332034788

2 3

[neutron] Bug deputy report for week of August 25th
by Miguel Lavalle 02 Sep '24

02 Sep '24

I was the Neutron bugs deputy last week. Following is a brief summary of the reported bugs: Critical bugs ----------------- None High bugs -------------- https://bugs.launchpad.net/neutron/+bug/2078382 [OVN] User defined router flavor with no LSP associated to router interfaces. Assignee Miguel Lavalle. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/917800/ https://bugs.launchpad.net/neutron/+bug/2077942 [OVN][FT] ``test_agent_delete`` is randomly failing. Assignee Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/927236 https://bugs.launchpad.net/neutron/+bug/2078518 neutron designate scenario job failing with new RBAC. Needs owner Under investigation ------------------------- https://bugs.launchpad.net/neutron/+bug/2077879 [OVN] Router gateway does not have a chassis defined Edit. Reporter is testing solution given by ihrachys based on https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926940. Waiting from reporter to indicate whether the solution is satisfactory or not.

1 0

[nova][ci] openstack-tox-docs job failing, hold your rechecks
by melanie witt 01 Sep '24

01 Sep '24

Hey all, Just a heads up... The last several openstack-tox-docs job runs on the master branch have failed, both gate and check queues: https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-docs&pro… because of AFAICT the following warning which is treated as an error: 2024-08-28 19:24:04.604443 | ubuntu-jammy | /home/zuul/src/opendev.org/openstack/nova/.tox/docs/lib/python3.10/site-packages/openstackdocstheme/page_context.py:70: RemovedInSphinx90Warning: Sphinx 9 will drop support for representing paths as strings. Use "pathlib.Path" or "os.fspath" instead. 2024-08-28 19:24:04.649996 | ubuntu-jammy | src_file = full_src_file[len(str(app.builder.env.srcdir)):].lstrip('/') The error looks like it's coming from Sphinx and I did find an upper-constraints bump of Sphinx happened earlier today: https://review.opendev.org/c/openstack/requirements/+/927358 The version was bumped from 7.4.6 to 8.0.2 -- which may possibly be related to the failures. I will look more into the failure but if anyone else knows how to fix this, please feel free to propose something. -melwitt

2 3